Skip to main content
SpringerLink
Log in

Consensus conferences must include a systematic search and categorization of the evidence

  • Original Articles
  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Ideally, a consensus panel combines expert knowledge with external evidence derived from the literature. To date, many consensus conferences do not use a structured approach to search the literature, but simply compile an add-on reference list from all papers cited by the panelists. This study examined how well such panelists retrieved the relevant literature.

Methods

We used the reference lists of nine surgeons who took part in a consensus conference on common bile duct stones. We included all papers that were referred to as randomized controlled trials (RCTs). We then compared this list with a database search in order to calculate sensitivity and specificity.

Results

The nine experts cited between 35 and 518 papers, but only eight papers on average were RCTs. Of the 49 papers that the experts believed to be RCTs, only 23 actually were RCTs. The sensitivity resp. specificity for correctly identifying an RCT was 0.21 (95% Cl, 0.11–0.30) resp. 0.80 (95% Cl; 0.64–0.95). RCTs that included the word “randomized” in their title were significantly more likely to be identified (relative risk, 1.31; 95% Cl, 1.18–1.45).

Conclusion

Our data indicate that consensus panelists usually do not perform systematic literature searches, but simply use their favorite papers to back up their arguments. Because this may lead to a biased selection of the evidence base on which the consensus statements are founded, a systematic search of all relevant articles should become a mandatory task in any consensus or guideline process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R Rennie D, Schulz K, Simel D, Stroup DF (1996) Improving the quality of reporting randomiozed controlled trials. The CONSORT statement. JAMA 276: 637–639

    Article  CAS  Google Scholar 

  2. Chalmers TC, Frank CS, Reitman D (1990) Minimizing the three stages of publications bias. JAMA 263: 1392–1395

    Article  CAS  Google Scholar 

  3. Cook DJ, Greengold NL, Ellrodt AG, Weingarten SR (1997) The relation between systematic reviews and practice guidelines. Ann Intern Med 127: 210–216

    Article  CAS  Google Scholar 

  4. Dickersin K, Scherer R, Lefebvre C (1994) Identification of relevant studies for systematic reviews. Br Med J 309: 1286–1291

    Article  CAS  Google Scholar 

  5. Egger M, Davey Smith G (1998) Bias in location and selection of studies. Br Med J 316: 61–66

    Article  CAS  Google Scholar 

  6. Field MJ, Lohr KN (1990) Clinical practice guidelines: directions of a new program. National Academy Press, Washington, DC

    Google Scholar 

  7. Fink A, Kosecoff J, Chassin M, Brook RH (1984) Consensus methods: characteristics and guidelines for use. Am J Public Health 74: 979–983

    Article  CAS  Google Scholar 

  8. Fletcher SW, Fletcher RH (1998) Development of clinical guidelines. Lancet 352: 1876

    Article  CAS  Google Scholar 

  9. Fraser GM, Pilpel D, Hollis S, Kosecoff J, Brook RH (1993) Indications for cholecystectomy: the results of a consensus panel approach. Qual Assur Health Care 5: 75–80

    Article  CAS  Google Scholar 

  10. Gøtzsche PC (1987) Reference bias in reports of drug trials. Br Med J 295: 654–656

    Article  Google Scholar 

  11. Hayward RS, Wilson MC, Tunis SR, Bass EB, Guyatt G, for the Evidence-based Medicine Working Group (1995) Users’ guides to the medical literature. VIII: How to use clinical practice guidelines. A: Are the recommendations valid? JAMA 274: 570–574

    Article  CAS  Google Scholar 

  12. Joyce J, Rabe-Hesketh S, Wessely S (1998) Reviewing the reviews: the example of chronic fatigue syndrome. JAMA 280: 264–266

    Article  CAS  Google Scholar 

  13. Lohr KN, Leazer K, Mauskopf J (1998) Health policy issues and applications for evidence-based medicine and clinical practice guidelines. Health Policy 46: 1–19

    Article  CAS  Google Scholar 

  14. Lomas J, Anderson G, Enkin M, Vayda E, Roberts R, MacKinnon B (1988) The role of evidence in the consensus process: results from a Canadian consensus exercise. JAMA 259: 3001–3005

    Article  CAS  Google Scholar 

  15. McGlynn EA, Kosecoff J, Brook RH (1990) Format and conduct of consensus development conferences: multination comparison. Int J Technol Assess Health Care 6: 450–469

    Article  CAS  Google Scholar 

  16. Millat B, Fingerhut A, Flamant Y, Hay JM, Fagniez PL, Farah A, Duron JJ, Courchevel JM, the French Association for Research in Surgery (1999) Survey of the impact of randomised clinical trials on surgical practice in France. Eur J Surg 165: 87–94

    Article  CAS  Google Scholar 

  17. Muir Gray JA (1997) Evidence-based, locally-owned, patient-centred guideline development. Br J Surg 84: 1636–1637

    Article  CAS  Google Scholar 

  18. Mullan F, Jacoby I (1985) The town meeting for technology: the maturation of consensus conferences. JAMA 254: 1068–1072

    Article  CAS  Google Scholar 

  19. Neugebauer E, Troidl H (1995) Consensus methods as tools to assess medical technologies. Surg Endosc 9: 481–481

    CAS  PubMed  Google Scholar 

  20. Ollenschläger G, Helou A, Kostovic-Cilic L, Perleth M, Raspe HH, Reinhoff O, Selbmann HK, Oesingmann U (1998) Checklist for methodological quality of guidelines: a contribution to quality promotion of medical guidelines [in German] Z Ärztl Fortbild Qualitätssich 92: 191–194

    PubMed  Google Scholar 

  21. Paul A, Millat B, Holthausen U, Sauerland S, Neugebauer E, Scientific Committee of the European Association for Endoscopic Surgery (1998) Diagnosis and treatment of common bile duct stones (CBDS): results of a consensus development conference. Surg Endosc 12: 856–864

    Article  Google Scholar 

  22. Petrie J, Barnwell E, Grimshaw J, for the Scottish Intercollegiate Guideline Network (SIGN) (1999) Criteria for appraisal of clinical guidelines for national use. http://www.show.scot.nhs.uk/sign/critmain.htm (accessed on June 9, 1999)

  23. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J (1999) Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA 281: 1900–1905

    Article  CAS  Google Scholar 

  24. Shekelle PG, Woolf SH, Eccles M, Grimshaw J (1999) Developing guidelines. Br Med J 318: 593–596

    Article  CAS  Google Scholar 

  25. Sniderman AD (1999) Clinical trials, consensus conferences, and clinical practice. Lancet 354: 327–330

    Article  CAS  Google Scholar 

  26. Stocking B, Jennett B, Spiby J (1991) Criteria for change: the history and impact of consensus development conferences in the UK. King’s Fund Centre, London

    Google Scholar 

  27. U.S. Agency for Health Care Policy and Research (AHCPR) (1992) Acute pain management: operative or medical procedures and trauma. AHCPR, Rockville, MD

    Google Scholar 

  28. Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J (1999) Potential benefits, limitations, and harms of clinical guidelines. Br Med J 318: 527–530

    Article  CAS  Google Scholar 

  29. Wortman PM, Smyth JM, Langenbrunner JC, Yeaton WH (1998) Consensus among experts and research synthesis: a comparison of methods. Int J Technol Assess Health Care 14: 109–122

    Article  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Biochemical and Experimental Section, 2nd Department of Surgery, University of Cologne, Ostmerheimer Strasse 200, 51109, Cologne, Germany

    S. Sauerland & E. Neugebauer

Authors
  1. S. Sauerland
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. E. Neugebauer
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

Online publication: 29 August 2000

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sauerland, S., Neugebauer, E. Consensus conferences must include a systematic search and categorization of the evidence. Surg Endosc 14, 908–910 (2000). https://doi.org/10.1007/s004640000283

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s004640000283

Key words

Search

Navigation