Review Article
Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies

https://doi.org/10.1016/j.jclinepi.2013.12.002Get rights and content

Abstract

Objective

To investigate how consensus is operationalized in Delphi studies and to explore the role of consensus in determining the results of these studies.

Study Design and Settings

Systematic review of a random sample of 100 English language Delphi studies, from two large multidisciplinary databases [ISI Web of Science (Thompson Reuters, New York, NY) and Scopus (Elsevier, Amsterdam, NL)], published between 2000 and 2009.

Results

About 98 of the Delphi studies purported to assess consensus, although a definition for consensus was only provided in 72 of the studies (64 a priori). The most common definition for consensus was percent agreement (25 studies), with 75% being the median threshold to define consensus. Although the authors concluded in 86 of the studies that consensus was achieved, consensus was only specified a priori (with a threshold value) in 42 of these studies. Achievement of consensus was related to the decision to stop the Delphi study in only 23 studies, with 70 studies terminating after a specified number of rounds.

Conclusion

Although consensus generally is felt to be of primary importance to the Delphi process, definitions of consensus vary widely and are poorly reported. Improved criteria for reporting of methods of Delphi studies are required.

Introduction

What is new?

  • Definitions of consensus in Delphi studies vary widely and are poorly reported.

  • Attainment of consensus is infrequently used as a criterion for ending a Delphi study.

  • Methodologic criteria are proposed for the reporting of Delphi studies.

A Delphi study is a widely used method to obtain input from a group of experts [1], [2], [3]. Although initially developed to predict cold war enemy attack probabilities, the technique has been widely used across numerous disciplines as a method to seek expert opinion in an iterative structured manner. The key features of the method are anonymity between participants with controlled feedback provided in a structured manner. Participants then may adjust their initial ratings based on feedback from the group in a number of subsequent iterations [1], [4]. Although classically the method seeks to elicit consensus on the topic under study, there are variations of the Delphi such as the dissensus or Policy Delphi that aim to seek a broad range of opinions without achieving consensus [5].

Although the notion of consensus is fundamental to many Delphi studies, the definition of what constitutes consensus is less clear [3], [4], [6]. Investigators have adopted different approaches to defining consensus including formal measures of agreement, degree of uncertainty around a point estimate, decreases in variance of group responses, or the proportion of participants agreeing to a particular viewpoint [1], [2], [6], [7].

The objective of this systematic review was to document the range of definitions of consensus in a representative, randomly selected, cross-disciplinary selection of Delphi studies. The rationale for the study was based on the fact that although the Delphi technique is widely regarded as a consensus development technique, our impression from reading Delphi publications was that achievement of consensus is often assumed to occur by virtue of performing a Delphi study. We were concerned that many studies do not adequately define criteria for achievement of consensus and that even when consensus has been defined, it is not always clear whether the prespecified criteria for consensus have been a factor in deciding when to stop the Delphi process. We believe that failure to adequately define and use criteria for consensus challenges the notion that the results of a Delphi study reflect the consensus of the group of experts.

Section snippets

Search strategy

The ISI Web of Science (Thompson Reuters, New York, NY) and Scopus (Elsevier, Amsterdam, NL) databases were used to conduct the search. The objective of using these databases was to obtain a cross-disciplinary selection of Delphi studies. The search was performed using “Delphi” as a text word term. The results were limited to articles published in calendar years 2000–2009 in English. The results were exported into an EndNote X4.0 database (Thompson Reuters) and duplicates removed via both the

Results

A total of 3,056 unique manuscripts formed the “study population.” About 247 of these articles were reviewed to obtain the 100 manuscripts describing a Delphi process (Appendix).

Characteristics of the “study sample” are provided in Table 2. The articles selected were evenly balanced among the years covered by the review. Almost half of the studies originated in North America, with studies from Europe making up the second largest group. Almost three-quarters of the Delphi studies were within

Discussion

Most Delphi studies were conducted for a specific number of rounds, without a formal criterion for consensus. Therefore, when authors conclude, as most did, that the results of the study reflect the consensus opinion, it would seem that the achievement of consensus within a Delphi study is assumed to be an integral part of the technique, as has been suggested in the literature [1], [2], [3], [6], [7]. The fact that consensus was not reached in only 1 of the 64 studies where an a priori

Conclusion

We believe that there is a need to improve the reporting of Delphi studies, along the lines of a CONSORT-like guideline, as is used for randomized controlled trials [8]. We propose that the investigators who undertake Delphi studies should report a standard set of quality indicators. Proposed indicators are listed in Table 6. If the aim of the Delphi study is to elicit consensus, then a clear definition for what constitutes consensus should be provided a priori together with threshold values

Acknowledgments

The authors would like to thank Elizabeth Uleryk for her assistance with the search strategy.

I.R.D. was supported by a Fellowship award from the Canadian Institutes of Health Research with additional support from the Surgeon Scientist Training Program, Department of Surgery, University of Toronto. He is also the recipient of a Graduate Studentship Award from the Canadian Liver Foundation and the Chisholm Memorial Fellowship, Post Graduate Medical Education Office, the University of Toronto.

References (8)

There are more references available in the full text version of this article.

Cited by (1592)

View all citing articles on Scopus

Conflict of interest declaration: The authors do not have any conflict of interest to disclose in terms of this manuscript.

View full text