Elsevier

The Lancet

Volume 374, Issue 9695, 26 September–2 October 2009, Pages 1097-1104
The Lancet

Series
Challenges in evaluating surgical innovation

https://doi.org/10.1016/S0140-6736(09)61086-2Get rights and content

Summary

Research on surgical interventions is associated with several methodological and practical challenges of which few, if any, apply only to surgery. However, surgical evaluation is especially demanding because many of these challenges coincide. In this report, the second of three on surgical innovation and evaluation, we discuss obstacles related to the study design of randomised controlled trials and non-randomised studies assessing surgical interventions. We also describe the issues related to the nature of surgical procedures—for example, their complexity, surgeon-related factors, and the range of outcomes. Although difficult, surgical evaluation is achievable and necessary. Solutions tailored to surgical research and a framework for generating evidence on which to base surgical practice are essential.

Introduction

Evaluation of a therapeutic, procedure-based intervention presents several methodological and practical challenges for the surgical research community. Few, if any, of these challenges apply only to surgical procedures; many arise during the assessment of other non-pharmacological interventions, such as interventional radiology, technical procedures and devices, rehabilitation, behavioural interventions, and psychotherapy.1 However, what is arguably unique to surgery is the way in which many of these challenges coincide. Perhaps this situation leads many surgeons to view randomised controlled trials (RCTs)—although theoretically advantageous—to be too difficult and impractical to undertake, and at worst, irrelevant to their practice because of concerns about generalisability.2 Most of the same challenges also affect non-randomised studies and, in some cases, to a greater extent. Despite the barriers, an RCT remains the best possible study design for the assessment of therapeutic interventions.

This report, the second of three papers on surgical innovation and evaluation, presents the conclusions of a meeting held by the Balliol Collaboration on April 3, 2009. By identifying many issues related to surgical research and deconstructing them into constituent methodological parts, we targeted several important areas to develop guidance for appropriate, evidence-based surgical practice. Here, we discuss the challenges related to study design of surgical research and the challenges related to the nature of surgical interventions. Recommendations for improvement and solutions are presented in the third report in this Series.3

Section snippets

Randomised controlled trials

RCTs are considered the gold standard for establishing safety and efficacy of an intervention. Despite calls for surgical research to be more rigorous, the overall frequency of RCTs has been consistently low since the 1970s.4 Large, high-quality RCTs have been done in a variety of surgical specialties, but those of the surgical procedure itself are less common. Most surgical RCTs have focused on other aspects of the intervention, such as anaesthesia or pharmacological interventions, in

Complexity of surgical procedures

We need to recognise that many surgical interventions are complex and require appropriate evaluation.43 Surgical interventions, like other non-pharmacological interventions such as therapist-based and educational interventions, consist of several components that cannot be separated.44 This situation contrasts with most pharmacological interventions, which can be readily defined and standardised. Although the surgical procedure itself requires attention, a surgical intervention can depend on

Traditional master–student model

The traditional hierarchical system of surgery epitomises eminence-based medicine. This master–student apprenticeship tradition holds that the master has all the knowledge and skill and the student learns by observation and emulation. This approach can prevent new models and information from entering independent practice. Despite attempts to implement change with aggressive knowledge translation methods,64 adoption of best practice guidelines in surgery remains poor without involvement of

Conclusions

Rigorous evaluation of new surgical interventions, although difficult, is achievable and necessary. The complexity of surgical procedures makes it difficult, if not generally impossible, to mirror some aspects of pharmacological research. This shortcoming has contributed to uncertainty about the risk of biases and has led to scepticism about the value of surgical research. Although much criticism is aimed at RCTs of surgical procedures, few of the challenges apply only to this type of study

References (73)

  • SJ Rangel et al.

    Recent trends in the funding and utilization of NIH career development awards by surgical faculty

    Surgery

    (2004)
  • L Fischer et al.

    Four years of teaching principles in clinical trials—a continuous evaluation of the postgraduate workshop for surgical investigators at the study center of the German Surgical Society

    J Surg Educ

    (2009)
  • I Boutron et al.

    Extending the CONSORT statement to randomized trials of nonpharmacologic treatment: explanation and elaboration

    Ann Intern Med

    (2008)
  • MJ Solomon et al.

    Clinical studies in surgical journals—have we improved?

    Dis Colon Rectum

    (1993)
  • MN Wente et al.

    Perspectives of evidence-based surgery

    Dig Surg

    (2003)
  • M Zwarenstein et al.

    Improving the reporting of pragmatic trials: an extension of the CONSORT statement

    BMJ

    (2008)
  • TB Freeman et al.

    Use of placebo surgery in controlled trials of a cellular-based therapy for Parkinson's disease

    N Engl J Med

    (1999)
  • B Gordon et al.

    The Ipswich Childbirth Study: 1. A randomised evaluation of two stage postpartum penned repair leaving the skin unsutured

    Br J Obstet Gynaecol

    (1998)
  • L Neumayer et al.

    Open mesh versus laparoscopic mesh repair of inguinal hernia

    N Engl J Med

    (2004)
  • JN Weinstein et al.

    Surgical vs nonoperative treatment for lumbar disk herniation: the Spine Patient Outcomes Research Trial (SPORT): a randomized trial

    JAMA

    (2006)
  • RJ Lilford et al.

    Trials and fast changing technologies: the case for tracker studies

    BMJ

    (2000)
  • AM Grant et al.

    Minimal access surgery compared with medical management for chronic gastro-oesophageal reflux disease: UK collaborative randomised trial

    BMJ

    (2008)
  • MK Campbell et al.

    Preference trials

    Wiley encyclopedia of clinical trials

    (2007)
  • KM Taylor

    The doctor's dilemma: physician participation in randomized clinical trials

    Cancer Treat Rep

    (1985)
  • P McCulloch et al.

    Tolerance of uncertainty, extroversion, neuroticism and attitudes to randomized controlled trials among surgeons and physicians

    Br J Surg

    (2005)
  • JD Harrison et al.

    Surgical and oncology trials for rectal cancer: who will participate?

    Surgery

    (2007)
  • CE Margo

    When is surgery research? Towards an operational definition of human research

    J Med Ethics

    (2001)
  • S Pocock
  • JPT Higgins et al.

    Chapter 8. Assessing the risk of bias in included studies

  • I Boutron et al.

    Reporting methods of blinding in randomized trials assessing nonpharmacological treatments

    PLoS Med

    (2007)
  • JB Moseley et al.

    A controlled trial of arthroscopic surgery for osteoarthritis of the knee

    N Engl J Med

    (2002)
  • AJ London et al.

    Placebos that harm: sham surgery controls in clinical trials

    Stat Methods Med Res

    (2002)
  • RW Poolman et al.

    Reporting of outcomes in orthopaedic randomized trials: does blind of outcome assessors matter?

    J Bone Joint Surg

    (2007)
  • PJ Devereaux et al.

    Need for expertise based randomised controlled trials

    BMJ

    (2005)
  • S Hunt

    A fair way of donating hearts for transplantation

    BMJ

    (2000)
  • P Glasziou et al.

    When are randomised trials unnecessary? Picking signal from noise

    BMJ

    (2001)
  • Cited by (497)

    View all citing articles on Scopus
    View full text