SeriesChallenges in evaluating surgical innovation
Introduction
Evaluation of a therapeutic, procedure-based intervention presents several methodological and practical challenges for the surgical research community. Few, if any, of these challenges apply only to surgical procedures; many arise during the assessment of other non-pharmacological interventions, such as interventional radiology, technical procedures and devices, rehabilitation, behavioural interventions, and psychotherapy.1 However, what is arguably unique to surgery is the way in which many of these challenges coincide. Perhaps this situation leads many surgeons to view randomised controlled trials (RCTs)—although theoretically advantageous—to be too difficult and impractical to undertake, and at worst, irrelevant to their practice because of concerns about generalisability.2 Most of the same challenges also affect non-randomised studies and, in some cases, to a greater extent. Despite the barriers, an RCT remains the best possible study design for the assessment of therapeutic interventions.
This report, the second of three papers on surgical innovation and evaluation, presents the conclusions of a meeting held by the Balliol Collaboration on April 3, 2009. By identifying many issues related to surgical research and deconstructing them into constituent methodological parts, we targeted several important areas to develop guidance for appropriate, evidence-based surgical practice. Here, we discuss the challenges related to study design of surgical research and the challenges related to the nature of surgical interventions. Recommendations for improvement and solutions are presented in the third report in this Series.3
Section snippets
Randomised controlled trials
RCTs are considered the gold standard for establishing safety and efficacy of an intervention. Despite calls for surgical research to be more rigorous, the overall frequency of RCTs has been consistently low since the 1970s.4 Large, high-quality RCTs have been done in a variety of surgical specialties, but those of the surgical procedure itself are less common. Most surgical RCTs have focused on other aspects of the intervention, such as anaesthesia or pharmacological interventions, in
Complexity of surgical procedures
We need to recognise that many surgical interventions are complex and require appropriate evaluation.43 Surgical interventions, like other non-pharmacological interventions such as therapist-based and educational interventions, consist of several components that cannot be separated.44 This situation contrasts with most pharmacological interventions, which can be readily defined and standardised. Although the surgical procedure itself requires attention, a surgical intervention can depend on
Traditional master–student model
The traditional hierarchical system of surgery epitomises eminence-based medicine. This master–student apprenticeship tradition holds that the master has all the knowledge and skill and the student learns by observation and emulation. This approach can prevent new models and information from entering independent practice. Despite attempts to implement change with aggressive knowledge translation methods,64 adoption of best practice guidelines in surgery remains poor without involvement of
Conclusions
Rigorous evaluation of new surgical interventions, although difficult, is achievable and necessary. The complexity of surgical procedures makes it difficult, if not generally impossible, to mirror some aspects of pharmacological research. This shortcoming has contributed to uncertainty about the risk of biases and has led to scepticism about the value of surgical research. Although much criticism is aimed at RCTs of surgical procedures, few of the challenges apply only to this type of study
References (73)
CABG is still the best treatment for multivessel and left main disease, but patients need to know
Ann Thorac Surg
(2006)- et al.
No surgical innovation without evaluation: the IDEAL recommendations
Lancet
(2009) - et al.
An evidence-based medicine review of lymphadenectomy extent for gastric cancer
Am J Surg
(2009) - et al.
Perceptions of equipoise are crucial to trial participation: a qualitative study of men in the ProtecT study
Control Clin Trials
(2003) - et al.
Randomised, prospective, single-blind comparison of laparoscopic versus small-incision cholecystectomy
Lancet
(1996) - et al.
Is surgical science dead? The Excelsior Society lecture
J Am Coll Surg
(2004) - et al.
Survival after bilateral versus single lung transplantation for patients with chronic obstructive pulmonary disease: a retrospective analysis of registry data
Lancet
(2008) - et al.
Practical considerations in the management of large multi-institutional databases
Ann Thorac Surg
(1994) - et al.
Surgeon-related variables and the clinical trial
Lancet
(1978) - et al.
Measuring the quality of surgical care: structure, process, or outcomes?
J Am Coll Surg
(2004)
Recent trends in the funding and utilization of NIH career development awards by surgical faculty
Surgery
Four years of teaching principles in clinical trials—a continuous evaluation of the postgraduate workshop for surgical investigators at the study center of the German Surgical Society
J Surg Educ
Extending the CONSORT statement to randomized trials of nonpharmacologic treatment: explanation and elaboration
Ann Intern Med
Clinical studies in surgical journals—have we improved?
Dis Colon Rectum
Perspectives of evidence-based surgery
Dig Surg
Improving the reporting of pragmatic trials: an extension of the CONSORT statement
BMJ
Use of placebo surgery in controlled trials of a cellular-based therapy for Parkinson's disease
N Engl J Med
The Ipswich Childbirth Study: 1. A randomised evaluation of two stage postpartum penned repair leaving the skin unsutured
Br J Obstet Gynaecol
Open mesh versus laparoscopic mesh repair of inguinal hernia
N Engl J Med
Surgical vs nonoperative treatment for lumbar disk herniation: the Spine Patient Outcomes Research Trial (SPORT): a randomized trial
JAMA
Trials and fast changing technologies: the case for tracker studies
BMJ
Minimal access surgery compared with medical management for chronic gastro-oesophageal reflux disease: UK collaborative randomised trial
BMJ
Preference trials
Wiley encyclopedia of clinical trials
The doctor's dilemma: physician participation in randomized clinical trials
Cancer Treat Rep
Tolerance of uncertainty, extroversion, neuroticism and attitudes to randomized controlled trials among surgeons and physicians
Br J Surg
Surgical and oncology trials for rectal cancer: who will participate?
Surgery
When is surgery research? Towards an operational definition of human research
J Med Ethics
Chapter 8. Assessing the risk of bias in included studies
Reporting methods of blinding in randomized trials assessing nonpharmacological treatments
PLoS Med
A controlled trial of arthroscopic surgery for osteoarthritis of the knee
N Engl J Med
Placebos that harm: sham surgery controls in clinical trials
Stat Methods Med Res
Reporting of outcomes in orthopaedic randomized trials: does blind of outcome assessors matter?
J Bone Joint Surg
Need for expertise based randomised controlled trials
BMJ
A fair way of donating hearts for transplantation
BMJ
When are randomised trials unnecessary? Picking signal from noise
BMJ
Cited by (497)
The quest for excellence in surgical research
2023, SurgeonRisk-of-bias assessment of the randomized clinical trials and systematic reviews on surgical treatments for breast cancer-related lymphedema: A mapping review
2023, Journal of Plastic, Reconstructive and Aesthetic SurgeryAntibiotic-loaded bone cement vs. plain cement as an infection prophylaxis in primary elective shoulder arthroplasty
2023, Seminars in Arthroplasty JSES
- ‡
For members see Lancet 2009; 374: 1089–96