Elsevier

Academic Radiology

Volume 16, Issue 1, January 2009, Pages 28-38
Academic Radiology

Original investigation
Assessment of Radiologist Performance in the Detection of Lung Nodules: Dependence on the Definition of “Truth”1

https://doi.org/10.1016/j.acra.2008.05.022Get rights and content

Rationale and Objectives

Studies that evaluate the lung nodule detection performance of radiologists or computerized methods depend on an initial inventory of the nodules within the thoracic images (the “truth”). The purpose of this study was to analyze (1) variability in the “truth” defined by different combinations of experienced thoracic radiologists and (2) variability in the performance of other experienced thoracic radiologists based on these definitions of “truth” in the context of lung nodule detection in computed tomographic (CT) scans.

Materials and Methods

Twenty-five thoracic CT scans were reviewed by four thoracic radiologists, who independently marked lesions they considered to be nodules ≥3 mm in maximum diameter. Panel “truth” sets of nodules were then derived from the nodules marked by different combinations of two and three of these four radiologists. The nodule detection performance of the other radiologists was evaluated based on these panel “truth” sets.

Results

The number of “true” nodules in the different panel “truth” sets ranged from 15 to 89 (mean 49.8 ± 25.6). The mean radiologist nodule detection sensitivities across radiologists and panel “truth” sets for different panel “truth” conditions ranged from 51.0 to 83.2%; mean false-positive rates ranged from 0.33 to 1.39 per case.

Conclusions

Substantial variability exists across radiologists in the task of lung nodule identification in CT scans. The definition of “truth” on which lung nodule detection studies are based must be carefully considered, because even experienced thoracic radiologists may not perform well when measured against the “truth” established by other experienced thoracic radiologists.

Section snippets

Patient Image Data

A total of 25 thoracic helical CT scans were collected from a single LIDC site in accordance with the previously published inclusion criteria (7, 14). Appropriate local institutional review board approval was obtained for the research use of scans that had been acquired in accordance with established clinical or ongoing research imaging protocols. Each CT scan had been acquired from a different patient (10 females, 15 males; age 40–75 years, median 59) on Aquilion (Toshiba Medical Systems,

Number of Nodules

A total of 91 lesions were identified as “nodule ≥3 mm” by at least one of the four radiologists. The number of nodules identified by each of the four radiologists is shown in Table 1. Radiologist C defined the fewest lesions as nodules (n = 20), and Radiologist A defined the most lesions as nodules (n = 63). For the nodules that were identified by each radiologist, Figure 1 presents the numbers of those nodules that were identified by that radiologist alone, by the radiologist and one other

Discussion

Several limitations are inherent in this study. First, the task of identifying nodules in the context of establishing “truth” for research studies differs from the identification task in the clinical setting, and the radiologists were asked to identify lesions without the benefit of accompanying clinical data. Second, pathologic information was not available for any of the lesions. Third, to define the study targets, radiologists were forced to make binary decisions as to the presence of

References (20)

There are more references available in the full text version of this article.

Cited by (0)

1

Supported in part by USPHS Grants U01CA091085, U01CA091090, U01CA091099, U01CA091100, and U01CA091103.

View full text