Introduction/Background The potential for artificial intelligence (AI) models to attain high accuracy in specific pathology related tasks has become clear in recent years. One illustrative example is our deep-learning model, trained to automatically detect Serous Tubal Intraepithelial Carcinoma (STIC), found in fallopian tube specimens. However, evaluating the true worth of an AI model in a diagnostic context requires more than just assessing its standalone performance. We therefore conducted a reader study to examine how the integration of this deep learning model influences the diagnostic performance of pathologists in detecting STIC.

Methodology Twenty-six pathologists and pathology residents, from 11 countries took part in a fully crossed multi-reader multi-case study. The study involved the review of 100 H&E-stained slides of fallopian tubes, comprising 30 cases and 70 controls. Participants examined all 100 slides both with and without the assistance of AI, with a washout period between sessions. The impact of the deep-learning model on accuracy, slide review time, and perceived diagnostic certainty was assessed using mixed-models analysis.

Results The use of our AI model resulted in a significant improvement in accuracy, whereby the average sensitivity increased from 82% to 93%. Additionally, readers had an average time reduction of 44 seconds per slide, corresponding to a 32% decrease in slide review time. Finally, there was a significant increase in participants’ perceived certainty, reflected by a 0.24-point increase on a 10-point scale.

Conclusion Among a group of 26 pathologists and pathology residents, the use of AI support resulted in a significant improvement in the accuracy of diagnosing STIC, accompanied by a substantial decrease in slide review time. This AI model holds the promise of offering valuable assistance to pathologists in the diagnosis of STIC, ultimately streamlining and optimizing the overall diagnostic process for STIC.

Disclosures This project was funded by the Dutch Cancer Society (KWF), project number 12950.