Authors: S. Karunasena, M. Milne, M. G. Vasimalla, L. Danaher, M. Wilson, Q. Buchlak, C. Jones; Sydney/AU
Poster presented at the European Society of Radiology Congress 2022 | Poster number C-21465
DOI: 10.26044/ecr2022/C-21465
This study provides insight into radiologists’ feedback and perceptions following the implementation of an artificial intelligence (AI) assist device that comprehensively detects radiologic findings on chest radiograph (CXR) into a national-scale radiology network. We present data from an initial pilot investigation, as well as feedback from radiologists two months post wider network implementation across a diverse range of clinical practices.
Background
AI has the potential to significantly advance clinical radiology and improve patient outcomes through radiologists’ diagnostic accuracy, and thereby reductions in the rate of missed findings of importance in imaging studies [1, 2]. Most studies published to date evaluating AI devices, however, are conducted in non-clinical, experimental settings, without means of measuring impact on clinical outcomes and overall benefits to healthcare systems. Furthermore, positive impacts of AI device usage established in experimental settings may not necessarily translate to improved performance in the real world. Reports from a study conducted by Strohm et al. indicate that radiologists are less accepting of evidence for AI device validation when it does not originate from a clinical setting [3].
Many AI models have been developed for the purpose of improving radiologists’ diagnostic performance when interpreting CXRs [4]. The AI device reviewed in this study, Annalise Enterprise CXR, has been validated for performance in the detection of 124 findings on CXR [5]. Validation of diagnostic accuracy, however, is just the first step towards having an AI device being utilised in clinical practice.
Successful real-world implementation is dependent on multiple factors, including seamless integration into existing workflows, consistent performance across heterogenous clinical settings, and overall user acceptance, adoption, and engagement. The latter factors are particularly relevant and may present one of the most difficult challenges to overcome in widespread device implementation as it is known that individual predispositions towards the use of AI in clinical practice are highly variable. A study conducted by Huisman et al. demonstrated that radiologists’ attitudes towards AI are directly related to their overall understanding of it [6].
Pilot Study:
A 6-week pilot study was performed from November–December 2020, including eleven consultant radiologists situated in four states across Australia in a diverse range of practice settings. These radiologists had varying levels of experience post completion of radiology training. Five radiologists had 5 years or less of consultant experience, three had 6-10 years, and three had more than 10 years. Radiologists reported their normal daily case load, with the AI device used in every CXR interpretation. CXRs of patients from public hospitals, private hospitals, and community clinic settings were reported, and onsite reporting and remote reporting were incorporated into the study to best replicate real-world practice.
Figure 1: Flow diagram illustrating how the AI device was integrated into the radiologist workflow and how feedback was collected through a modified device user interface. N.B. RIS: radiology information system; PACS: picture archiving and communication system.
The AI device displayed CXR findings in a widget-style viewer linked to the existing PACS – when a case was opened, a findings list was generated and a CXR image displayed with selected findings highlighted by a colour overlay. Users compared their interpretation of the CXR with the AI device findings and provided feedback on device performance and clinical impact of the AI output for every case reported during the pilot period (a flow diagram of the process is presented in Figure 1). Feedback was obtained through a modified device interface developed for this study (Figure 2) that allowed the radiologist to answer the following questions per-case with minimal additional effort and time:
Did you agree with the findings provided by the device?
Did the device output significantly change the report?
Did the device output significantly change patient management?
Did the device output lead to recommendation of further imaging?
Figure 2: Example of the modified user interface where radiologist has indicated that they missed one finding in their interpretation (that the AI device detected). They can then select whether this significantly changed their report, impacted patient management, or led to further imaging.
Upon completion of the pilot period, radiologists were asked to complete a post-pilot survey, to obtain feedback on the usefulness of the CXR viewer, how easy it was to use in daily practice, and how it affected their perceptions of AI in radiology. The questions asked in this survey can be seen with the results for pilot study (Figures 5-7).
Feedback Post Wider Implementation:
In March 2021 the AI device was deployed across the entire IMED Radiology Network in Australia, including community clinics, hospital departments with inpatients and outpatients, as well as teleradiology services. A survey was distributed to collect further feedback from radiologists using the device two months after implementation. The aim of the survey was to gather insight on how many radiologists were using the AI device, the perceived usefulness through impact on CXR report findings, impact on reporting time and general sentiments towards using the device. The questions included in the survey are detailed in Appendix 1.
Results of Pilot Study:
In total, 2,972 CXR cases, from 2,665 unique patients, were reported during the pilot study, with a median patient age of 67 (IQR 50-77) and a 52.7% male to 47.3% female distribution.
Based on per-case feedback, radiologist agreement with AI device predictions was high with complete agreeance with the AI device findings in 2,572 cases (86.5%). There were 306 cases (10.2%) where the radiologists rejected one finding by the AI device, and 84 cases (2.8%) where radiologists rejected two or more findings. In 15 cases (0.5%), radiologists identified additional findings that were not detected by the AI device.
Figure 3: Results of per-case feedback from pilot study.
Radiologists’ feedback also indicated that use of the AI device resulted in significant changes to 92 cases across the group (3.1%). There were 43 cases (1.4%) for which the radiologists identified that patient management was changed as a result of pathology detected by the AI device. In 29 cases (1.0%), further imaging was recommended due to findings identified by the AI device. The data for this aspect of the pilot study is summarised in Figure 3.
Figure 4: Proportion of reports with significant changes due to use of AI device compared to radiologists’ experience.
Correlation was also made with radiologist experience and number of reports with significant changes, with a higher proportion of significant report changes for radiologists with less than 5 years experience (5% compared to group average of 3.1%) – see Figure 4.
Additionally, the correlation between number of findings detected by the device and rate of report change was reviewed. It was found that a higher number of priority findings detected by the AI device was associated with a higher rate of report change, with a median number of 8 findings in cases where the AI device led to changes to the report, compared to 5 findings for reports without significant changes.
End-of-Pilot Survey Results:
The survey was completed by 10 of 11 radiologists. Survey data indicated that 9 out of 10 respondents were satisfied with the accuracy of the CXR tool. 7 respondents were satisfied with reporting time using the AI tool and all participants thought the system was easy to use. Only 2 out of 10 radiologists stated they would not be disappointed if device access was removed.
Finally, 9 out of 10 radiologists indicated that their attitudes towards the AI device, as well AI in general, were improved after completing the study.
We
Figure 5: Radiologists’ satisfaction with reporting time using device and device accuracy at the conclusion of the pilot.
Figure 6: Radiologists’ feedback regarding AI device ‘ease of use’ and level of disappointment if device was removed from practice at the conclusion of the pilot.
Figure 7: Radiologists’ attitudes towards the AI device and AI in general at the conclusion of the pilot.
Feedback Post Wider Implementation:
The AI device was made accessible to more than 300 radiologists situated across community outpatient, inpatient hospital departments and remote teleradiology services. The post implementation survey was completed by 58 radiologists.
93% of respondents continued using the device after 2 months, with 90% indicating that it positively impacted their reporting. 75% of radiologists felt that reporting times were not negatively impacted; in fact, 25% thought their reporting times improved when using the device. 75% of respondents did not encounter any major technical difficulties whilst using the device. In terms of training prior to device usage, 88% were satisfied with the documentation provided. Finally, 73%
indicated they would be disappointed if they were no longer able to use the device. These results are summarised in Figure 8.
Figure 8: Radiologists’ feedback 2 months post widespread implementation of AI device
Radiologists’ feedback, reflected through results from the pilot study, as well as feedback two months post wider implementation throughout the radiology network, indicated high satisfaction with regards to device accuracy, impact on CXR reporting, user training and integration into existing workflows. The pilot study data showed that a significant number of CXR reports contained changes attributable to use of the AI device. Notably, this proportion was higher for radiologists with less than 5 years consultant experience, and there was also a correlation between number of findings per CXR and report changes. In addition to this, most radiologists felt there was no perceived reduction in reporting efficiency when using the device; in fact, 25% of radiologist who provided feedback two months after using the device felt it actually improved their reporting speed. Overall radiologist satisfaction with the AI device is reflected strongly in the high proportion of radiologists who continued using the device two months after implementation at their practice. Furthermore, 73% of radiologists stated they would actually be disappointed if device access was removed from their practice.
Importantly, it was also shown that attitudes of radiologists towards the AI device, and the use of AI in clinical practice in general, improved after using the device. This is a very positive outcome, as it paves the way for implementation of other AI devices into the radiology network to improve radiologist performance, and ultimately, the quality of patient care in healthcare systems.
S. Karunasena: Employee: I-MED Radiology; M. Milne: Employee: annalise.ai; M. G. Vasimalla: Employee: annalise.ai; L. A. Danaher: Employee: I-MED Radiology; M. Wilson: Employee: annalise.ai; Q. Buchlak: Employee: annalise.ai; C. M. Jones: Employee: annalise.ai, Employee: I-MED Radiology
Appendix 1 – Post-Implementation Survey Questions
| · How likely are you to recommend Annalise CXR to a friend or colleague? |
| · What are your main reasons for giving this score? |
| · Would you be disappointed if Annalise CXR was removed from your practice? |
| · How regularly do you use Annalise CXR when reporting CXR? |
| · How long did it take to become familiar or comfortable using Annalise CXR? |
| · How do you feel Annalise CXR has impacted your reporting time? |
| · How satisfied are you with the accuracy of findings displayed by Annalise CXR? |
| · Since using Annalise CXR, how often do you feel the findings have positively influenced or changed your reports? |
| · How satisfied were you with the training and documentation provided prior to using Annalise CXR? |
| · Have you encountered any major technical issues stopping you from using Annalise CXR? |
| · How much do you agree with the following statement: “I am confident the use of AI assisted CXR tool in radiological practice will lead to better patient outcomes” ? |
