Method of evaluation of autonomous software based on artificial intelligence technologies for mass preventive studies

The introduction of artificial intelligence (AI) technologies into medical practice requires a thorough assessment of their effectiveness, especially for systems operating autonomously. The method proposed in this study is based on a synthesis of the requirements of national standards in the field of medical AI developed by experts of the Scientific and Practical Clinical Center for Diagnostics and Telemedicine Technologies and data obtained as part of the "Moscow Experiment" on the introduction of innovative technologies. The testing was carried out on three AI software products used to analyze fluorographic studies in the period from January to May 2023. The evaluation included an analysis of the accuracy of algorithms (sensitivity, specificity), effectiveness in real clinical conditions, as well as a comparative analysis of the results with a quantitative interpretation of the data. The emphasis in the evaluation was on providing the AI system with a high level of diagnostic sensitivity, which will allow doctors to relieve themselves of routine monotonous work in mass preventive studies. The developed method demonstrated the possibility of a comprehensive assessment of autonomous AI systems, identifying differences in the effectiveness of products by key metrics. The proposed method allows systematizing the process of validating medical AI solutions, minimizing the risks of their incorrect use in autonomous operation. The results of the study can be used to standardize the assessment of AI tools in radiology and other areas of medicine that require a high level of diagnostic reliability.

1. Pugachev P.S., Gusev A.V., Kobyakova O.S., et al. Global trends in the digital transformation of the healthcare industry. National Health Care (Russia). 2021;2(2):5–12. (In Russ.). https://doi.org/10.47093/2713-069X.2021.2.2.5-12

2. Hoc Group on Application of AI Technologies. Artificial Intelligence in Healthcare: Directions of Standardization. In: Handbook of Artificial Intelligence in Healthcare: Vol 2: Practicalities and Prospects. Cham: Springer; 2022. pp. 231–257. https://doi.org/10.1007/978-3-030-83620-7_10

3. Koshechkin K.A., Svistunov A.A., Lebedev G.S., Fartushny E.N. Practice of using systems based on artificial intelligence in the field of circulation of medicines. Vestnik Roszdravnadzora. 2022;(3):27–33. (In Russ.).

4. Johnson A.E.W., Pollard T.J., Berkowitz S.J., et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data. 2019;6. https://doi.org/10.1038/s41597-019-0322-0

5. Do Hu.M., Spear L.G., Nikpanah M., et al. Augmented Radiologist Workflow Improves Report Value and Saves Time: A Potential Model for Implementation of Artificial Intelligence. Academic Radiology. 2020;27(1):96–105. https://doi.org/10.1016/j.acra.2019.09.014

6. Lomakov S.Yu., Strogonov E.A. Sovremennye aspekty organizatsii profilakticheskikh luchevykh issledovanii metodom flyuorografii. Mezhdunarodnyi akademicheskii vestnik. 2019;(11):4–6. (In Russ.).

7. Markelov Yu.M., Shchegoleva L.V. Clinical and Economic Aspects of Tuberculosis Detection During Mass Fluorographic Examinations of the Population. Journal of radiology and nuclear medicine. 2021;102(3):148–154. (In Russ.). https://doi.org/10.20862/0042-4676-2021-102-3-148-154

8. Markelov Yu.M., Schegoleva L.V. Evaluation of Clinical and Economic Efficiency and Impact of Mass Fluorography Screening on Tuberculosis Epidemiological Rates in Four Federal Districts of the Russian Federation with Different Levels of Population Coverage with Mass Fluorography Screening. Tuberculosis and Lung Diseases. 2023;101(1):8–16. (In Russ.). https://doi.org/10.58838/2075-1230-2023-101-1-8-16

9. Rubis L.V., Markelov Y.M. Evaluation of the efficiency of preventive examinations of the adult population, aimed at early tuberculosis diagnosis. Russian Medicine. 2021;27(3):227–235. (In Russ.). https://doi.org/10.17816/0869-2106-2021-27-3-227-235

10. Kelly B.S., Judge C., Bollard S.M., et al. Radiology artificial intelligence: a systematic review and evaluation of methods (RAISE). European Radiology. 2022;32:7998–8007. https://doi.org/10.1007/s00330-022-08784-6

11. Tanguay W., Acar Ph., Fine B., et al. Assessment of Radiology Artificial Intelligence Software: A Validation and Evaluation Framework. Canadian Association of Radiologists Journal. 2023;74(2):326–333. https://doi.org/10.1177/08465371221135760

12. Schaffter Th., Buist D.S.M., Lee Ch.I., et al. Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening Mammograms. JAMA Network Open. 2020;3(3). https://doi.org/10.1001/jamanetworkopen.2020.0265

13. Vasiliev Yu.A., Vlazimirsky A.V., Omelyanskaya O.V., et al. Methodology for testing and monitoring artificial intelligence-based software for medical diagnostics. Digital Diagnostics. 2023;4(3):252−267. https://doi.org/10.17816/DD321971

14. Gusev A.V., Astapenko E.M., Ivanov I.V., Zarubina T.V., Kobrinskii B.A. Principles for building confidence in artificial intelligence systems for healthcare. Vestnik Roszdravnadzora. 2022;(2):25–33. (In Russ.).

15. Gómez Ó., Mesejo P., Ibáñez Ó., et al. Evaluating artificial intelligence for comparative radiography. International Journal of Legal Medicine. 2024;138(1):307–327. https://doi.org/10.1007/s00414-023-03080-4

16. Thodberg H.H., Thodberg B., Ahlkvist J., Offiah A.C. Autonomous artificial intelligence in pediatric radiology: the use and perception of BoneXpert for bone age assessment. Pediatric Radiology. 2022;52(7):1338–1346. https://doi.org/10.1007/s00247-022-05295-w

17. Daye D., Wiggins W.F., Lungren M.P., et al. Implementation of Clinical Artificial Intelligence in Radiology: Who Decides and How? Radiology. 2022;305(3):555–563. https://doi.org/10.1148/radiol.212151

18. Çallı E., Sogancioglu E., Van Ginneken B., Van Leeuwen K.G., Murphy K. Deep learning for chest X-ray analysis: A survey. Medical Image Analysis. 2021;72. https://doi.org/10.1016/j.media.2021.102125

19. Morozov S.P., Vladzymyrskyy A.V., Sharova D.E., Akhmad E.S., Zinchenko V.V. The first Russian Federation national standards for artificial intelligence systems in medicine. Quality management in medicine. 2022;(1):58–62. (In Russ.).

20. Vasil'ev Yu.A., Arzamasov K.M., Vladzimirskii A.V., et al. Podgotovka nabora dannykh dlya obucheniya i testirovaniya programmnogo obespecheniya na osnove tekhnologii iskusstvennogo intellekta. [b.m.]: Izdatel'skie resheniya; 2024. 140 p. (In Russ.).

21. Vladzimirskii A.V., Vasil'ev Yu.A., Arzamasov K.M., et al. Komp'yuternoe zrenie v luchevoi diagnostike: pervyi etap Moskovskogo eksperimenta. [b.m.]: Izdatel'skie resheniya; 2023. 388 p. (In Russ.).

22. Vasilev Yu.A., Tyrov I.A., Vladzymyrskyy A.V., Arzamasov K.M., Pestrenin L.D., Shulkin I.M. A New Model of Organizing Mass Screening Based on Stand-Alone Artificial Intelligence Used for Fluorography Image Triage. Public Health and Life Environment – PH&LE. 2023;31(11):23–32. (In Russ.) https://doi.org/10.35627/2219-5238/2023-31-11-23-32

23. Mijwil M.M., Aggarwal K., Sonia S., et al. Has the Future Started? The Current Growth of Artificial Intelligence, Machine Learning, and Deep Learning. Iraqi Journal for Computer Science and Mathematics. 2022;3(1). https://doi.org/10.52866/ijcsm.2022.01.01.013