Platform for testing radiological artificial intelligence-powered software

The amount of AI-based software used in radiology has been rapidly increasing in recent years, and the effectiveness of such AI services should be carefully assessed to ensure the quality of the developed algorithms. Manual assessment of such systems is a labor-intensive process. In this regard, an urgent task is to develop a specialized unified platform designed for automated testing of AI algorithms used to analyze medical images. The proposed platform consists of three main modules: a testing module that ensures interaction with the software being tested and collects data processing results; a viewing module that provides tools for visually evaluating the obtained graphic series and structured reports; a metrics calculation module that allows calculating diagnostic characteristics of the effectiveness of artificial intelligence algorithms. During the development, such technologies as Python 3.9, Apache Kafka, PACS and Docker were used. The developed platform has been successfully tested on real data. The obtained results indicate the potential of using the developed platform to improve the quality and reliability of AI services in radiation diagnostics, as well as to facilitate the process of their implementation in clinical practice.

1. Alowais S.A., Alghamdi S.S., Alsuhebany N., et al. Revolutionizing Healthcare: The Role of Artificial Intelligence in Clinical Practice. BMC Medical Education. 2023;23(1). https://doi.org/10.1186/s12909-023-04698-z

2. Tyurin I.E. Radiology in the Russian Federation. Journal of Oncology: Diagnostic Radiology and Radiotherapy. 2018;1(4):43–51. (In Russ.).

3. Strohm L., Hehakaya Ch., Ranschaert E.R., Boon W.P.C., Moors E.H.M. Implementation of Artificial Intelligence (AI) Applications in Radiology: Hindering and Facilitating Factors. European Radiology. 2020;30(10):5525–5532. https://doi.org/10.1007/s00330-020-06946-y

4. Kelly B.S., Judge C., Bollard S., et al. Radiology Artificial Intelligence: A Systematic Review and Evaluation of Methods (RAISE). European Radiology. 2022;32:7998–8007. https://doi.org/10.1007/s00330-022-08784-6

5. Vasilev Yu.A., Vladzymyrskyy A.V., Arzamasov K.M., Shulkin I.M., Astapenko E.V., Pestrenin L.D. Limitations of Using Artificial Intelligence Services to Analyze Chest X-Ray Imaging. Digital Diagnostics. 2024;5(3):407–420. (In Russ.). https://doi.org/10.17816/DD626310

6. Gusev A.V., Morozov S.P., Kutichev V.A., Novitsky R.E. Legal Regulation of Artificial Intelligence Software in Healthcare in the Russian Federation. Medical Technologies. Assessment and Choice. 2021;43(1):36–45. (In Russ.). https://doi.org/10.17116/medtech20214301136

7. Vasiliev Yu.A., Vladzimirsky A.V., Omelyanskaya O.V., et al. Methodology for Testing and Monitoring Artificial Intelligence-Based Software for Medical Diagnostics. Digital Diagnostics. 2023;4(3):252−267. https://doi.org/10.17816/DD321971

8. Erickson B.J., Kitamura F. Magician's Corner: 9. Performance Metrics for Machine Learning Models. Radiology: Artificial Intelligence. 2021;3(3). https://doi.org/10.1148/ryai.2021200126

9. Hicks S.A., Strümke I., Thambawita V., et al. On Evaluation Metrics for Medical Applications of Artificial Intelligence. Scientific Reports. 2022;12. https://doi.org/10.1038/s41598-022-09954-8

10. Müller D., Soto-Rey I., Kramer F. Towards a Guideline for Evaluation Metrics in Medical Image Segmentation. BMC Research Notes. 2022;15. https://doi.org/10.1186/s13104-022-06096-y

11. Brink L., Coombs L.P., Veettil D.K., et al. ACR's Connect and AI-LAB Technical Framework. JAMIA Open. 2022;5(4). https://doi.org/10.1093/jamiaopen/ooac094

12. Pavlov N.A., Andreychenko A.E., Vladzymyrskyy A.V., Revazyan A.A., Kirpichev Y.S., Morozov S.P. Reference Medical Datasets (MosMedData) for Independent External Evaluation of Algorithms Based on Artificial Intelligence in Diagnostics. Digital Diagnostics. 2021;2(1):49−66. https://doi.org/10.17816/DD60635

13. Gidde P.S., Prasad Sh.S., Singh A.P., et al. Validation of Expert System Enhanced Deep Learning Algorithm for Automated Screening for COVID-Pneumonia on Chest X-Rays. Scientific Reports. 2021;11. https://doi.org/10.1038/s41598-021-02003-w

14. Vladzimirskii A.V., Vasil'ev Yu.A., Arzamasov K.M., et al. Komp'yuternoe zrenie v luchevoi diagnostike: pervyi etap Moskovskogo eksperimenta. Moscow: Izdatel'skie resheniya; 2023. 388 p. (In Russ.).

15. Artyukova Z.R., Petraikin A.V., Kudryavtsev N.D., et al. Experience with Artificial Intelligence Algorithms for the Diagnosis of Vertebral Compression Fractures Based on Computed Tomography: From Testing to Practical Evaluation. Digital Diagnostics. 2024;5(3):505–518. (In Russ.). https://doi.org/10.17816/DD624250