Researchers from the Center for Diagnostics and Telemedicine created a platform that autonomously tests AI algorithms designed for medical tasks, like analyzing diagnostic images. A prototype of the platform is hosted on GitHub.
The first working prototype of the platform is hosted on the popular GitHub service, and developers from all over the world can take part in its improvement by adding verification criteria depending on the purpose of the services.Source: Center for Diagnostics and TelemedicineAlgorithms need to be tested for their technical readiness and verify that they meet the characteristics needed before implementation. Once an algorithm passes the tests, it can then be integrated into the medical system. The new platform creates an autonomous analytical validation process for algorithms.
Currently, analytical validation is performed manually. This process risks human error and accidental or deliberate deviations from the approved test program. Datasets are vulnerable to manipulation and potentially put test participants in unequal conditions.
The new platform allows developers to independently conduct preliminary tests of their algorithms. It gives researchers access to unlimited samples of data instances to fine-tune the algorithms. The platform has uniform rules of use and it is possible to test several algorithms simultaneously. It records the time a given algorithm spends on data processing. Developers receive automatic testing results as soon as their algorithm is done. The automated process minimizes the human factor and therefore eliminating human error and data manipulation.
A comparison of service verification results with reference data is transparent. Developers can see the metrics used and how the final result in the report was reflected. The platform can be improved with additional metrics. The metrics are used to evaluate algorithm performance for certain medical purposed. Additions to the platform are closely monitored because only metrics with scientific justification will be included.
A paper on the new testing platform was presented at the European Congress of Radiology.
