Purpose: To survey scientific evidence for all CE marked artificial intelligence (AI) based software products for radiology available as of April, 2020.
Materials and Methods: We created an online overview of CE-certified AI software products for clinical radiology based on vendor-supplied product specifications (www.aiforradiology.com). For these products, we conducted a systematic literature study on Pubmed for original, peer-reviewed, English articles published between Jan 1, 2015 and April 14, 2020 on the efficacy of the AI software. Papers were included when the product and/or company name were mentioned, when efficacy level 2 to 6 according to Fryback was reported on an independent dataset, and when the tool was applied on in vivo human data.
Results: Our product overview consisted of 100 CE-certified software products from 51 different vendors. Among the 839 papers screened, 108 met the inclusion criteria. For 70/100 products we did not find papers that met the inclusion criteria. The evidence of the other 30 products was predominantly (84%) focused on diagnostic accuracy (efficacy level 2). Half of the available evidence (49%) was independent and not (co)-funded or (co)-authored by the vendor. In more than half (55%) of the papers the version number of the product used in the study was not mentioned. From all studies, 20 (18%) used validation data from multiple countries, 42 (39%) were multicenter studies, 25 (23%) were performed with acquisition machines from multiple manufacturers.
Conclusion: One hundred CE-certified AI software products for radiology exist today. Yet, for the majority, scientific evidence on the clinical performance and clinical impact is lacking. These insights should raise awareness that certification may not guarantee technical and clinical efficacy of an AI product.
Clinical relevance: Our findings identify the available evidence for commercially available AI software, aiming to contribute to a safe and effective implementation of AI software in radiology departments.