AI-support for the detection of intracranial large vessel occlusions: One-year prospective evaluation

K. Leeuwen, M. Becks, D. Grob, F. de Lange, J. Rutten, S. Schalekamp, M. Rutten, B. van Ginneken, M. de Rooij and F. Meijer

Heliyon 2023;9(8).



Few studies have evaluated real-world performance of radiological AI-tools in clinical practice. Over one-year, we prospectively evaluated the use of AI software to support the detection of intracranial large vessel occlusions (LVO) on CT angiography (CTA).


Quantitative measures (user log-in attempts, AI standalone performance) and qualitative data (user surveys) were reviewed by a key-user group at three timepoints. A total of 491 CTA studies of 460 patients were included for analysis.


The overall accuracy of the AI-tool for LVO detection and localization was 87.6\%, sensitivity 69.1\% and specificity 91.2\%. Out of 81 LVOs, 31 of 34 (91\%) M1 occlusions were detected correctly, 19 of 38 (50\%) M2 occlusions, and 6 of 9 (67\%) ICA occlusions. The product was considered user-friendly. The diagnostic confidence of the users for LVO detection remained the same over the year. The last measured net promotor score was −56\%. The use of the AI-tool fluctuated over the year with a declining trend.


Our pragmatic approach of evaluating the AI-tool used in clinical practice, helped us to monitor the usage, to estimate the perceived added value by the users of the AI-tool, and to make an informed decision about the continuation of the use of the AI-tool.