The advent of machine learning and artificial intelligence were set to revolutionize medicine and radiology seemed like the best place to start! Artificial intelligence (AI) computer vision systems have already been employed to deal with imaging tasks. The field was rich with data for machine learning (ML) model training. The path towards reporting automation looked clear cut.
Fast forward a few years. Despite having a developed AI imaging provider ecosystem and a variety of AI radiology products, the ‘AI revolution’ is rather an evolutionary tread towards widespread clinical adoption.
Oxipit COO Jogundas Armaitis shares his views on AI in radiology reporting steps and missteps.
In the past years the media focused on ‘AI taking over radiology’. Why did the concept of industry disruption fail to materialize?
AI in radiology merges the fields of medical and data sciences. We can look at it as a clash of professional cultures. In terms of data science, the roadmap for model and product development is quite clear. However, data science people are used to quantifiable exact truths. More data = more precise metrics.
In the medical field doctors daily make data driven executive decisions – and that sounds a lot like data science. However, data scientists often underestimate the level of subjectivity in radiology, as evidenced by inter- and intra-radiologist variance. It is absolutely normal for medical professionals to initially disagree and reach consensus only after a prolonged discussion. Add the diversity of patients, edge cases and a simple data problem suddenly becomes very complex.
Multi-reader studies are a crucial part of the research and development process at Oxipit. This means that multiple radiologists report on the same study. One clear takeaway from this research is that some specialists are three times more likely to report certain findings as compared to others. Such variations make reporting automation a subtle and challenging endeavour.
This does not mean that consensus is entirely absent. At Oxipit, we have identified and quantified subsets of radiological images where consensus exists. We believe that automated AI radiology reporting should start there.
Can we clearly say that – despite the improvements in AI technology – the full automation of radiologist workflow will not happen? Or should we expect this to happen in the next 10-20-30 years?
We believe that it is not only a question of technology but also of product development, ecosystem formation, and careful integration into medical processes. Even now AI is capable of performing at a similar level as an average radiologist on some tasks, yet few-to-none such products are used in a routine clinical setting.
Due to the subjective nature of medical imaging, product development should firstly focus on diagnosis where the widest consensus exists. We have learned that the absolute majority of radiologists concur on X-rays featuring no significant findings. Therefore, we have chosen healthy patient reporting as the first step towards automation.
As ChestEye can already identify 75 radiological findings, we can identify normal chest X-rays with a high degree of certainty. The large number of supported findings enables us to quickly make further product iterations.
Could you describe the ‘under the hood’ technical changes that went into the development of healthy patient reporting product?
A constant dialogue between the customers, and our radiology and data science teams is key. This led us to change our product deployment path with the aim to build trust and provide evidence-based confidence for our customers. Our three-stage approach starts with threshold optimization which also allows us to quantify the performance of the product on retrospective data.
We then proceed to the second stage, where AI is shadowing the radiologist. This enables us to demonstrate the prospective performance. The customer is kept informed by periodic analytics reports. The final stage is autonomous reporting by the AI with regular sampling audits to ensure quality.
These steps build understanding and trust of how our software performs at a particular customer site with their specific X-ray devices, protocols, and patient demographics. There is no other way to be absolutely confident in the performance of AI in a particular institution.
What is the feedback from medical institutions where the new trial Oxipit product is deployed?
We have deployed the product in a number of medical institutions in two European countries with a variety of patient profiles and cases since the start of the year. The feedback has so far been strongly positive. The number of false negatives has been consistently close to zero at all sites.
Aren’t you concerned that such a step-by-step product roadmap does not sound so grand as a ‘right here, right now’ full workflow automation?
Trust is more important than moving rapidly. Oxipit in particular has a CE-marked product which supports 75 radiological findings, however, we have narrowed our focus down for that reason. Despite the early enthusiasm, it is now obvious that the industry is not even close to being ready for full radiology workflow automation. At Oxipit, we firmly believe that AI solutions can already create substantial value albeit in a more limited scope than initially hoped for.
We are a pragmatic organization, focusing on practical applications that can improve quality of care for patients and assist radiologists. We strongly feel that identifying subsets where broad consensus exists and working on their automation is the right way to go, aiming for a widening scope of workflow automation while keeping consistent performance and trust in the focus. Only by building trust in the community can we expect widespread clinical adoption of AI radiology reporting.
Oxipit is a computer vision software startup specialized in medical imaging. With a team of award-winning data scientists and medical doctors, the company aims to introduce innovative Artificial Intelligence/Deep Learning breakthroughs to everyday clinical practice. Oxipit is the authors of CE certified multi-award winning ChestEye radiology imaging suite.