Can AI outperform radiologists in X-ray reporting accuracy? In the opinion of Darius Barušauskas, a Kaggle Grandmaster and Head of Data Science at Oxipit, AI will definitely perform well in cases where radiologist disagreement is expected to be low.
However, X-rays challenge AI in a wide spectrum of cases where human disagreement is likely. Can we trust AI at the same level as a radiologist performing the same challenging task?
In this interview we discuss AI machine learning model development, how can AI address radiologist subjectivity and the most challenging tasks in X-ray reporting automation.
In 2018 Oxipit launched its first AI medical imaging suite – ChestEye – which provides preliminary reports for 75 findings. What was instrumental in developing this vast finding library?
With ChestEye we aimed to cover a large scope of findings encountered by the radiologists on the daily basis. Initially we started with information extraction from radiology reports, identifying the most common findings. Probably around 20 chest X-ray findings are most prevalent. But to create a platform for routine medical practice, it should as well cover less frequent findings. Thus we arrived at 75.
We developed our general model to cover all 75 findings together with additional task-specific models to address most subtle and important pathologies.
Could you highlight key improvements in Oxipit machine learning models over the course of ChestEye development?
We put a lot of effort into improving the specialized models. The task specific models significantly improved our detection of pneumothorax, nodules and fractures, which are the most challenging for radiologists to identify, as well as detection of intubation and catheter malpositions.
What are key prerequisites in X-ray machine learning model development?
We are now working with a 1.7M image dataset, with new data added constantly. The significant performance gains are achieved when moving from tens-of-thousands to hundreds-of-thousands dataset size. Having even more data helps to make better models on rare and important pathologies.
The whole development process is incremental, as there are quite a lot of ways to improve performance: upgrading models with additional high quality data, improving data augmentations or identifying new machine learning approaches for the task at hand.
At Oxipit multi-read radiologist studies are a part of the model training process. A team of radiologists – from different geographies and with different experience levels – report on the same X-ray image. The consensus opinion helps us to improve our training and validation approaches, thus reducing model error rates over time.
What is specific to AI working on X-rays as compared to other modalities?
Datasets coming from the hospitals feature single-read imaging studies. Regarding X-rays it should not be too surprising that some reports are made by a less experienced radiology resident or a radiographer, as experienced radiologists prefer mastering other modalities. It is clear that an X-ray report is only a specialist opinion on a highly subjective medical image. And this makes it hard to identify what the “absolute truth” is.
The subjectivity in X-ray reports is the biggest challenge in developing highly efficient machine learning models for AI X-ray reporting. Some high profile data scientists, who were unaware of this problem, predicted that the currently available technology will be able to automate radiology in a few years. I have a feeling that full automation in any medical imaging modality is as hard as solving a self driving car problem – and many believe that this is not going to happen soon.
Overall, X-rays might appear as a simple medium – two dimensional projection as compared to CT or MRI which encompass much more data. However, better representation of the human body in CT and MRI results in a higher reporting certainty – which is very useful for training AI models.
The data science problem in AI X-ray reporting narrows down to high expectations for AI to be objective and infallible in a very subjective medical imaging modality – which is hard to achieve by definition.
Oxipit is now working on automating healthy patient chest X-ray reports. What technical changes went into development of this product?
The models we developed for ChestEye serve as a stepping stone for further product development. While with ChestEye we focused to cover the widest scope of findings, the current development is to perfect the models to reliably achieve zero false negatives.
We seek to eliminate false negatives by analyzing them on a case-by-case basis. There are ongoing research-based pilots at 4 locations where AI Is operating alongside radiologists. This helps us to identify the weakest links in the current models. Although we are already at zero false negatives at all trial locations, for Q3 we aim to introduce improvements in nodule and consolidation detection.
You are a die-hard Kaggler, standing at 4th at some time in the global rankings. What attracted you to the platform? And what professional achievement in the field of AI medical imaging would you regard as a personal epic win?
I enjoy working with data and Kaggle provides data and problems from many different industries – which itself is very entertaining from a modelling perspective. This is a contrast to having a day job where most of the time you would work with small tasks in a specific industry.
Moreover, kaggle provides a ranking system which is like an ATP rating for data scientists. At some time I was very involved in the platform. Now the developments at Oxipit and my family takes most of my time so I only kaggle occasionally.
Prior to Kaggle and Oxipit I spent 5 years in developing ML financial models. What attracted me to the medical imaging field is the complexity of machine learning problems and the value it can bring to patient care if these issues are solved.
Medical AI tools require regulatory approval. There is a lot of human subjectivity. Widespread adoption of AI will require a paradigm shift in the medical framework.
So for the epic win I would consider the development of autonomous algorithm which – after ticking all certification and regulatory approval boxes – can make clinical decisions. As currently the biggest bottleneck in healthcare is the lack of medical specialists – including the fact that ⅔ of the world population do not have access to medical imaging services, this would bring a substantial improvement in care.
About Oxipit
Oxipit is a computer vision software startup specialized in medical imaging. With a team of award-winning data scientists and medical doctors, the company aims to introduce innovative Artificial Intelligence/Deep Learning breakthroughs to everyday clinical practice. Oxipit is the authors of CE certified multi-award winning ChestEye radiology imaging suite.