AI Model Determines Race From X-Ray Images, and Nobody Knows How
An international team of scientists, including researchers from the Massachusetts Institute of Technology and Harvard Medical School, have trained an AI model with the surprising ability to accurately predict the race of a patient from their X-rays – something that even clinical experts are unable to do.
Published in the prestigious medical journal The Lancet, the research came about after scientists noticed how an AI program was more likely to miss signs of illness in Black patients, according to a report in the Boston Globe.
“We asked ourselves, how can that be if computers cannot tell the race of a person?” said Leo Anthony Celi, a co-author and an associate professor at Harvard Medical School.
The research
The team, consisting of scientists from the United States, Canada, Australia, and Taiwan, first trained an AI system by using X-rays and CT scans labeled with the person’s race. The diagnostic images came from different parts of the body, with no obvious markers of their race beyond the labels.
The resulting AI model was then shown images without labels, where it proceeded to identify the race with remarkable accuracy of well above 90 percent. According to Celi, it succeeded in definitizing people of different races when images from people of similar size, age, or gender were analyzed.
“This finding is striking as this task is generally not understood to be possible for human experts. We also showed that the ability of deep models to predict race was generalized across different clinical environments, medical imaging modalities, and patient populations…” wrote the authors.
One suspect cited in an interview is melanin, a pigment that determines skin color. The theory is that this information could somehow be embedded into X-rays and CT images in a way that human users have failed to notice. This remains an untested theory, however.
For now, the authors wrote: “Our study showed that medical AI systems can easily learn to recognize self-reported racial identity from medical images, and that this capability is extremely difficult to isolate. “
Given the findings, the authors are urging AI developers, regulators, and users involved in medical image analysis to use deep learning models with extreme caution due to the possibility of such models making racist or sexist decisions unwittingly.
“[The] use of as such information could be misused to perpetuate or even worsen the well documented racial disparities that exist in medical practice,” they concluded.
The full text of the report can be viewed here.
Image credit: iStockphoto/Vitalii Barida
