Professor warns of dangerous failure modes in medical AI

At the FLCC webinar, Dave Gadeau explained how seemingly small dataset artifacts can produce hazardous clinical errors: when images used to train models include rulers or other consistent marks, the model can learn to use those marks as predictors instead of underlying pathology. "If you don't have a ruler on your chest at that time, AI is not gonna identify it as a tumor," he said, describing published study examples and mitigation efforts.

Gadeau also discussed large‑scale initiatives such as IBM's Watson, which struggled to translate elite literature training into reliable clinical decision support because of inconsistent records, HIPAA constraints and unstructured notes. He emphasized that domain coverage, data access and interoperability matter for medical AI reliability and urged careful validation before deployment.