Researchers at the University of Michigan warn that AI-generated clinical notes may introduce biases and inaccuracies into medical records.

AI scribes are increasingly used in hospitals to reduce the time doctors spend on administrative tasks, but experts warn these tools may compromise the quality of clinical data. A recent article in the journal Learning Health Systems highlights risks including algorithmic bias, data hallucinations, and performance drift. Authors Taofeeq Oluwatosin Togunwa and Jodyn Platt argue that these systems require continuous auditing and measurement to ensure equitable and accurate patient care.

Risks of Bias and Inaccuracy

AI models are often trained on specific demographic data, which can lead to performance issues for patients with different accents. When inputs do not align with training data, the systems may produce 'hallucinations' or fabrications that become part of a patient's official record. While clinicians are expected to review these notes, researchers note that there is little incentive to perform line-by-line verification if the output appears generally correct.

Need for Continuous Auditing

The accuracy of AI scribes can degrade over time if the models are not updated to reflect real-world data, a phenomenon known as performance drift. To mitigate these risks, the authors emphasize the necessity of continuous auditing and a shift in how health systems measure the effectiveness of these technologies. They suggest that health systems must ensure that the benefits of AI, such as faster diagnosis or improved access to care, are distributed equitably among patients.