Every day, across thousands of American hospitals, artificial intelligence quietly shapes decisions that determine patient outcomes. An algorithm flags a patient as high risk for sepsis; a risk score informs whether a woman receives additional cancer screening; a deterioration model triggers an alert that sends a care team to a bedside. These tools are embedded in the workflows of nearly two-thirds of US hospitals, integrated into the electronic health record systems clinicians rely on daily. But many have never been reviewed by the FDA.

A new viewpoint in The Lancet Digital Health, co-authored by researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and Jameel Clinic, traces how this problem took root, why it carries serious consequences, and what genuine transparency would require to fix it.

The argument, the scientists say, is not that AI has no place in clinical decision-making. It is that a $4 billion market of clinical decision support tools operates largely beyond public accountability, leaving patients and providers often unable to know whether the tools influencing their care have been validated, by whom, or for which populations they work as intended.