AI medical tools keep changing and handling data with little public proof it's safe
A study finds most updated FDA-cleared AI devices report no evidence they still work well, while a New York hospital admits patient records were fed into AI tools without permission.
What happened: Two unrelated reports this week point to the same gap: hospitals and regulators aren't fully tracking how AI tools behave once they're already treating patients. A JAMA Network Open study found a third of 163 FDA-cleared AI medical devices got at least one algorithm update after clearance, but 85.7% of those updates came with no public evidence the change actually helped patients or staff. Separately, Weill Cornell Medicine disclosed that radiology patients' diagnoses, treatments, and medications were entered into AI tools without authorization during a research study in June, a violation of its own policies discovered three months later.
Why it matters: Patients increasingly encounter AI without knowing it: reading mammograms, flagging stroke signs on CT scans, interpreting heart rhythms. When a device's software changes, or a hospital's AI oversight fails, there's often no clear record of whether the tool still performs as promised or who saw sensitive data. The FDA's public clearance summaries are one of the few windows doctors and patients have into whether a device works, and the study found large gaps in testing details and outcomes. Weill Cornell's incident shows a separate risk: privacy rules don't automatically stop staff from feeding real patient records into AI systems during research, and the hospital still can't say how many patients were affected or which tools got the data.
How it works, plainly: Most AI medical devices are cleared under the FDA's 510(k) pathway, which checks that a product is similar enough to one already on the market, often without new clinical trials. After clearance, manufacturers can update features or the data an algorithm reads; only about 16% of updates were aimed purely at improving accuracy, while most added new capabilities. A December 2024 FDA rule called predetermined change control lets companies pre-approve future updates, so some changes now happen without any new public filing, meaning the true update count is likely higher than what's visible. Weill Cornell said no names or Social Security numbers were taken, but medical record numbers, provider names, and clinical histories were, details that still count as patient identifiers under HIPAA.
What happens next: Weill Cornell posted its notice on Nov. 5, right at the 60-day legal deadline after discovering the breach on Sept. 6. It has suspended the research, asked for the data to be deleted, and retrained staff, but hasn't confirmed deletion or disclosed how many patients were affected. The device study's authors are calling for standardized reporting and stronger postmarket monitoring, and the FDA already keeps a public list of AI-enabled devices. Patients can ask clinicians whether AI assisted in reading a scan, whether a human confirmed the result, and, for research participants, whether their data could be processed by AI tools at all.