Claudia Tanner

By Claudia Tanner

0

Share

Newsweek is a Trust Project member

See more of our trusted coverage when you search.Prefer Newsweek on Googleto see more of our trusted coverage when you search.


A leader in the health care and AI space has warned that racing China on AI could put patients at risk as President Donald Trump pushes to accelerate U.S. development.

“There is real concern that we’re going to harm people,” said Brian Anderson, CEO of the Coalition for Health AI at Newsweek’s AI Health Summit in New York.

He said that five years ago there was little appetite for slowing AI development to ensure models were safe. “I think that was the stage of let’s move fast, let’s innovate, let’s not lose this race with China,” he said.

Read More on Newsweek.AI

But Anderson explained that attitudes are changing as health systems explore increasingly autonomous AI applications in higher-risk settings where patients’ lives could be at stake.

He pointed to AI helping guide sepsis treatment as an example of a high-risk use where mistakes could harm patients.

Anderson also warned that AI agents in healthcare could “go rogue” and “access a skill” to operate outside a health system’s secure firewall, potentially leaking protected patient information.

His comments come after Trump vowed not to slow AI development in the race with China, saying “whoever wins AI, wins” and dismissing fears that the tools could destroy humanity as a “hoax.” He said the only "guardrails" needed for AI was a "strong and smart" president and that the technology “will not be stopped” during his presidency.

Dr. Nabil Chehade, chief clinical transformation, innovation and strategy officer at The MetroHealth System, echoed the concerns, and said his concern was not an isolated AI mistake but failure on a much larger scale.

“What keeps me up at night is when we fail at a colossal level, because now either suddenly the systems are not available, or because the system integrity is breached, and before we know it, a lot of harm has been caused.” He added: “I don't know what that will look like.”

Anderson said increasingly autonomous AI agents require close monitoring when deployed in hospitals and that some technology companies are already monitoring agents in real time.

“They’re monitoring each step of an agent’s workflow to make sure that it doesn’t go off and do something dangerous,” he said.

Even routine uses carry risks, Anderson said, pointing to an AI system being tested in Utah to renew existing prescriptions. A change in a patient’s lab results could mean a medication should be adjusted or stopped rather than renewed, he said. “It’s not without risk.”

Brendan Carr, CEO of Mount Sinai Health System, raised similar concerns about giving AI greater independence.

“There’s a new member of the team,” Carr said. “That member has incredible, incredible promise. But isn’t quite ready to be free range yet.”

Amid calls for government regulation to slow AI development, Carr argued that health systems could serve as “assurance partners” to technology companies, helping ensure increasingly autonomous systems are deployed safely.

Anderson also described a “standoff” between some health systems and frontier AI developers over who should control autonomous agents and bear responsibility when something goes wrong.

“When something goes bad, who's the responsible party right now… it's the health system.”

View 0 Comments

Request Reprint & Licensing

Submit Correction

View Editorial & AI Guidelines