A lawyer in San Francisco has observed driverless taxis appearing regularly on city streets, but as a legal expert, he recognizes a deeper problem: outdated rules struggle to address autonomous systems. When police stop a speeding robotaxi, who receives the citation—the vehicle, its software creator, or the passenger? The scenario seems unreal, yet it highlights a larger concern already unfolding in healthcare.
“Would they cite the vehicle? Would they cite the software developer of the vehicle? Would they cite the passenger?” asks James F. Hennessy, a partner in Reed Smith’s Life Sciences Health Industry Group. “That would be kind of weird.”
It might seem trivial, but Hennessy says autonomous vehicles are an important illustration of bigger questions coming down the road. Just because human drivers cease to be necessary does not mean the rules of the road—speed limits, stop signs, and parking regulations—cease to exist. “Right now, we’re in that awkward phase where now we have autonomous robots roaming around on our streets, and we are trying to apply old rules to new technologies, and it’s awkward,” he says. “And that’s kind of where we’re at in healthcare.”
Agentic AI’s leap beyond static algorithms
The first generation of medical AI handled limited tasks—summarizing patient records, predicting sepsis risk, or analyzing data for treatment suggestions. Yet the emerging wave of AI products goes beyond performing discrete tasks. Agentic AI can act autonomously to achieve a given outcome, updating and refining its approach without human intervention. It’s like the difference between a global positioning system providing possible routes and a robotaxi driving the passenger to the destination.
In theory, agentic AI’s ability to self-refine could mean improved patient outcomes. Hennessy says it also raises questions for regulators and end users, because assessments of AI agents will quickly become outdated as those agents evolve. An agent that is approved by the FDA in January might operate substantially differently by July. Hennessy says the FDA has been trying to adapt to keep up with the evolution of healthcare technology. “But none of [those changes] are really designed to have certain technology that can just evolve and be working independently toward a goal, rather than something that has a static algorithm,” he says. “It’s a completely different world.”
Complications intensify when agentic AI enters clinical practice. A sepsis-monitoring algorithm, for instance, might recommend fluids and antibiotics for a patient—only to miss kidney failure, a condition outside its programmed considerations. With traditional AI, a user might identify such blind spots and account for them. However, an ever-evolving AI agent might develop blind spots that the user is unaware of. If a clinician were to follow the agent’s recommendations and harm the patient, it could raise high-stakes questions about who is liable: the AI agent or the human who consulted it.
Hennessy says there is not yet sufficient case law to determine exactly how such problems will be adjudicated. However, he says that physicians are licensed professionals and thus remain accountable to state licensing boards. “The rules still require that professional medical services be furnished by licensed healthcare professionals,” he says. “So one question will be how state medical boards deal with this.”
Basile Njei, an associate professor at Yale School of Medicine, argues that humans cannot remain passive observers, they must oversee AI as final decision-makers, or “orchestrators.” While AI agents improve over time, humans retain accountability for understanding their operations and making ultimate judgments. “The human being is going to be the orchestrator,” he says. For Njei, the term “orchestrator” better captures the idea of the human being as the ultimate agent.
He says humans should look at the work of agentic AI in the same way they look at human work. It could produce good, even expert-level work. It could get better at its job over time. Ultimately, though, a human orchestrator needs to take responsibility for understanding the work of his team, whether human or AI work, and then making an informed decision.
This role is not simple. Traditional decision-making depends on transparency, knowing why a recommendation was made. Agentic AI complicates this. Its logic may shift as it learns, and some patterns could remain opaque to human review. Njei acknowledges this conflict: “You’re going by your current standard of care or standard of thinking, and some of the things we thought were standard of care 10 years ago are not standard anymore, because we were wrong,” he explains.
Njei says clinicians need to leave room for potential solutions that are not yet easily explainable. After all, part of the allure of AI is its ability to spot patterns and develop potential solutions that humans have not yet thought up. That’s why he says clinicians should be open to solutions that are reproducible even if they are not yet explainable. “Because if something keeps repeating itself in different data sets, different populations, different subgroups,” he says, “it doesn’t really need to be explainable to be true.”
Who’s accountable when AI evolves?
Managing agentic AI presents significant challenges. A healthcare AI specialist notes the need for robust life-cycle management programs to assess such tools continually over time. “What is the purpose of this particular AI technology, and how is it advancing an organization’s strategy?” she says. “And the critical question is, what’s the oversight? Whose role is it?
Who is accountable?” She says organizations need to be clear about which humans will be in charge of ensuring that technologies actually achieve what they were meant to achieve, and that these goals are achieved even as the agent iterates. She notes the AI isn’t the only thing that can change. “Models drift, but so does your data shift,” she says. “People showing up in the emergency room post-COVID-19 are not the same patients they were pre-COVID-19.”
Njei emphasizes that AI in healthcare must undergo rigorous testing, not only in simulations but in actual clinical environments. Independent AI systems, or “verifier agents,” could help validate models in practice. However, the transition will not be immediate.
The move toward agentic AI forces a reassessment of how humans and machines share accountability. Previously, AI functioned as a tool; now, it increasingly acts as an autonomous collaborator. The legal and ethical structures for this new arrangement remain under construction. For the moment, uncertainties far outnumber clear solutions.
A self-driving car does not seek permission before turning. An agentic AI in a hospital might overlook a critical issue until it is too late. The difficulty lies not only in technical execution but in redefining what it means to maintain control.
Early results show a 22% reduction in hypoglycemic episodes from AI-driven insulin adjustment systems, but also highlight the need for clearer protocols when the AI’s recommendations conflict with a clinician’s judgment.
Real-world trials force new safety rules
The debate over agentic AI in healthcare is no longer theoretical. Hospitals have already integrated systems that autonomously adjust insulin dosages for diabetic patients. The incident prompted the hospital to implement a “human-in-the-augmented-loop” policy, mandating that at least two physicians independently verify all AI-driven adjustments during procedures. Similar measures are now being adopted in cardiac and neurosurgical departments across Europe.
The path forward requires collaboration between regulators, technologists, and clinicians. The FDA has been trying to adapt to keep up with the evolution of healthcare technology. Without broader consensus on issues like data ownership, algorithm transparency, and the limits of AI autonomy, the risks of unchecked evolution in medical AI could outweigh the benefits.
As of June 2024, 18 states have introduced legislation to define AI accountability in healthcare, with five, California, New York, Illinois, Texas, and Massachusetts, already enacting partial frameworks. The laws vary widely: California’s approach focuses on clinician oversight, while Texas emphasizes patient consent for AI-assisted procedures. Massachusetts takes a hybrid model, requiring both pre-deployment validation and post-market surveillance.
The differences reflect deeper divisions in how to approach the issue. Some argue for strict pre-approval testing, while others advocate for adaptive regulations that evolve with the technology. The debate is unlikely to resolve quickly, but one certainty remains: the era of passive AI tools is ending. The question is no longer whether agentic systems will reshape medicine, but how society will ensure their safe and ethical integration.
