One of the most important questions facing regulated industries is not whether AI should be used.
It is:
How much human oversight should remain?
The easy answer is “keep a human in the loop.”
But that phrase is becoming too simplistic.
If a person receives an AI-generated recommendation and automatically clicks approve, meaningful human oversight has not occurred. Conversely, requiring a highly qualified professional to manually reperform every activity completed by AI may remove most of the value of using AI in the first place.
The better approach is risk-based human oversight.
Consider several different uses.
AI searches approved procedures and retrieves relevant documents.
AI summarises an investigation for a meeting.
AI identifies historical deviations that may be related to a new event.
AI recommends a possible root cause.
AI determines batch disposition.
These are very different activities.
They should not receive the same level of oversight.
A useful model is to consider the potential impact of an incorrect AI output on patient safety, product quality, data integrity and regulatory compliance.
For low-risk administrative support, human review may be relatively light.
For analytical recommendations, humans may need to review the evidence, assumptions and reasoning.
For critical GxP decisions, meaningful accountable human judgement may need to remain central.
This direction aligns with current regulatory thinking. FDA’s draft AI framework for drugs and biologics links credibility assessment to a clearly defined context of use and risk. U.S. Food and Drug Administration EMA likewise stresses trustworthy AI, risk management and the implications of AI across the medicinal-product lifecycle. European Medicines Agency (EMA) FDA and EMA’s joint principles now explicitly include human-centric design, risk-based approaches, clear context of use, multidisciplinary expertise, data governance and lifecycle management. U.S. Food and Drug Administration
There is another important issue: oversight must be effective, not ceremonial.
A reviewer needs sufficient information to challenge the AI.