Can Governance-by-Design Make AI Compliant in Pharma? (Part II)

Most conversations about artificial intelligence in regulated environments begin with hallucinations and content accuracy. Those concerns are real, but after more than a decade across GxP learning, quality systems, manufacturing, and research, I do not think they are the defining challenge. Documents are only the visible output of a regulated process. The harder problem begins after a document is approved. Someone still has to determine applicability, identify affected roles, update curricula, trigger qualification, verify competency, revise assessments, authorize execution, and assemble evidence that will withstand inspection.

AI may reduce the time required to draft an SOP from days to minutes. That sounds transformative until the document reaches the same approval queues, training matrices, disconnected systems, and people reconstructing impact by hand. Generating faster can simply feed more content into a governance model already struggling to keep up.

In Part I, I argued that the industry's biggest AI problem is not generation but governance. That left a more interesting question unanswered. If governance is the real constraint, what would an inspection-ready architecture actually look like? Rather than evaluating another vendor, I started from a blank sheet of paper and worked forward. The answer, as far as I can see, has surprisingly little to do with generation and almost everything to do with governance.

The Document Was Never the Hard Part

Across regulated organizations, I have repeatedly seen the same pattern after deviations, corrective actions and preventive actions (CAPAs), protocol amendments,¹ process improvements, and SOP revisions. Teams rarely struggle to produce the revised document; what they struggle to determine is operational consequence. Which manufacturing suites are affected? Which analysts or operators require retraining? Which qualifications remain valid? Which assessments must change? Which controlled documents reference the revised process? Which sites are in scope, and who has the authority to decide?

Those questions are often answered through spreadsheets, email chains, meetings, and institutional memory. A training administrator exports a learning management system report. A manager reviews a roster. Quality weighs in after someone notices an exception. Contractors, temporary staff, and recent role changes become edge cases. The decision may be correct, but its reasoning is rarely preserved as an inspectable chain from the originating change to the resulting readiness state.

Take a change to an aseptic intervention in a Grade A environment. The SOP revision may be approved quickly. From there, the downstream work starts almost from scratch. Someone identifies the affected operators. Someone else decides whether prior qualification remains acceptable. An assessment may need to change, assuming anyone remembers that it exists. A related work instruction may sit outside the immediate change package. Weeks later, the company has a controlled document and a collection of completed assignments, but that is not necessarily the same thing as a controlled operation.

During an inspection, the difference becomes obvious. The question is not merely whether the current SOP is approved. It is why these people were trained, why those people were not, what competency was required, who approved the determination, and what evidence shows the change was operationally absorbed before work continued. If the answer requires interviewing three experienced employees and searching old emails, that is a records problem, not a governance one.

What Governance-by-Design Would Require

A governance-first architecture would begin with authoritative sources rather than prompts. Regulations, SOPs, validation documentation, protocols, job roles, curricula, qualification requirements, quality policies, and organizational responsibilities would form the system's governing knowledge. Those sources would be translated into explicit rules that define applicability, required approvals, competency expectations, and evidence obligations. The AI would operate inside those boundaries rather than inventing them after the fact.

That distinction matters. A generative model can propose language. It should not silently decide which site is affected, whether an operator remains qualified, or whether a change can be released without additional evidence. Those are regulated determinations. The system can support them, but the governing logic has to be visible, reviewable, and owned by the organization.

The centerpiece would be an applicability engine. Every approved change would first be evaluated for operational consequence. The system would propose which sites, functions, roles, procedures, qualifications, learning objects, and assessments are affected, and it would show the sources and rules behind that proposal. A human reviewer would approve or modify the scope before downstream work begins. Only then would the system generate revised documents, targeted training, assessments, qualification tasks, approvals, and an evidence package linking the change to the resulting readiness decision.

It is important to recognize that this is not a vision of a single AI model making autonomous regulatory decisions. It is a governed workflow. Deterministic business rules, authoritative sources, and organizational policy establish the boundaries within which AI operates. When a change falls outside those boundaries because it is novel, ambiguous, or carries significant regulatory risk, authority intentionally returns to qualified human reviewers rather than asking the model to manufacture certainty. In that sense, the system's intelligence comes as much from knowing when not to automate as from automation itself. The architecture should therefore be viewed less as an AI system that performs governance and more as a governance engine that happens to use AI. Figure 1 maps how these elements relate: AI executing deterministic, rules-based work, assisting with coordination and evidence generation, while regulated judgment and accountability remain with qualified personnel.

That is the human-in-the-loop position that makes sense to me. The reviewer should sit where scope and consequence are determined, with enough context to challenge the machine before its answer propagates.^2,3 A confidently wrong applicability decision is more dangerous than a spreadsheet everyone knows not to trust. Governance-by-design has to make the reasoning easier to inspect, not merely the output easier to approve.

Completion Is Not Competency

The training question exposes another weakness in the current model. Quality organizations often respond to observations by assigning more training to more people. Retraining is visible, measurable, easy to document, and creates the appearance of action. But adding assignments does not correct the capability that contributed to the event.

A read-and-understand record proves that a person acknowledged material. It does not prove that the person can execute an aseptic manipulation, recognize an abnormal condition, perform a calculation correctly, or make the right escalation decision under pressure. Annual GMP refreshers may reinforce broad expectations, but they are weak evidence for a specific operational gap. Completion records that an assignment happened. Competency shows the work can actually be performed, correctly and consistently.

An effective AI system should therefore do more than accelerate assignment generation. It should connect the governing change to the competency actually created or modified by that change. In some cases, a targeted knowledge check may be sufficient. In others, the requirement may be an observed demonstration, a supervised execution, a requalification event, or no training at all. Blanket retraining is often administratively safe and operationally lazy. A governed system should make a narrower decision possible and preserve why that decision was justified.

Validation Is About Boundaries, Not Intelligence

Here is where AI discussions often become vague. A vendor says the platform is compliant, or that a human remains in control, as though either statement settles the matter. It does not. Part 11⁴ and GAMP 5⁵ are not badges a product earns in the abstract. Compliance is demonstrated through intended use, risk assessment, requirements, controls, traceability, testing, security, change management, and evidence inside the customer's quality system.

AI does not remove that burden, it changes what gets examined. If the intended use is limited to drafting nonbinding language for human review, the risk profile is one thing. If the system scopes affected roles, recommends qualification actions, or determines whether a site is operationally ready, the risk is materially different. The validation strategy has to follow the regulated function, not the marketing description.

For a governance engine, grammatical quality would be the least interesting test. The harder questions would be whether it applies approved rules consistently, identifies affected populations accurately, handles conflicting sources, exposes uncertainty, routes exceptions correctly, preserves version history, and prevents unapproved outputs from becoming operational instructions. Intelligence is not the validation criterion. Predictable, controlled behavior within a defined intended use is.

The Problem of Governance Drift

There is another failure mode that deserves more attention: governance drift. Organizations do not remain still after a system is configured. Employees transfer, departments reorganize, protocols and suppliers change, new equipment gets introduced, and temporary procedures become permanent. Training curricula are revised locally while job roles are maintained somewhere else. Over time, the documented operating model and the real operating model begin to separate.

I have seen organizations remove tens of thousands of unnecessary training line items by challenging assumptions embedded in their matrices. The assignments had accumulated one revision, CAPA, and cautious decision at a time. Each addition looked defensible alone; together they created noise, obscured the training that mattered, and made maintenance harder without producing an equivalent improvement in readiness.

A useful governance platform would not simply automate that accumulation. It would continuously reconcile roles, requirements, changes, and evidence against authoritative sources. It would flag contradictions and stale relationships rather than treating the existing matrix as truth. Maintaining alignment may ultimately be more valuable than generating another controlled document.

What Has Been Proved, and What Has Not

No publicly documented platform has demonstrated this complete architecture through repeated regulatory inspections. That matters. The industry should be skeptical of anyone presenting autonomous governance as a solved product category. A coherent diagram does not prove validation, and a convincing demonstration does not survive an inspection.

The absence of a mature commercial implementation does not make the architecture imaginary. Quality organizations already interpret governing sources, determine scope, assign authority, document approvals, and assemble evidence manually. The design question is whether those activities can become more consistent, transparent, and inspectable without transferring regulated authority to a model that cannot own it.

My answer is a qualified yes, and the qualification carries most of the weight. The system would have to be designed around constraints, provenance, human authority, and visible uncertainty from the beginning. Bolting a language model onto a system of record and asking a reviewer to catch what slips through is not governance-by-design. It is faster content production with the old controls hanging off the back.

Where Quality Leaders Should Start

You do not have to wait for a vendor to act on this argument. Map the regulated decisions that still depend on spreadsheets, email, hallway conversations, or tribal knowledge. Trace a meaningful change from approval through applicability, authorization, training, qualification, execution, and evidence. Do it the way work actually happens, not the way the SOP says it happens.

Mark every point where someone manually transfers information, interprets an ambiguous requirement, or makes a decision without a clearly named owner. Ask whether competency is demonstrated or merely recorded. Ask whether the evidence package could be reconstructed six months later without relying on the memories of the people who were in the room. Those gaps exist whether AI arrives next month or five years from now.

My conclusion remains the same as in Part I, I do not believe compliant AI begins with a language model. It begins with governance. The organizations most likely to benefit will not be the ones that generate documents fastest. They will be the ones that first define the rules, responsibilities, evidence, and human authority governing regulated work.

Generation is the easy part to demonstrate. Governance is the difficult part to prove. If AI is going to earn a durable place inside pharmaceutical quality systems, inspectors will eventually ask the same questions they ask of every other regulated process: Who approved this? Under what authority? According to which source? Who was affected? What evidence supports the decision? And can the organization reproduce the answer? A system that cannot answer those questions is not compliant AI. It is simply fast.

References

1. Getz KA, Smith Z, Botto E, et al. New benchmarks on protocol amendment practices, trends and their impact on clinical trial performance. Therapeutic Innovation & Regulatory Science. 2024;58(3):539-548. View source

2. Goddard K, Roudsari A, Wyatt JC. Automation bias: a systematic review of frequency, effect mediators, and mitigators. J Am Med Inform Assoc. 2012;19(1):121-127. View source

3. Yu F, Moehring A, Banerjee O, et al. Heterogeneity and predictors of the effects of AI assistance on radiologists. Nature Medicine. 2024;30:837-849. View source

4. US Food and Drug Administration. 21 CFR Part 11: Electronic Records; Electronic Signatures. Code of Federal Regulations, Title 21, Part 11. View source

5. International Society for Pharmaceutical Engineering. GAMP 5: A Risk-Based Approach to Compliant GxP Computerized Systems. 2nd ed. ISPE; 2022. View source