Europe Tried to Ban Generative AI From Critical GMP. The Ban May Not Survive. It Does Not Matter.

In July 2025, European regulators published a draft annex to the European good manufacturing practice (GMP) guidelines that would keep ChatGPT-class systems out of any manufacturing decision touching patient safety, product quality, or data integrity. The annex does not restrict its use. It excludes it from scope, with the plain instruction that such models should not be used in critical GMP applications.¹

Industry pushed back hard enough that EMA is now reconsidering the boundaries of that position.²

What the Draft Actually Says

Draft Annex 22 to EudraLex Volume 4 is 6 pages long. It was written by EMA’s GMP/GDP Inspectors Working Group (GMDP IWG) in cooperation with Pharmaceutical Inspection
Co-operation Scheme (PIC/S), whose members include regulators well beyond the European Union, and the FDA and the United Kingdom’s Medicines and Healthcare products Regulatory Agency (MHRA) sat in as observers. It supplements Annex 11 rather than replacing it. Treating this as a European problem is a mistake, and manufacturers exporting into the EU are expected to meet EU GMP wherever they operate.¹

The scope section does the work. Annex 22 covers machine learning models that got their functionality from training data rather than explicit programming, and only those that are static, meaning the parameters are frozen and do not adapt during use, and deterministic, meaning identical inputs produce identical outputs. Dynamic models that continuously learn are not covered and, in the draft’s words, should not be used in critical GMP applications.

The annex states that the document “does not apply to Generative AI and large language models (LLM), and such models should not be used in critical GMP applications.” They remain available for non-critical work, where qualified personnel are responsible for confirming the output is suitable for its intended use.¹

Everything after that is machinery: intended use characterization including rare inputs and bias risks; test metrics with acceptance criteria approved before testing begins; test data that is representative, stratified, correctly labeled, and rigorously walled off from training; explainability through feature attribution; confidence scores with thresholds that force an undecided flag rather than an unreliable answer; change control, configuration control, and drift monitoring on the input sample space.¹

All of it is sound.

All of it is also built around a category of model that excludes what most people mean when they say AI.

The FDA went a different way. Its January 2025 draft guidance does not propose validating AI models at all. It proposes a risk-based credibility assessment for a defined context of use, executed in 7 steps;³ credibility, for a context, not validation, for a system.

Neither agency extended the conventional validated-state concept cleanly across a technology whose behavior, provenance, and lifecycle may not remain sufficiently stable or observable for that claim to mean what it traditionally has.

Deterministic, and Why the Distinction Matters

For anyone who has not had to care about this distinction before, it is worth being plain about it, because much of the regulatory debate rests on what evidence collected today allows you to make claims about performance tomorrow.

A deterministic system produces the same output from the same input; press the button, get the result.

Traditional computerized-system validation does not require every regulated process to be literally deterministic. Pharma validates analytical methods, biological assays, manufacturing processes, and other systems that contain variability every day. What matters is that the expected performance and relevant sources of variability can be characterized, bounded, monitored, and maintained sufficiently for the intended use.

The validated state is therefore a claim about continuity. The evidence generated during qualification and validation remains meaningful because the system operating tomorrow is sufficiently connected to the system that was tested yesterday, and change control exists to identify when that claim may no longer hold.

Generative AI makes that connection harder.

Ask an LLM the same question twice, and you can get 2 different answers because the model samples from a probability distribution over possible next tokens rather than selecting a fixed one. But stochastic output alone is not the entire problem. Pharma already controls variable processes.

The harder problem is the combination: stochastic outputs, training data you may not be able to inspect, confidence signals that do not necessarily correspond to factual correctness, limited ability to reconstruct why an output occurred, and vendor-controlled changes to an underlying model you may not own.

That is a different assurance problem.

And yet pharma already manages another highly variable component in virtually every GMP process it operates: People.

Ask me how I'm doing on Monday and I say, “living the dream.” Ask the same question Tuesday, and I might say, “my car's in the shop. I hit 49 minutes of traffic, and I have to drive to PA after work to see my in-laws.”

Same input. Completely different output.

Nobody responds to human variability by demanding that people become deterministic. GMP controls the risk another way. It defines responsibilities. It establishes procedures. It qualifies people for assigned functions. It monitors performance. It reassesses competency when conditions change. And when the decision matters, it assigns accountability to a named person with authority to make it.

Every drug approved in the past century passed through systems full of people who were having different days, carrying different stress loads, and occasionally reaching different conclusions from the same information.

Pharma did not solve that problem by eliminating variability.

It built an assurance system around it.

That distinction matters because the bind created by generative AI is not simply technical. It is epistemological. The industry keeps asking how to make a probabilistic component behave enough like conventional software to fit the existing assurance model.

The better question may be what exactly needs to be demonstrated for the human-AI system to be fit for a defined use.

Where It Gets Genuinely Hard

The industry conversation tends to stop at stochastic output, as though the difficulty were a single technical inconvenience.

It is not.

Read Annex 22 closely with a generative system in mind and 4 of its own controls become difficult to apply as written. Not because the annex is poorly drafted, but because those controls assume properties foundation models may not give you.

1. Test data independence is unprovable.

Section 6 is the most rigorous part of the draft, and rightly so. Test data must never have been used in development, training, or validation. Staff who touched the test set should not work on training the same model. There should be no copies of test data outside the controlled repository, with access logged.¹

Now apply that to a model trained on a web-scale corpus you did not assemble and cannot inspect.

You want to test it on deviation investigations, batch record review, or specification drafting. How do you demonstrate that your test cases, or documents functionally identical to them, were not in the training data?

You cannot establish that from information you do not possess.

This is contamination, and in machine learning, it is the difference between measuring generalization and measuring recall. Annex 22 asks for evidence of independence that users of externally developed foundation models may be unable to produce.

2. Explainability can become narration.

Section 8 asks that systems capture the features contributing to a classification, using techniques like SHAP (SHapley Additive exPlanations) or LIME (Local Interpretable Model-Agnostic Explanations).¹ SHAP estimates each feature’s contribution to a prediction using Shapley values, whereas LIME approximates the model locally by fitting a simpler interpretable model around a specific prediction.

Those approaches can provide useful feature-attribution information for model classes where the relationship between inputs and outputs is sufficiently tractable.

Apply the same expectation to a large transformer and the problem changes.

A model can generate a plausible account of why it produced an answer. Whether that account faithfully describes the computation responsible for the answer is a separate question.

A model asked to explain its reasoning produces more model output.

That is not automatically a faithful trace.

For a regulator asking why a specific GMP decision was made, a plausible narrative can be worse than no explanation if the narrative creates confidence that the underlying evidence does not support.

3. Confidence scores are not confidence.

Section 9 is elegant on paper. Log the confidence score for each prediction, establish a threshold, and have the model flag an outcome as undecided rather than produce an unreliable answer.¹

For a classifier calibrated against a defined distribution, that can be a meaningful control.

For generative models, token probability is not an estimate of factual correctness. A system can produce fluent, high-probability content that is entirely wrong without generating an internal signal equivalent to “I am hallucinating.”

The undecided flag depends on the system reliably knowing when it does not know.

That is precisely the capability in question.

4. The frozen state may not be yours to freeze.

Section 10 puts the model under change and configuration control, with measures to detect unauthorized changes.¹

That assumes you control the artifact.

With a vendor-hosted model behind an API, you may not. The weights can change on the provider's schedule. System prompts can change. Safety layers can change. Supporting infrastructure can change. The practical question is whether those changes are observable, versioned, and governed sufficiently for the manufacturer to determine when its existing evidence no longer applies.

A change-control trigger you cannot observe is not much of a control.

It is a hope.

And this one becomes contractual before it becomes procedural.

There is a fifth trap that applies to every model, deterministic or not. Section 4.3 says acceptance criteria should be at least as high as the performance of the process being replaced.¹

This seems straightforward, until you try to state the error rate of the human process you are automating.

Most organizations have never measured it.

That clause quietly requires a baseline almost nobody has.

The Word the Draft Never Defines

Vendor after vendor is now selling agents into pharma with systems that chain steps, call tools, and take actions rather than return a single answer.

Draft Annex 22 does not use the word ‘agent’ anywhere in its 6 pages.

That does not mean agents are unaddressed. It means they get sorted by the properties the draft does define. An agentic system built on a LLM is generative. One that adapts during use is dynamic. One that can take a different path through the same task on 2 runs is probabilistic.

Any of those properties affects where it fits under the draft and what assurance problem the manufacturer inherits.

So, the category the industry is deploying fastest is the category the draft handles largely by implication.

If you are running an agent anywhere near a batch record, the exercise is not waiting for somebody to define the word for you. It is identifying exactly what the system does, what decisions it influences, which components can change, where human review occurs, and who owns the outcome when it is wrong.

What the Consultation Actually Tells You

The consultation closed in October 2025. EMA’s own account is that stakeholders indicated support for enabling technologies such as generative AI and LLMs.²

Then EMA did something more interesting than issue a response.

On June 30 and July 1, 2026, EMA convened a 2-day workshop, open session followed by a closed drafting-group review, to gather expert input on control and mitigation measures such as guardrails, described as part of a proposed risk-based approach for manufacturers wanting to use these technologies.²

Read the framing.

The conversation has moved toward the conditions under which these technologies might be controlled. I do not think regulators build a 2-day evidence-gathering exercise around guardrails for a technology they have no intention of reconsidering, though that is my read on institutional behavior rather than anything EMA has stated.

The workshop is now complete. EMA's public record confirms that the second day was a closed review by the Annex 22 drafting group and that the Agency expects a report capturing the expert input. As of August 11, 2026, no final Annex 22 appears in EudraLex Volume 4, and PIC/S still lists Annex 22 under draft guidelines. The July 2025 exclusion, therefore, remains the published draft position. EMA's current GMDP Inspectors Working Group plan targets Q4 2026 to provide the European Commission with a final text.² ⁴ ⁵ ⁶

So here is the useful question, and it is not the one the industry is asking.

If EMA does open a pathway, what could the guardrail possibly be?

Not determinism alone, not a frozen state alone, not confidence scores, explainability, or test independence alone.

Each can contribute evidence where technically appropriate. None absorbs accountability for the final GMP decision.

What remains at that decision point is a person.

And the draft already says so.

The 2 Clauses that Already Answered It

Section 3.3 addresses the case where a model provides input to a decision made by a human operator. Where the effort to test the model has been diminished, the description of intended use should include the responsibility of the operator.

And then: the training and consistent performance of the operator should be monitored like any other manual process.

Section 10.5 completes it, noting that depending on process criticality and the level of model testing, human review may require consistent review or test of every output, according to a procedure.¹

That creates an inverse relationship.

The less assurance you can establish around the model, the more assurance you need around the human control layer.

The scope section points in the same direction: generative models remain available for non-critical applications, where qualified personnel carry responsibility for confirming output suitability.¹

EMA convened a workshop to find guardrails for a problem its own draft had already exposed.

The final control layer was never going to be purely technical.

It was always going to end with a qualified human at the decision point.

Pharma Already Has the Word for This

The vocabulary problem is doing more damage here than the technology.

The industry keeps asking whether AI can be validated.

That question is too broad to be useful.

Pharma has never limited validation to processes that produce mathematically identical results every time. A cell-based potency assay varies by nature, and nobody pretends otherwise. It is controlled through defined performance characteristics, acceptance criteria, system suitability, monitoring, and qualified execution.

Operators vary even more, and 21 CFR 211.25 has required appropriate education, training, and experience for assigned functions since long before anyone in this industry said the word model.⁷

Validation establishes evidence that a process, method, or system performs as intended under defined conditions.

Qualification establishes evidence that equipment, environments, systems, or people are fit for a defined use.

Generative AI forces those control concepts to meet in the same workflow (Table 1).

Generative AI does not fit neatly into either column in isolation.

That is the category error.

The relevant unit is the workflow.

What an Aseptic Process Simulation Actually Qualifies

There is a better model for this than anything I have seen in the AI literature, and I have spent a fair part of my career running it.

An aseptic process simulation does not test the room, and it does not test the process in isolation.

Media replaces product. Operators perform the full set of interventions the process demands. The environment runs under representative conditions. Everything happens together.

What passes or fails is the combination.

You cannot pass a media fill with a qualified room and operators who cannot execute the interventions correctly. You cannot pass it with excellent operators in an environment that cannot maintain control.

The unit under test is the process configuration and the people working inside it, together.

I have seen a media fill fail on technique rather than equipment, and there is no version of that result you can pin on the room.

Annex 1 does not treat this as a one-time event either. Personnel remain a source of variability, so assurance has to be maintained.

Apply the same correction here.

Chasing a validated-state claim for GPT-whatever in isolation misses the more useful unit of analysis.

What can be demonstrated is whether a specific person, using a specific system and version, for a specific task, under defined conditions, can produce an acceptable outcome against criteria established in advance.

That is a qualification problem.

And once you define it that way, the record must change.

The model has to fit the work

There is another mistake hiding inside the phrase “AI qualification.” It treats AI as though it were a capability rather than a category of technologies.

Large language models are unusually useful for certain kinds of work because the work itself is expressed through language. Reading scientific literature, extracting information from documents, comparing records, drafting text, summarizing evidence, and writing code all give the model patterns it was built to operate on. That does not mean the same model should be assumed competent at every scientific problem simply because the problem happens to occur inside a pharmaceutical company.

This distinction gets lost when organizations evaluate “AI” instead of evaluating a system against a task. A model that performs well at synthesizing the literature around a biological mechanism has not demonstrated that it can reason reliably about the biology itself. An agent that can retrieve prior deviation investigations and summarize them has not demonstrated that it can determine root cause. Those are different claims, and the evidence supporting one does not automatically transfer to the other.

That means qualification cannot rescue a poorly matched use case. Before asking whether a person can competently supervise an AI-enabled workflow, the organization first has to establish that the underlying system has demonstrated sufficient capability for the specific function being delegated to it.

The question therefore is not whether LLMs work in pharma. It is narrower: which part of this pharmaceutical task matches what this particular system can demonstrably do, and where does its capability end?

That boundary should be explicit. A use case is not “deviation management.” It is a defined task, intended function, and demonstrated capability boundary. “Retrieve and summarize potentially related closed deviations for human investigation” is a materially different claim from “determine probable root cause from historical deviation records.” Same model. Same records. Completely different evidentiary burden.

Qualification Is a Compound Key, Not an Attribute

This is where most AI training programs go wrong before they start.

They treat AI qualification as a status a person holds: completed the course, marked qualified, move on.

It is not an attribute.

It is a relationship among the person, system, version, and use case, bounded by predefined acceptance criteria.

A reviewer is not “AI qualified.”

A reviewer is qualified for this system, at this version, for this use case, against these criteria.

Change any component of that relationship, and the qualification has to be reassessed.

Swap the system. Change the use case. Change the person's role. Let the vendor push a material update.

What you have on file may now be evidence for a state that no longer exists (Table 2).

That record is the practical consequence of the entire argument.

Person × System × Version × Use Case.

That is the qualified state.

Not “AI trained.”

Not “AI certified.”

Not “completed generative AI course.”

A defined person demonstrated competent performance using a defined system configuration for a defined regulated task that the system itself has demonstrated sufficient capability to support.

That is a claim GMP already knows how to test.

What Does Not Dissolve

Two problems survive this reframe, and it would be dishonest to pretend otherwise.

The first is that the human must catch failures they cannot necessarily see.

An operator qualified on an aseptic intervention can observe the process they are performing. A reviewer checking a model's output is looking at a finished artifact with no necessarily faithful trace of how it was produced, assessing a system that can be confidently wrong in the same register it is confidently right.

Automation bias makes this worse rather than better.⁸

That is a harder qualification problem than most of what the industry currently asks of operators. Honest acceptance criteria therefore have to test detection of specific known failure modes, not generic “AI awareness” or an instruction to review carefully.

The following are testable:

• Can the reviewer detect a fabricated reference?
• Can they identify an unsupported causal claim in a deviation?
• Can they recognize when the model has silently exceeded its intended use?
• Can they identify a materially altered fact while working under representative time pressure?

“Understands AI risk” is not.

The second problem is that accountability does not divide.

Work can be distributed between a person and a model.

A signature cannot.

The FDA made that concrete in April 2026, giving inappropriate use of AI in pharmaceutical manufacturing its own subsection in a warning letter, citing 21 CFR 211.22(c)⁹ after AI-generated specifications and master production records went into use without adequate quality-unit review.¹⁰

The tool changed.

The signature did not move, because there is nowhere for it to go.

Vendor documentation can support the manufacturer's assurance strategy. It does not transfer GMP responsibility, and no supplier agreement has ever made a quality unit less accountable for what it approved.

The Question that Survives Either Outcome

If the exclusion holds, generative AI stays in non-critical work, where the draft already assigns responsibility to qualified personnel.

If it softens, any pathway into critical use will still have to establish whether the system is appropriate for the task, who is qualified to use it, under what conditions, for which decisions, and with what evidence, because technical controls do not absorb GMP accountability.

Both roads end in the same place.

The organizations that work this out before the final text arrives will not necessarily be the ones with the best models.

They will be the ones that recognized the unit they needed to qualify was never the model by itself.

It was the system's demonstrated capability for a bounded task, the person using it, the version in operation, and the use case that tied them together.

And pharma has been running versions of that protocol for 50 years.

It was just filed under a different name.

References

1. Annex 22: Artificial Intelligence. Consultation draft. EudraLex Volume 4. European Commission. July 7, 2025. Accessed August 7, 2026. https://health.ec.europa.eu/document/download/5f38a92d-bb8e-4264-8898-ea076e926db6_en

2. Multistakeholder workshop on expert contributions to artificial intelligence guidance development (Annex 22). European Medicines Agency. June 30 and July 1, 2026. Accessed August 7, 2026. https://www.ema.europa.eu/en/events/good-manufacturing-practice-multistakeholder-workshop-expert-contributions-artificial-intelligence-guidance-development-annex-22

3. Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products. Draft Guidance for Industry. Docket No. FDA-2024-D-4689. US Food and Drug Administration; January 2025. Accessed August 7, 2026. https://www.fda.gov/media/184830/download

4. The 3-year work plan for the Inspectors Working Group. European Medicines Agency. Current work plan accessed August 11, 2026. https://www.ema.europa.eu/en/documents/other/3-year-work-plan-inspectors-working-group_en.pdf

5. PIC/S Publications. Draft guidelines: New Annex 22 - Artificial intelligence. Pharmaceutical Inspection Co-operation Scheme. Accessed August 11, 2026. https://picscheme.org/en/publications

6. EudraLex Volume 4 - Good Manufacturing Practice (GMP) guidelines. European Commission. Accessed August 11, 2026. https://health.ec.europa.eu/medicinal-products/eudralex/eudralex-volume-4_en

7. 21 CFR 211.25. Personnel qualifications.

8. Goddard K, Roudsari A, Wyatt JC. Automation bias: a systematic review of frequency, effect mediators, and mitigators. J Am Med Inform Assoc. 2012;19(1):121-127. doi:10.1136/amiajnl-2011-000089

9. 21 CFR 211.22(c). Responsibilities of quality control unit.

10. Warning Letter 320-26-58: Purolea Cosmetics Lab. US Food and Drug Administration. April 2, 2026. Accessed August 7, 2026. https://www.fda.gov/inspections-compliance-enforcement-and-criminal-investigations/warning-letters/purolea-cosmetics-lab-722591-04022026

About the Author

Brian Drapeau is founder and principal consultant of GxP Frame, where he advises pharmaceutical and biotech manufacturers on AI system optimization, workforce qualification, governance, and ROI recovery. He is available for consulting engagements with organizations implementing or reassessing AI in regulated environments.