Michael Pfeiffer, PhD is global quality manager at Boehringer Ingelheim Pharma GmbH&Co. KG, Binger StraÃe 173, 55216Ingelheim a. Rh., Germany; Tel.: +49 (0) 6132-777013;michael.pfeiffer@boehringer-ingelheim.com.

Michael Pfeiffer, PhD

 (to hear), the term “audit” dates back to the Middle Ages, when independent examiners would have a company’s bookkeepers read account books out loud to them, to ensure that no theft or sloppy accounting was happening.

	While the term can strike fear into the unprepared, the audit can be a huge benefit to any company, allowing for a systematic, independent examination of processes. In the pharmaceutical industry today, quality systems are usually audited by a trained and knowledgeable team. The audit’s outcome should always be a written report. 
	In a recent research project, audit reports written by in-house and independent auditors were randomly chosen for two five-year periods (from 2002 through 2006 and 2012 through 2016). The number of observations made, the severity of those observations (i.e., whether deficiencies were “critical,” “major,” or “minor”), and the auditors’ recommendations were assessed and results were then compared.

	Findings showed that the observations of the earlier audits were more critical than those made a decade later.

	One possible reason could be that suppliers have optimized their processes and have become more accustomed to hosting on-site audits.

	However, the earlier audits lasted longer than the later ones (as measured by audit person-days, derived by multiplying the number of days required for the audit by the number of auditors performing it). Fewer days available for on-site audits means that each auditor has less time to dive deeper into more complex processes. The result is fewer, and possibly more superficial, observations. As a result, the audit reports may be less meaningful, and of less benefit to both the audited company and its pharmaceutical industry customer. 
	 

	Reports from audits of suppliers and service providers (see 

) were randomly chosen and compared. The total number of reports reviewed was 129-39 reports for the first five-year period (2002–2006) and 90 reports for the second five-year period (2012–2016), which included audits performed by third-party audit providers. The numbers of observations (including recommendations) and their severity, whether “critical,” “major,” or “minor” within both time periods were compared and the differences obtained were highlighted. 

	For the second five-year period (2012–2016), more than 40% of the reports from on-site audits issued by third-party audit providers were available. Such reports were not available during the earlier period, however. In addition, there were questions of whether there were differences between the audits made by inhouse auditors and those done by independent third parties, in terms of the quantity and criticality of observations.  

, the mean number of audit observations decreased by approximately 38% from the earlier to the later period, with 11 observations made per audit between 2002 and 2006, and eight between 2012 and 2016.

	When comparing in-house versus outside auditor results, the number of observations was more or less comparable in both cases. To calculate the mean “severity” of observations, the total number of “critical” observations was multiplied by a factor of 5; “major” observations were multiplied by a factor of 2, and “minor” observations, by 0.5.  The number of recommendations issued was multiplied by a factor of 0.1 (1). The resulting figures were added, and the resulting sum was divided by the total number of audits executed. Results showed that the number and severity of observations fell from 6.5 to 5.85 during the 10 years separating the two audit periods (see 

	What led to these differences? During the earlier period, two out of three audits lasted approximately one day and one out of three took longer than one day (see 

). In contrast, more than 90% of the audits performed during the later period lasted one day, with the balance lasting more than one day.

	When audit person-days were compared within these two time periods, the differences were even more striking (see 

). During the earlier period, more than 90% of the audits lasted more than one person-day, whereas only 78% of all audits conducted during the later period lasted more than one person-day.

	Correlating the reduction in audit person-days with the decrease in number of audit findings and lower criticality of observations made: The average audit duration was 2.4 audit person-days in 2002-2006 and 1.9 audit person-days in 2012–2016 (down by 26%), and this reduction in audit duration resulted in 38% fewer observations and an 11% drop in critical observations (see 

	On-site audits are part of the supplier qualification process. Although supplier qualification is more than on-site auditing, 

planned audits should be of sufficient duration 

(2) if they are to provide an appropriate level of confidence that suppliers and vendors are able to supply consistent quality of materials, components, and services to customers in accordance with regulatory requirements.

	How long should an audit last? Suggested lengths range from two to eight person-days (3, 4) up to 18 person-days, depending on the supplier’s size and the complexity of its products and manufacturing processes (5). Most sources refer to the duration of regulatory inspections, for which official recommendations exist.

	The recommended duration of supplier audits by customers is not stated, but some sources refer to “narrow time schedules” for customer audits (6) without quantifying what “narrow” really means. Narrow audit schedules might influence audit quality, because the number of observations will decrease due to restricted time frames. 

	Because the auditor does not have sufficient time to audit complex processes intensively, weak points that might negatively affect product quality and consequently harm patients may not be detected. In such cases, superficial audit findings, which are often only formal deviations from GxP requirements, would not reflect severe systemic quality problems.

	The gap between expectations and reality

	Ironically, between the earlier and later periods studied, regulatory requirements have increased audit requirements, mandating not only that customer audits identify any and all of the supplier’s GxP deficiencies, but that the entire supply chain be fully mapped out to ensure integrity and transparency (7). 

	The gap between regulatory expectations and audit reality is growing, with regulators calling for greater stringency and suppliers becoming less willing to host on-site audits for more than one day. The result has been less meaningful audits with less findings, and thus, fewer suggestions on how suppliers might improve (see 

). Such reports are more surrogates than meaningful reports, yet regulators rely on them when they inspect manufacturers’ supplier qualification records and quality systems (8). 

	The number of findings made during the 2012-2016 period has decreased by more than one third compared to that during the 2002–2006 period. The severity of findings has also decreased, by approximately 11% (because audit reports during the earlier period did not always distinguish between “major” and “minor” deviations, a percentage comparison of observations had to be estimated for that period). Between the two periods being studied, the percentage of critical observations dropped from more than 5% to 1% (see 

	Experience has shown that suppliers have become less willing to host on-site customer audits, as shown in 

, which provides supplier responses to customer requests for on-site audit appointments. This leaves pharmaceutical manufacturers in a difficult position, because regulators are demanding more stringent audits at a time when suppliers are less willing to host them. So what can customers do? 

One thing that is of vital importance is to plan and prepare very carefully for the audit, so that the exercise is benefitial for both the auditor and the auditee. In this context, it is essential to be aware of the audit “environment” (e.g., whether GxP or ISO), and whether the audit focuses more on systems or on products. Suppliers are more willing to devote time to on-site customer audits for larger customers, so it might be helpful to consolidate business with strategic suppliers, rather than divide work among many smaller suppliers. In addition, it might be worth working with a third-party audit provider that audits supplier facilities for more than one customer who buys the same product or service (see 

 shows that the results of audits performed by a company’s own auditors are comparable with those made by third parties, based on the number of observations made (in both cases, approximately eight observations). The criticality of observations made by the two groups is only slightly different: 6.04 compared to 5.58 when calculated according to reference 1. However, as seen in 

, the percentage distribution of observations made by third-party auditors compared to in-house auditors did not differ from each other; therefore, it can be said that  the results of on-site audits being executed by either party are more or less comparable. 

	The fact that more than 40% of audits in the earlier period were carried out by independent third parties had no influence on audit quantity and quality.  What seemed to have the most impact was the fact that suppliers are less willing to grant, or devote sufficient time to, onsite audits by their pharmaceutical industry customers.

	Suppliers and service providers today are less willing to host onsite customer audits for more than one day than they were ten years ago. At the same time, however, global regulatory authorities expect thorough on-site audits as part of the supplier qualification process, given the increasing emphasis on supply-chain transparency and integrity. The unwillingness of suppliers/service providers to host audits longer than one person-day results in audits with fewer, less critical observations, compared to audits performed 10 to 15 years ago.

	If audit reports are to be meaningful, customers should make the audit as beneficial as possible for their suppliers, while ensuring that it meets regulator’s expectations. The following strategies (see 

		Well-prepared, risk-based audits that makes the experience a “win/win” for both customer and supplier 

		Qualification of strategic suppliers and service providers

		Increased use of third-party auditors, who audit suppliers for several customers at a time, reducing the number of audits required.

 75 (12), 1924-1928 (2013).
	2. European Commission, 

, Volume 4, Good Manufacturing Practice Guidelines

, Chapter 5 (Production).
	3. Certqua, Newsletter 10/2013, Wichtige Fristen für die Re-zertifizierung
	4. C. Johner, 

 .
	5. ISO/IEC 27006:2015, Requirements for Bodies Providing Audits, DAkkS, Deutsche Akkreditierungsstelle, Übergang, as of Apr. 11, 2016 (revision 1).
	6. M. Hiob, 

, Logfile No. 33/2016 .
	7. E. Norton, “

,” MHRA Inspectorate, Feb. 27, 2017.
	8. GMP News, 

Welche externen Auditberichte darf man verwenden


	Supplement: Outsourcing Resources
	Vol. 41
	August 2017
	Pages: s10-s15

	When referring to this article, please cite it as M. Pfeiffer, "Trends in On-Site Supplier Audits," 

Articles

Research suggests that on-site audits are becoming less meaningful because suppliers are less willing to devote time to them.

Trends in On-Site Supplier Audits

Audits of suppliers and service providers are becoming more important in a globalized world. Well-trained auditors will be able to detect potential weak points of the firms that they audit, before such weak points lead to quality problems in routine business with severe consequences (e.g., significant additional costs). 
	
	Thus, supplier audits and qualification have become one of the most important elements in quality assurance and are in the focus of authority inspections. The British Medicines and Healthcare Product Regulatory Agency (MHRA) has published an 

 of the most important GMP deviations within a time span of 12 months, and “poor supplier qualification” was ranked sixth in this overview (1). 

	In principle, a pharmaceutical manufacturer may have an audit as part of supplier qualification, executed by its own employees; or can authorize a suitable third-party audit provider to perform the audit on its behalf; or can buy an audit report from a commercial provider, for example, the Asociación Forum Auditorías (AFA). 

	The main focus of interest lies on the final audit report, regardless of whether the audit has been performed by the pharmaceutical manufacturer itself, by a third party, or whether the report has been purchased from a provider. In any case, the report should be written informatively and distinctly, and should enable the reader to gain a comprehensive impression of the audited firm. This article reports some experiences with the outcome of in-house audits, audits by third parties, and purchased audit reports.

	Within a period of three years, 2012-2015, a total of 126 reports from audits executed by company employees, by third-party audit providers, by “others,” and reports from shared audits bought from commercial providers have been assessed. For the assessment, first a “scale” was issued (see 

), based on Kettelhoit’s description of the desired format and content of an audit report (2). 

	The number of achieved points per audit report was divided by the maximum number of achievable points, and the resulting quotient was then multiplied by 100 to achieve the percentage rate, which stands for the report quality. When the final percentage rate was below 50%, the report was regarded to have “room for optimization.” A report was considered “acceptable” if it scored up to 65% and “satisfactory” for scores of up to 80%. In cases of more than 80%, the report was deemed to be very good.  

 indicates the number of audit reports in each category.

	The quantity and quality of the issues (observations) that have been made is another criterion to be considered when deciding whether to share or to buy an audit report rather than executing an audit. For classification, four grades were used: “critical” (very serious observation), “major” (not critical, but serious), “minor” (observation could become a problem if not corrected [in a timely manner]), and “recommendations” (potential improvement). 
	
	

 shows that there were only a few “critical” observations in this assessment. (In the case of “others,” the relatively high rate of “critical” issues was regarded to be an outlier due to the fact that only very few audit reports from “others” were available, thus one or two “critical” issues within a small total number of issues resulted in a high percentage of critical issues). Approximately 20% were serious or “major” observations, and most observations were classified “minor” or “recommendations.”

, the observed differences of the overall results are not striking. The quantity and quality of observations made are comparable and do not differ too much from each other, with some exceptions in the case of “other” audits, which had a relatively high percentage of “critical” observations, a percentage of “major” observations above the mean value (+ one standard deviation), and only a few recommendations in comparison to the other audit reports; these were considered outliers as described perviously. 

	To have another parameter that might show differences in audit report qualities, the mean values of the individual report qualities (percentage rates) were compared with each other. In 

, the summary of the individual qualities was presented. The respective qualities of the individual reports were summarized and the mean values were calculated and differentiated between “third party,” “shared,” “own audits,” and “others.” The resulting mean values were 73.9%, 71.8%, 79.6%, and 68.1% respectively. The differences in mean audit qualities were not striking, but to see whether the differences measured were statistically relevant, the “t-test” for comparison of mean values (4) was applied (normal distribution of the individual values pre-conditioned). 

, reports from audits performed by the company’s own auditors turned out to have significantly higher report qualities in all cases, whereas the report qualities from third-party audit providers, shared audits, and others were comparable with each other (no significance difference in any case).

	The supply chain is increasingly becomimg an area of interest in authority inspections (1, 5). Supply-chain integrity also includes supplier qualification. An on-site audit is only one but nonetheless an important test of a supplier’s qualification to manage supply-chain control effectively (6). Lack of transparency and the corresponding uncertainty along the supply chain might cause severe quality risks. Often, measures that should assure quality are not sufficiently known, such as the meaning and validity of foreign certificates, quality assurance agreements, analytical controls, and on-site auditing. More suppliers refuse an individual on-site audit by the customer because of several reasons, such as “…the quantities announced do not interest us...”, “...considering that the business for which we will be audited for by you does not proceed…”, “…we have invested a lot of money in our certificates…”, among others. When in cases where an on-site audit by the customer is denied by the supplier, the customer can try to instruct a third-party audit provider to perform the audit on its behalf, or can buy an audit report from a commercial service provider who has perfomed an audit on behalf of more than one contract giver (i.e., a shared audit). 

	In the end, a company has to integrate a purchased audit report into its own quality system, and this integration preconditions some precautions (7), as the risk to collect a surrogate (i.e., an audit report of little quality) of an audit report should be avoided by all means.

	There are various preconditions that have to be laid down proactively before integrating an external audit report, either purchased or issued by a third party, into a company’s quality management system (7). The auditor who issues the report has to avoid conflicts of interest, be independent, and be qualified to execute the audit. The report issued by the auditor must contain, beside formal aspects, sufficient information of the product-specific processes and workflows. On the basis of such information, the contract giver can decide whether the supplier is sufficiently qualified or not. Moreover, it should be kept in mind that a report of an on-site audit is only one part of the entire supplier qualification. 

	To guarantee that the audit report (either issued by the company’s own auditiors, or provided by other sources) meets the qualitative requirements necessary to assess whether the supplier is sufficiently qualified, a scale (

 that the quality of reports from audits being performed by own employees was significantly better than the quality of reports being issued by third parties or being purchased from shared audits through an external audit provider (“other audits” that did not fit in either group mentioned will be left out of the discussion because the number of audits/audit reports from “others” was too small). The differences in quality might be because the scale issued was adapted to the report template for audits being performed by own auditors. To raise the calculable quality of the reports being issued by a third party, a company might consider providing the third party with its own report template; by doing so, the third-party auditor will be more inclined to fill in the template according to the format is expected. As can be derived from 

, “time aspects” are not being met by auditors others than own auditors. 

	In case of reports from audits performed by others instead of one’s own auditors, the company has to accept the format and content of the report as it is, because there might be little possibility to gain influence. 

	It can be concluded that on-site audits are an integral part of supplier qualification and management (1). Given that there might be several restrictions preventing a company’s auditors from perform their own on-site audits (e.g., expert required, cultural restrictions, personnel shortages), one has to consider obtaining audit reports from other sources-whether it’s buying audit reports from a commercial service provider, sharing audits with other companies, or using a third-party audit provider to execute an on-site audit on behalf of the company. 

 lists some examples of criteria to consider when deciding what type of audit to use.

	One has to ascertain that audits performed by other auditors are qualitatively comparable with the company’s own audits. This practice is also a requirement of official normative guidelines (8) and might be assured by, among other things, administratively integrating a commercial provider for the execution of on-site audits into one’s own quality assurance (QA) system. For example, one might request that the third-party audit provider is certified according to DIN/EN/ISO 17020 (9) or ISO 19010:2010 (10). When being certified according to an international standard, one can have the assurance that the audits performed by such a certified body follow a standardized procedure. 

	If a company fails to integrate an audit provider into its own quality system, the company might be in danger of receiving surrogates rather than valuable audit reports that will not be accepted in case of an authority inspection. It is important that requirements that are to be followed by a commercial provider of on-site audits are agreed upon prior to an audit. By doing so, it is more likely to satisfy authorities during inspections that external audits are well under control.

	On-site audits play an important role in supplier qualification. Suppliers, however, are sometimes not willing to host on-site audits by their customers for various reasons. In such cases, a customer can either:

		Buy reports from audits being executed at the suppliers’ site, 

		Authorize a commercial audit provider to conduct the required audit on its own behalf.

		Perform audits with other customers of the same supplier who will then share the issued report. 

	If the second option is preferred, one has to carefully select the audit provider and integrate the provider administratively into one’s own quality management system. Finally, it has to be assessed by the contract giver whether the quality of the audit being executed by a commercial audit provider is comparable to audits being performed by its own auditors. Quality assessment will be based on the overall quality of the issued report in percent (

), the quantity and quality of observations made (

), and the statistical comparison of the achieved data (

). The outcome of these assessment and the consequences resulting from it can be seen in 

. When such conclusions from these assessments are laid down proactively, authority inspectors might be convinced that the supplier qualification process is in order, even if parts of the qualification process have been outsourced.

, April 2011-March 2012.
	2. S. Kettelhoit, 

 72 (2) 242-247 (2010).
	3. European Commission, 

GMP Guide: EudraLex-Volume 4 (Human and Veterinary)-Good Manufacturing Practice (GMP) guidelines

, https://ec.europa.eu/health/documents/eudralex/vol-4_en, accessed 12 Dec. 2016.
	4. L. Cavalli-Sforza, Biometrie - Grundzüge biologisch-medizinischer Statistik, Gustav Fischer Verlag, Stuttgart (1974).
	5. A. Shanley, 

., 28 (7) 25-27 (2016).
	6. S. Barr, Staying Ahead in OutsourcingMeeting the Changing Needs of the Pharma Industry, 

, p. 28-29, July/August 2016. 
	7. GMP News, 1 Feb. 2016, www.gmp-navigator.com/dnews_05166_Welche-externen-GMP-Auditberichte-darf-man-verwenden.html, accessed 20 Dec. 2016. 
	8. Arzneimittel und Wirkstoffherstellungs-verordnung of 3 Nov. 2006, BGBl, Part I, p. 2523, version of 28 Oct. 2014.
	9. DIN/EN/ISO 17020: Conformity AssessmentRequirements for the Operation of Various Types of Bodies Performing Inspections (ISO/IEC 17020:2012), Beuth Verlag, Berlin.
	10. ISO 19010:2010, 

Guidelines for Auditing Management Systems

, 
	ISO copyright office, Case Postale 56, CH-210 Geneve 20. 

Pharmaceutical Technology Europe
	Outsourcing Resources 2017 Supplement

	When referring to this article, please cite it as M. Pfeiffer, "Key Considerations in Outsourced “On-Site” Audits as Part of Supplier Qualification,"

 Pharmaceutical Technology Europe, Outsourcing Resources 2017

This article reviews experiences with the outcome of in-house audits, audits by third parties, and purchased audit reports.