Shane Ernst

Regulatory, consumer, and political organizations continually question the pharmaceutical industry concerning the quality and cost of the medicine that it manufactures. Contract manufacturers experience even further pressure as existing and potential customers weigh the quality and cost of manufacturing their products themselves 

 outsourcing them. This pressure was increased by a recent study, 

Pharmaceutical Manufacturing Research Project–Final Benchmarking Report

, conducted by the McDonough School of Business at Georgetown University and the Olin School of Business at Washington University in St. Louis (1). While the summary report contains useful observations for contract manufacturers to consider such as linking electronic deviation-management systems to superior performance metrics, the study also concluded that contract manufacturing correlates with inferior performance metrics.   

While the report's conclusion raises significant questions about contract manufacturers and their performance, contract manufacturers also may have questions about the process used and data collected to arrive at this conclusion Such questions arising from the report include:  					

Were they adequately defined and did they meaningfully compare similar processes as they relate to contract manufacturing? 

What was the study's scope and how did it influence and generate these benchmarks for contract manufacturers and their product-owner brethren?

Was this scope adequate to draw such a conclusion?  

While the enormous amount of data and statistical analysis in the report may suggest pathways for further study, it seems premature to draw such a broad conclusion regarding the contract-manufacturing industry based solely upon this data and analysis. This article examines the performance metrics used in the benchmarking study by outlining several concerns with the context used to link contract manufacturing to these performance metrics. This article also reviews key operational strategies contract manufacturers should glean from this study and suggests a framework for continual improvement and efficiency. 

Study design, participation, and data pooling 

A brief review of the study's design and the level of participation from the contract-manufacturing industry and those firms that manufacture their own products is necessary to begin to understand the data and analysis upon which the study's conclusions on contract manufacturing is based. The four-year study statistically analyzed survey data collected from 42 pharmaceutical manufacturing facilities, focusing on selected performance indicators such as batches failed, cycle time, yield, and deviation management. The study used a survey instrument to gather a large amount of data for each facility regarding general company and business-unit information, manufacturing facilities, human resources, product development, performance metrics, teams, deviation management, and supplement management. Study participants were asked to select and provide performance data for up to five representative products that were manufactured in their facilities between 1999 and 2003. Participants were requested to preferably select large-volume products that had been introduced within the past five years and had at least three years worth of data. Batches failed, cycle time, yield, and deviations were selected as representative key performance indicators.The study attempted to statistically demonstrate relationships between these performance indicators and numerous independent variables spanning from facility size to batch disposition authority.  

 The researchers asked each facility to complete a survey, and each facility was asked whether it was a contract manufacturer. According to the study, the 42 manufacturing facilities that participated were actually owned by only 19 companies. This distinction between company and facility is important because it suggests that a company could manufacture its own products and provide contract-manufacturing services. A total of eight injectable facilities participated in the study, of which only two were engaged in contract manufacturing. Twenty-two oral and topical facilities participated in the study: 13 facilities were engaged in contract manufacturing. Given that 50% of the 30 facilities were engaged in contract manufacturing, it seems probable that some of the 19 companies that participated in the study both manufacture their own products and provide contract- manufacturing services. This duality may present a potential concern because companies that manufacture their own products and provide contract-manufacturing services may operate differently than those companies that are solely dedicated and designed to provide contract-manufacturing services. Unfortunately, the study did not account for this distinction. Despite clear differences in processes, the data from oral, topical,  and injectable manufacturers was pooled as one set for analysis and discussion, further complicating the data pooling. This pooling becomes problematic for key performance metrics such as cycle time and yield. 

 as "the number of batches per month that failed and were not able to be successfully reworked" (1). This definition, however, does not account for the monthly volume of production for the product in question. Although 

 was one of the performance measures described in the study and requested by the survey, it does not appear to have been used for this metric.  

To understand why this omission is important, consider the following example. Assume Facility A and Facility B both make Product X. Facility A manufactures approximately 10 batches of Product X each month on average. Facility B manufactures approximately 100 batches of Product X each month on average.  If both facilities reject 1% of their monthly production of Product X on average, how do these facilities compare in terms of batches failed? Facility A has a batches-failed metric of 0.1 batches per month. Facility B has a batches-failed metric of 1.0 batches per month. According to the study, the mean for batches failed for oral, topical, and injectable facilities was 0.13 batches failed per month, but this does not consider the number of batches manufactured. Using this example, Facility A performed slightly better than the mean for Product X, and Facility B performed poorly and well above the mean for Product X. Both facilities, however, rejected an equal percentage of their monthly production. 

 What is the relevance of this point for the contract manufacturers that participated in this study? To address that question, it is important to examine the size and scope of those participating contract-manufacturing facilities. For injectable manufacturing facilities, the two contract-manufacturing facilities both were fairly large. For example, one contract-manufacturing facility ranked first with the largest average facility size, highest number of compounds, and largest average number of employees and operating hours. Oral and topical contract manufacturers generally were large as well. Five of the nine facilities that were above an average of 22,000 m

  from 1999 to 2003 were contract manufacturers, one of which was ranked the largest of all facilities that provided data for the study. A contract manufacturer also ranked first in terms of average total facility employees. In fact, three of the four facilities reporting over 1000 total facility employees were contract manufacturers. Contract manufacturers held five of the top six rankings in terms of manufacturing-facility operator hours per month, averaging over 600 hours per month from 1999 to 2003. Additionally, contract manufacturers held the top four rankings of the 18 facilities that reported data concerning the number of compounds manufactured.   

Not surprisingly, the top three contract manufacturers had a combined total of as much as 200 compounds manufactured. A large number of compounds, employees, and operating hours using a large facility suggests a large monthly production volume that should be accounted for in benchmarking batch failure. The need to account for monthly production volume for the product under analysis becomes even more evident when considering that the study reported a large range—0 to 19—of batches failed per month. While using an absolute metric such as batches-failed-per-month may provide an indication of overall product unavailability and waste, it does not provide the best comparative measure for a facility's ability to manufacture a product in such a way that will ultimately lead to its release.        

 as "the average number of days between batch start and those batches either accepted or rejected during the month" (1). It reports the mean cycle time for all oral, topical, and injectable facilities as 27.14 days. Although this may seem like a reasonable standard of comparison for benchmarking both contract manufacturers and product-owners alike, two points should be considered.   

First, in order to compare facilities that manufacture their own products with contract-manufacturing operations, it must be assumed that the cycle processes that each facility follows are equivalent; however, this situation often is not the case for contract manufacturers. Because the product's license holder assumes the ultimate legal responsibility for product release to the market, product owners often review the associated batch documentation and provide release and authorization for shipment to the contract manufacturer. This additional step can take days or even weeks, depending on the prioritization and schedule of the product's owner.  This difference does not appear to have been accounted for in the study.   

Second, this study pools cycle times stemming from many different processes that may have inherent cycle-time limitations. For example, an injectable product typically requires sterility testing as part of its release criteria. Samples undergoing this test have a fixed incubation time that must be built into any cycle-time expectations. Nonsterile processes such as those governing oral and topical products do not require this time-consuming test. Compressing a batch of tablets may take hours; some freeze-drying cycles for sterile products can take nearly a week. This point is made clear by examining the standard deviation and the range of cycle times given for the study. The standard deviation was given as 28.76 with a range of 1.00–187.00 days. For a benchmarking study, the need for large pools of data to yield statistically significant conclusions makes sense; however, this need must be weighed against the similarity of the data and its ability to be pooled. In this case, given the large variation among the processes and the enormous variability within the data, the presented pooling does not appear to be nearly as useful as it might seem. 

 is defined as "the ratio of the theoretical amount of output that would be produced at any appropriate phase of production of a particular drug product to the quantity of components used, in the absence of any loss or error in production, stated as a percentage" (1). The 

then is the ratio of the actual yield to the theoretical yield at the same phase of production. The mean actual yield for this study was reported as 96.38%. While the mean itself could be recognized by pharmaceutical facilities across the industry as a reasonable expectation, the standard deviation of 6.64% with a range of 24.00–124.00% suggests otherwise.    

The fact that this study allows and pools yields taken from any appropriate phase of production is somewhat surprising given the number of potential production phases in oral, topical, and injectable operations. Granulation, tableting, encapsulation, film-coating, sorting, liquid manufacturing after filtration, semisolid manufacturing completion, and filling and packaging operations are among the many phases where a calculation of yield may be both appropriate and necessary.  The report does not appear to categorize survey data according to the yields reported in phases; instead, the data is pooled into the overall assessment. Should a value that measures the amount of sterile bulk filled against the quantity that was manufactured be compared with a value representing the number of compressed tablets arising from the amount of powdered granulation manufactured? Would the expectations be the same? A review of the standard deviation and range suggests that in many cases, similar processes are not being compared. Again, the need for meaningful comparisons appears to have been sacrificed to generate a statistically significant data pool upon which to base conclusions. 

Deviations were divided into three categories: raw material, product component, and product/process specification. This metric reports the number of each type of deviation that occurred for the product in question per month.  As with batches-failed, this metric does not account for the product's production volume.  Although this metric provides an indication of the amount of time wasted by addressing each type of deviation, it does not provide the best comparative measure for a facility's ability to manufacture a product in such a way as to avoid deviations. The metric, however, does look at how the number of deviations of each type changes over time. With regard to contract manufacturing, the study concluded, "OT&I [oral, topical, and injectables] contract manufacturing has no impact on production unavailability or the level of deviations. It does lead to lower product and process specification deviations over time" (1).  

The study did an effective job selecting key representative performance metrics and listing the numerous variables that may affect them. Taken as a whole, these metrics suggest some strategic guidelines that all contract manufacturers should follow. 

The study shows that contract manufacturers were among those facilities with the greatest number of compounds and processes. Each new compound that a contract manufacturer introduces provides an opportunity to expand its library of pharmaceutical knowledge. When captured and used properly, this knowledge may lead to improved technical transfers, reduced deviation rates, fewer batch failures, improved yields, and improved cycle times.  Contract manufacturers must have systems in place that identify, collect, and assess all necessary information regarding the processes they use to manufacture their customers'products and the resulting outcomes. 

 To fully capitalize on manufacturing experience, contract manufacturers should define, track, and trend metrics for key performance indicators such as those considered for this study. Management must understand what these metrics are intended to measure. For example, everyone would agree that tracking a measurement for batch failure is of critical importance.  

Of equal importance, however, is defining which aspect of batch failure to focus on. For example, is a company interested in what percentage of its total production is rejected? This metric may indicate how well overall operations are performing. Perhaps a company wants to know what percentage of a newly introduced product fails. This metric may give an indication of the technical transfer success. A company may even be interested in how much cost in time, materials, and opportunities is lost because of batch failures. Each of the measurements is valuable, and each would be calculated and presented differently. Contract manufacturers should make clear to their entire value chain what is being measured, how it is being measured, and why it is being measured. These metrics must be tailored to fit the existing and potential customer's needs and concerns. Contract manufacturing is a service business, and the metrics are only meaningful if the customer can understand them in terms of their own operations.  Metrics can be a tool that not only serves to suggest areas where operations can be improved, but also may be used to help convince potential customers to outsource a similar product type. 

Employ electronic deviation-management systems.

 Contract manufacturers should use electronic deviation-management systems. The study looked at how companies employed information technology systems to track and manage their deviation process. Specifically, the study questioned whether the deviation-management system used had the capability to: 					

track people assigned to resolving the deviation;

Contract manufacturers were among the first to adopt these systems. For orals and topicals, seven of the 10 contract manufacturers had such a system, and four of the seven have had their system in use since 1999. The study links the routine usage of such a system to improvements in cycle times, batches failed, and deviations. These systems allow deviations to be easily categorized, tracked, and, most importantly, trended.  Once a trend is identified, corrective and preventative actions (CAPA) can be initiated, and future occurrences of similar deviations can be avoided. Once this CAPA has been implemented, repeat occurrences of deviations will disappear,  potentially leading to an overall decrease in the number of deviations, as the contract-manufacturing data from the study observed. Fewer deviations will lead to fewer failed batches. It also may result in a decrease in the overall cycle time from not having to waste days performing time-consuming and costly investigations. 

 conducted by the McDonough School of Business at Georgetown University and the Olin School of Business of Washington University in St. Louis (1) is useful for contract manufacturers because it accurately identifies the performance metrics most associated with quality and cost. Few pharmaceutical manufacturers would disagree that the need to avoid deviations and failed batches drives quality, and that reducing cycle time while optimizing yield affects cost. The study further offers a multitude of variables that may affect these parameters. Unfortunately, when considering how to define these key performance metrics, the study fails to take production volume into account and thus fails to assess each facility's ability to manufacture a batch without deviation or need for rejection.  

The study also fails to recognize the need to further categorize the data before pooling, and therefore the data lose any meaningful comparison. As a result, the study's conclusion that contract-manufacturing performance corresponds with inferior performance metrics can't be supported because of the limitations of the data. Despite these limitations, the study does illuminate several strategic operating points for contract manufacturers to consider for optimizing their operations and improving both quality and cost.  

First, the sheer number of variables considered in the study serves as a reminder of the importance of having meaningful metrics. A variable cannot be improved unless it can be measured, but the variable should not be measured until it is understood what is being measured and why it is being measured.   

Second, the data suggesting that the contract-manufacturing facilities that participated in the study were generally large in size and scope can be an enormous strength for contract manufacturers if they have systems in place to ensure they capitalize on every experience.  

Lastly, to reduce the number of deviations and batch failures, contract manufacturers must identify the root cause and take the appropriate corrective and preventative actions.   The study accurately linked the use of electronic deviation-management systems with success in that endeavor. Contract manufacturers should employ electronic deviation-management systems to make trending, correction, and prevention easier. Supported by a strong, metric-driven trending program stemming from an electronic deviation-management system and robust processes that capitalize on every experience, contract manufacturers should have little trouble demonstrating superior performance to potential customers. 

Shane Ernst* is the quality assurance director at DSM Pharmaceutical Products, Inc., PO Box 1887, 5900 NW Greenville Blvd., Greenville, NC 27835-1887, tel. 252.707.2327, fax 252.707.7512, 

				*To whom all correspondence should be addressed.			

 1.  J. Macher and J. Nickerson, "Pharmaceutical Manufacturing Research Project–Final Benchmarking Report," McDonough School of Business, Georgetown University, Washington, DC and Olin School of Business, Washington University in St. Louis, St. Louis, MO, Sept. 2006, 

www.olin.wustl.edu/faculty/nickerson/results/

Articles

The author analyzes the study design, participation, and data pooling from a recent benchmarking study on pharmaceutical manufacturing and raises concerns about the process used and data collected in evaluating the performance of contract manufacturers.