Regulatory, consumer, and political organizations continually question the pharmaceutical industry concerning the quality
and cost of the medicine that it manufactures. Contract manufacturers experience even further pressure as existing and potential
customers weigh the quality and cost of manufacturing their products themselves versus outsourcing them. This pressure was increased by a recent study, Pharmaceutical Manufacturing Research Project–Final Benchmarking Report, conducted by the McDonough School of Business at Georgetown University and the Olin School of Business at Washington University
in St. Louis (1). While the summary report contains useful observations for contract manufacturers to consider such as linking
electronic deviation-management systems to superior performance metrics, the study also concluded that contract manufacturing
correlates with inferior performance metrics.
While the report's conclusion raises significant questions about contract manufacturers and their performance, contract manufacturers
also may have questions about the process used and data collected to arrive at this conclusion Such questions arising from
the report include:
- What were the metrics used?
- Were they adequately defined and did they meaningfully compare similar processes as they relate to contract manufacturing?
- What was the study's scope and how did it influence and generate these benchmarks for contract manufacturers and their product-owner
- Was this scope adequate to draw such a conclusion?
While the enormous amount of data and statistical analysis in the report may suggest pathways for further study, it seems
premature to draw such a broad conclusion regarding the contract-manufacturing industry based solely upon this data and analysis.
This article examines the performance metrics used in the benchmarking study by outlining several concerns with the context
used to link contract manufacturing to these performance metrics. This article also reviews key operational strategies contract
manufacturers should glean from this study and suggests a framework for continual improvement and efficiency.
Study design, participation, and data pooling
A brief review of the study's design and the level of participation from the contract-manufacturing industry and those firms
that manufacture their own products is necessary to begin to understand the data and analysis upon which the study's conclusions
on contract manufacturing is based. The four-year study statistically analyzed survey data collected from 42 pharmaceutical
manufacturing facilities, focusing on selected performance indicators such as batches failed, cycle time, yield, and deviation
management. The study used a survey instrument to gather a large amount of data for each facility regarding general company
and business-unit information, manufacturing facilities, human resources, product development, performance metrics, teams,
deviation management, and supplement management. Study participants were asked to select and provide performance data for
up to five representative products that were manufactured in their facilities between 1999 and 2003. Participants were requested
to preferably select large-volume products that had been introduced within the past five years and had at least three years
worth of data. Batches failed, cycle time, yield, and deviations were selected as representative key performance indicators.The
study attempted to statistically demonstrate relationships between these performance indicators and numerous independent variables
spanning from facility size to batch disposition authority.
Survey sample. The researchers asked each facility to complete a survey, and each facility was asked whether it was a contract manufacturer.
According to the study, the 42 manufacturing facilities that participated were actually owned by only 19 companies. This distinction
between company and facility is important because it suggests that a company could manufacture its own products and provide
contract-manufacturing services. A total of eight injectable facilities participated in the study, of which only two were
engaged in contract manufacturing. Twenty-two oral and topical facilities participated in the study: 13 facilities were engaged
in contract manufacturing. Given that 50% of the 30 facilities were engaged in contract manufacturing, it seems probable that
some of the 19 companies that participated in the study both manufacture their own products and provide contract- manufacturing
services. This duality may present a potential concern because companies that manufacture their own products and provide contract-manufacturing
services may operate differently than those companies that are solely dedicated and designed to provide contract-manufacturing
services. Unfortunately, the study did not account for this distinction. Despite clear differences in processes, the data
from oral, topical, and injectable manufacturers was pooled as one set for analysis and discussion, further complicating
the data pooling. This pooling becomes problematic for key performance metrics such as cycle time and yield.
Batches failed. The study defines batches failed as "the number of batches per month that failed and were not able to be successfully reworked" (1). This definition, however,
does not account for the monthly volume of production for the product in question. Although batches started was one of the performance measures described in the study and requested by the survey, it does not appear to have been used
for this metric.
To understand why this omission is important, consider the following example. Assume Facility A and Facility B both make Product
X. Facility A manufactures approximately 10 batches of Product X each month on average. Facility B manufactures approximately
100 batches of Product X each month on average. If both facilities reject 1% of their monthly production of Product X on
average, how do these facilities compare in terms of batches failed? Facility A has a batches-failed metric of 0.1 batches
per month. Facility B has a batches-failed metric of 1.0 batches per month. According to the study, the mean for batches failed
for oral, topical, and injectable facilities was 0.13 batches failed per month, but this does not consider the number of batches
manufactured. Using this example, Facility A performed slightly better than the mean for Product X, and Facility B performed
poorly and well above the mean for Product X. Both facilities, however, rejected an equal percentage of their monthly production.
What is the relevance of this point for the contract manufacturers that participated in this study? To address that question,
it is important to examine the size and scope of those participating contract-manufacturing facilities. For injectable manufacturing
facilities, the two contract-manufacturing facilities both were fairly large. For example, one contract-manufacturing facility
ranked first with the largest average facility size, highest number of compounds, and largest average number of employees
and operating hours. Oral and topical contract manufacturers generally were large as well. Five of the nine facilities that
were above an average of 22,000 m2 from 1999 to 2003 were contract manufacturers, one of which was ranked the largest of all facilities that provided data
for the study. A contract manufacturer also ranked first in terms of average total facility employees. In fact, three of the
four facilities reporting over 1000 total facility employees were contract manufacturers. Contract manufacturers held five
of the top six rankings in terms of manufacturing-facility operator hours per month, averaging over 600 hours per month from
1999 to 2003. Additionally, contract manufacturers held the top four rankings of the 18 facilities that reported data concerning
the number of compounds manufactured.
Not surprisingly, the top three contract manufacturers had a combined total of as much as 200 compounds manufactured. A large
number of compounds, employees, and operating hours using a large facility suggests a large monthly production volume that
should be accounted for in benchmarking batch failure. The need to account for monthly production volume for the product under
analysis becomes even more evident when considering that the study reported a large range—0 to 19—of batches failed per month.
While using an absolute metric such as batches-failed-per-month may provide an indication of overall product unavailability
and waste, it does not provide the best comparative measure for a facility's ability to manufacture a product in such a way
that will ultimately lead to its release.
Cycle time. The study defines cycle time as "the average number of days between batch start and those batches either accepted or rejected during the month" (1). It
reports the mean cycle time for all oral, topical, and injectable facilities as 27.14 days. Although this may seem like a
reasonable standard of comparison for benchmarking both contract manufacturers and product-owners alike, two points should
First, in order to compare facilities that manufacture their own products with contract-manufacturing operations, it must
be assumed that the cycle processes that each facility follows are equivalent; however, this situation often is not the case
for contract manufacturers. Because the product's license holder assumes the ultimate legal responsibility for product release
to the market, product owners often review the associated batch documentation and provide release and authorization for shipment
to the contract manufacturer. This additional step can take days or even weeks, depending on the prioritization and schedule
of the product's owner. This difference does not appear to have been accounted for in the study.
Second, this study pools cycle times stemming from many different processes that may have inherent cycle-time limitations.
For example, an injectable product typically requires sterility testing as part of its release criteria. Samples undergoing
this test have a fixed incubation time that must be built into any cycle-time expectations. Nonsterile processes such as those
governing oral and topical products do not require this time-consuming test. Compressing a batch of tablets may take hours;
some freeze-drying cycles for sterile products can take nearly a week. This point is made clear by examining the standard
deviation and the range of cycle times given for the study. The standard deviation was given as 28.76 with a range of 1.00–187.00
days. For a benchmarking study, the need for large pools of data to yield statistically significant conclusions makes sense;
however, this need must be weighed against the similarity of the data and its ability to be pooled. In this case, given the
large variation among the processes and the enormous variability within the data, the presented pooling does not appear to
be nearly as useful as it might seem.
Yield. The study defines theoretical and actual yield. The theoretical yield is defined as "the ratio of the theoretical amount of output that would be produced at any appropriate phase of production
of a particular drug product to the quantity of components used, in the absence of any loss or error in production, stated
as a percentage" (1). The actual yield then is the ratio of the actual yield to the theoretical yield at the same phase of production. The mean actual yield for
this study was reported as 96.38%. While the mean itself could be recognized by pharmaceutical facilities across the industry
as a reasonable expectation, the standard deviation of 6.64% with a range of 24.00–124.00% suggests otherwise.
The fact that this study allows and pools yields taken from any appropriate phase of production is somewhat surprising given
the number of potential production phases in oral, topical, and injectable operations. Granulation, tableting, encapsulation,
film-coating, sorting, liquid manufacturing after filtration, semisolid manufacturing completion, and filling and packaging
operations are among the many phases where a calculation of yield may be both appropriate and necessary. The report does
not appear to categorize survey data according to the yields reported in phases; instead, the data is pooled into the overall
assessment. Should a value that measures the amount of sterile bulk filled against the quantity that was manufactured be compared
with a value representing the number of compressed tablets arising from the amount of powdered granulation manufactured? Would
the expectations be the same? A review of the standard deviation and range suggests that in many cases, similar processes
are not being compared. Again, the need for meaningful comparisons appears to have been sacrificed to generate a statistically
significant data pool upon which to base conclusions.
Deviations. Deviations were divided into three categories: raw material, product component, and product/process specification. This metric
reports the number of each type of deviation that occurred for the product in question per month. As with batches-failed,
this metric does not account for the product's production volume. Although this metric provides an indication of the amount
of time wasted by addressing each type of deviation, it does not provide the best comparative measure for a facility's ability
to manufacture a product in such a way as to avoid deviations. The metric, however, does look at how the number of deviations
of each type changes over time. With regard to contract manufacturing, the study concluded, "OT&I [oral, topical, and injectables]
contract manufacturing has no impact on production unavailability or the level of deviations. It does lead to lower product
and process specification deviations over time" (1).
Strategic operating guidelines
The study did an effective job selecting key representative performance metrics and listing the numerous variables that may
affect them. Taken as a whole, these metrics suggest some strategic guidelines that all contract manufacturers should follow.
Capitalize on experience. The study shows that contract manufacturers were among those facilities with the greatest number of compounds and processes.
Each new compound that a contract manufacturer introduces provides an opportunity to expand its library of pharmaceutical
knowledge. When captured and used properly, this knowledge may lead to improved technical transfers, reduced deviation rates,
fewer batch failures, improved yields, and improved cycle times. Contract manufacturers must have systems in place that identify,
collect, and assess all necessary information regarding the processes they use to manufacture their customers'products and
the resulting outcomes.
Define, track, and trend metrics. To fully capitalize on manufacturing experience, contract manufacturers should define, track, and trend metrics for key performance
indicators such as those considered for this study. Management must understand what these metrics are intended to measure.
For example, everyone would agree that tracking a measurement for batch failure is of critical importance.
Of equal importance, however, is defining which aspect of batch failure to focus on. For example, is a company interested
in what percentage of its total production is rejected? This metric may indicate how well overall operations are performing.
Perhaps a company wants to know what percentage of a newly introduced product fails. This metric may give an indication of
the technical transfer success. A company may even be interested in how much cost in time, materials, and opportunities is
lost because of batch failures. Each of the measurements is valuable, and each would be calculated and presented differently.
Contract manufacturers should make clear to their entire value chain what is being measured, how it is being measured, and
why it is being measured. These metrics must be tailored to fit the existing and potential customer's needs and concerns.
Contract manufacturing is a service business, and the metrics are only meaningful if the customer can understand them in terms
of their own operations. Metrics can be a tool that not only serves to suggest areas where operations can be improved, but
also may be used to help convince potential customers to outsource a similar product type.
Employ electronic deviation-management systems. Contract manufacturers should use electronic deviation-management systems. The study looked at how companies employed information
technology systems to track and manage their deviation process. Specifically, the study questioned whether the deviation-management
system used had the capability to:
- track deviations by lot;
- track deviations by type of issue;
- track people assigned to resolving the deviation;
- centrally store the data.
Contract manufacturers were among the first to adopt these systems. For orals and topicals, seven of the 10 contract manufacturers
had such a system, and four of the seven have had their system in use since 1999. The study links the routine usage of such
a system to improvements in cycle times, batches failed, and deviations. These systems allow deviations to be easily categorized,
tracked, and, most importantly, trended. Once a trend is identified, corrective and preventative actions (CAPA) can be initiated,
and future occurrences of similar deviations can be avoided. Once this CAPA has been implemented, repeat occurrences of deviations
will disappear, potentially leading to an overall decrease in the number of deviations, as the contract-manufacturing data
from the study observed. Fewer deviations will lead to fewer failed batches. It also may result in a decrease in the overall
cycle time from not having to waste days performing time-consuming and costly investigations.
The benchmarking study Pharmaceutical Manufacturing Research Project–Final Benchmarking Report conducted by the McDonough School of Business at Georgetown University and the Olin School of Business of Washington University
in St. Louis (1) is useful for contract manufacturers because it accurately identifies the performance metrics most associated
with quality and cost. Few pharmaceutical manufacturers would disagree that the need to avoid deviations and failed batches
drives quality, and that reducing cycle time while optimizing yield affects cost. The study further offers a multitude of
variables that may affect these parameters. Unfortunately, when considering how to define these key performance metrics, the
study fails to take production volume into account and thus fails to assess each facility's ability to manufacture a batch
without deviation or need for rejection.
The study also fails to recognize the need to further categorize the data before pooling, and therefore the data lose any
meaningful comparison. As a result, the study's conclusion that contract-manufacturing performance corresponds with inferior
performance metrics can't be supported because of the limitations of the data. Despite these limitations, the study does illuminate
several strategic operating points for contract manufacturers to consider for optimizing their operations and improving both
quality and cost.
First, the sheer number of variables considered in the study serves as a reminder of the importance of having meaningful metrics.
A variable cannot be improved unless it can be measured, but the variable should not be measured until it is understood what
is being measured and why it is being measured.
Second, the data suggesting that the contract-manufacturing facilities that participated in the study were generally large
in size and scope can be an enormous strength for contract manufacturers if they have systems in place to ensure they capitalize
on every experience.
Lastly, to reduce the number of deviations and batch failures, contract manufacturers must identify the root cause and take
the appropriate corrective and preventative actions. The study accurately linked the use of electronic deviation-management
systems with success in that endeavor. Contract manufacturers should employ electronic deviation-management systems to make
trending, correction, and prevention easier. Supported by a strong, metric-driven trending program stemming from an electronic
deviation-management system and robust processes that capitalize on every experience, contract manufacturers should have little
trouble demonstrating superior performance to potential customers.
Shane Ernst* is the quality assurance director at DSM Pharmaceutical Products, Inc., PO Box 1887, 5900 NW Greenville Blvd.,
Greenville, NC 27835-1887, tel. 252.707.2327, fax 252.707.7512, firstname.lastname@example.org
*To whom all correspondence should be addressed.
1. J. Macher and J. Nickerson, "Pharmaceutical Manufacturing Research Project–Final Benchmarking Report," McDonough School
of Business, Georgetown University, Washington, DC and Olin School of Business, Washington University in St. Louis, St. Louis,
MO, Sept. 2006,
http://www.olin.wustl.edu/faculty/nickerson/results/, accessed Apr. 2. 2007.