Finally, Standardized KPIs are Front and Center

October / November 2016

Most firms are now using a standard set of key performance indicators

During the past 15 years, my teams at the Tufts Center for the Study of Drug Development (Tufts CSDD) have frequently encountered organizations struggling with the collection and application of meaningful and relevant management metrics. These struggles have largely been specific to each given company’s culture and operating structure: In the past, it was not unusual to find companies participating in Tufts CSDD working group studies who are not actively collecting certain management-level metrics. On the opposite end of the spectrum, some participating companies have appeared to be collecting too much performance and quality data, often without a clear sense for its purpose. In many cases select financial and quality metrics have been too difficult for many organizations to collect, as this data resides in separate functions and cannot be accessed easily.

Perhaps one of the largest challenges for organizations has been benchmarking multi-company performance, efficiency and quality metrics. Ten years ago, Tufts CSDD working group companies each defined their operating level metrics differently. Wide variation in metrics-gathering practices was observed not only at the corporate level but also between functional areas and even clinical teams. These conditions made it very difficult to compare project performance and quality within a single company and across peer companies. As a result, Tufts CSDD frequently had to assist participating companies in establishing and collecting consensus-defined metrics. Over time, my study teams routinely turned to standardized performance and quality metrics developed by the Metrics Champion Consortium (MCC) to support our research activity.

Although the drug development enterprise has been slow to adopt standardized performance and quality metrics definitions, progress has been made. Indeed, a new study just published by the MCC suggests that the majority of companies are now using a standard set of senior-management level key performance indicators (KPIs) and adoption of functional level standardized performance metrics has reached an inflection point.

Adoption drivers

A confluence of factors has no doubt contributed to widespread adoption of standardized metrics. To name but a few: the growing use and coordination of eClinical technology solutions has made data capture and access easier and more convenient. Facing perennial operating challenges—including clinical project delays; high staff turnover and workload; longer cycle times; poor patient recruitment and retention rates; high levels of unanticipated protocol amendments and change orders; and the rising cost of clinical trial management and support—sponsors and contract research organizations (CROs) are hyper-sensitized to monitoring performance and quality actively.

Increasing awareness of the root cause drivers of poor development performance and economics has also stimulated management interest in gathering metrics and analyzing organizational practices, behaviors and activities. A growing body of research conducted during the past 15 years shows that protocol complexity is highly associated with clinical trial performance, quality and cost. Protocol designs that include a relatively large number of eligibility requirements and unique procedures conducted frequently have more delays, lower study volunteer recruitment and retention rates, higher average numbers of protocol amendments and protocol deviations, and generate lower quality clinical data than designs without such features.

As an aside, a 2016 Tufts CSDD soon to be published indicates that protocol complexity—as measured by the number of unique procedures, total procedures performed, eligibility criteria, planned study volunteer visits—is not only rising, it is accelerating, largely in response to demand for data to support more secondary, tertiary and exploratory (i.e., “non-core”) protocol endpoints.

Rising reliance on outsourcing and the use of more collaborative partners to support drug development activity has also contributed to growing demand for management metrics at both the senior level and day-to-day operating level. Performance data transparency has become essential to expectation setting, routine relationship management and incentive tracking.

Regulatory agencies and public-private partnerships have also contributed to the adoption of standardized metrics. Agencies have released guidance and regulations encouraging the clinical research enterprise to build practices ensuring a higher level of quality into development planning to drive downstream efficiency and performance. The Quality by Design (QbD) movement, formally introduced in 2011, incorporates quality and risk-management principles into clinical trial oversight and execution. During the past several years, both TransCelerate and the MCC have launched risk assessment and risk mitigation tools to assist pharmaceutical and biotechnology companies in assessing and managing project- and protocol-level risk and quality (e.g., lower error rates, higher compliance and better patient safety) prior to initiating clinical trials.

The 10th anniversary of the MCC’s founding is yet another indicator of the growing recognition by clinical research professionals of the importance of standardized operating function-level and corporate-level performance metrics. More than a hundred companies are participating MCC members. And several years ago I joined the MCC board to reflect my personal commitment and belief in the critical need for, and benefit of, standardized benchmark performance measures.

Majority using KPIs

The results of a new 2016 MCC survey conducted among 44 predominantly large and mid-sized companies (with 2,500 total employees or more) are striking. At this time, nearly two-thirds of organizations surveyed report that they are now using a standard set of senior management-level KPIs. The majority of organizations (73%) review these metrics at least on a quarterly basis. In addition, about half of sponsor companies report setting annual operating performance goals and developing improvement action plans using end-of-year KPI results.

Companies responding to the survey indicated that they are using an average of eight senior-level KPI metrics that measure a variety of areas, including clinical operations, data management and quality assurance/quality control. The most common metrics include:

• Proportion of final databases locked on time
• Mean number of protocol amendments post-protocol approval
• Proportion of studies completing patient enrollment on time
• Mean number of protocol deviations per study volunteer
• Proportion of studies with investigative sites activated on time
• Mean number of times databases are unlocked per study
• Proportion of vendors with critical findings following an audit

At the functional level, the use of standardized performance metrics is very high in clinical operations and data management departments. More than 90% report currently using (83%) or planning to use (8%) standardized metrics. The most common clinical operations metrics—some that flow into corporate level oversight—include the percentage of studies completing enrollment on time; the percentage of studies activated on time and the percentage of regulatory packets approved the first time. For data management, the percentage of final databases locked on time and the percentage of case report forms finalized on time are among the most commonly used standardized metrics.

The MCC is releasing a full report on the results of the 2016 survey on its website followed by articles published in trade journals.

Up next

The International Conference on Harmonization (ICH) is expected to release its R2 guideline in November. This new guideline replaces the one implemented in 1997; accommodates the scale and complexity of current drug development strategy, management and practice; and promotes better data quality and human subject protection through the integration of risk-management processes.

With cloud-based technologies and the availability of ever-larger databases of structured and unstructured data and information, the opportunities are even greater for rich and robust performance analytics to inform the management and execution of increasingly open collaborative teams. Newer technology solutions will enable sponsors and their partners to identify leading standardized risk and ultimately standardized predictive indicators.

These are exciting times for the drug development enterprise. Selfishly, this is particularly welcome news for those of us actively involved in benchmarking enterprise performance and practice.