How can the success of clinical partnerships be measured?
How can the success of clinical partnerships be measured?
Partnerships at every stage of clinical trials – not just between Sponsor and CRO, but also with vendors - are essential to the success of a study. However, measuring the performance of a particular partnership is not always easy. Which metrics and KPIs should be tracked to judge not just the performance of the CRO or vendor, but the performance of the combined effort? We asked five industry experts this question, getting the view from pharma, service providers and consortia.
Sol Yates, Associate Director, Global Regulatory Affairs, Grünenthal
“This is an aspect we are currently still defining rules for. In the end, there are two main goals – ensuring quality requirements are met and that agreed timelines are adhered to. When outsourcing, it is also important to ensure that the cooperation with the CRO improves or at most equals workload at the Sponsor compared to fully inhouse-ran trials. This is typically achieved by working with a preferred CRO that will thus be familiar with the company’s processes.”
Patricia Moenaert, Director, Portfolio Sourcing and Relationship Management (PSRM), Celgene Intl
“Relationship KPIs include data such as data currency, TMF completeness and quality, protocol deviation/violation but also costs difference between initial agreed budget and final one. Sometimes a performance survey is used both at the CRO and Sponsor and used as a ‘soft skills’ indicator of the relationship.”
Dr. Frank Berger, Head of Analytics, Global Clinical Operations, Boehringer Ingelheim
“One of the most critical pieces of the combined effort is agreeing upfront on a realistic (and not too ambitious) plan regarding milestones and timelines (and cost). Creating such a plan will require time and resources on its own, but this is well invested. The performance of the partnership can then be assessed by measuring whether the actual figures meet the plan with regards to Time, Budget and Quality – the eternal Triangle of Constraints. But this makes sense only if the plan was realistic in the first place, with buy-in from both partners.”
Tina Bees, Project Director, Precision for Medicine, Oncology and Rare Disease
“Whilst metrics, timelines and KPIs are important to objectively measure performance of a clinical partner, the strength of the relationship and demonstrated flexibility to rapidly triage and adapt to the dynamic execution of an ongoing clinical trial are also critically important attributes for success. What metrics and KPIs should be measured? KPIs should lend insight into status of project adherence and timeliness (projected vs actual) including: site activations, enrolment, data completion, query rates, protocol deviations and overall achievement of key milestones as examples. Not just performance of the CRO/vendor but the performance of the combined effort.”
Keith Dorricott, Metrics Champion Consortium
“There are a number of approaches that could be used. At the Metrics Champion Consortium (MCC), we have developed a survey tool to be used between Sponsor and CRO/vendor along with associated metrics and recommended analysis. This can be used to provide a measure of the opinion across the partnership of how well it is working and could be monitored over time to establish any trends.
Over time, you would also expect that in a well-functioning partnership, behaviours would change. By defining what behaviours you expect to change, metrics could be developed to monitor these over time. For example: how often are risk assessments carried out jointly, do lessons from issues lead to updates to SOPs? Outcomes could also be measured, for example: is the volume of change orders reducing, is the number of critical quality issues reducing? By determining the questions you want to answer, metrics can be developed to help answer those questions. Organizations such as the MCC have metrics already defined that can assist.”
Clinical Trial Vendor Oversight and Governance
By Poonam Balani, PhD
Clinical Trial Vendor Oversight and Governance
By Poonam Balani, PhD
This piece represents the views of the author and not necessarily the views of Informa Connect.
Outsourcing of clinical trials is undertaken by pharmaceutical companies for a number of reasons including reduction of costs of conducting clinical trials, reducing wastage by engaging a vendor with manpower and expertise in the area of interest for the sponsor, and achieving reduced cycle times of introducing a new product in the market.
While these reasons form the basis of outsourcing, the sponsor treads a thin line distinguishing between micromanaging each and every step and effective oversight in the implementation of the clinical trial.
Vendor oversight is a trending topic as both, the vendors and the sponsors come to terms to what is expected of each of them and the associated liabilities and responsibilities while partnering together. With revisions to the regulatory requirements concerning vendor management and shared responsibilities of the sponsor and the vendor, it is important to understand how they impact vendor management and vendor.
The sponsor needs to be specific with the choice of the vendor for clinical trials and the selection of vendors that meet the outsourcing requirements. There is a need to understand the expectations for vendor oversight and how to implement a practical vendor oversight plan.
Vendor Selection: Procurement Strategies and Contract Design
The selection process of a contract research organization (CRO) involves screening of potential vendors for attributes such as cost estimates, timeline for the project, along with an analysis of the past completed trials and the reputation of the vendor. Cross functional teams on the part of the sponsor including project management teams, Research and Development (R&D) and Quality Control (QC), are involved in the vendor selection process and evaluate the ease of working with a potential vendor while maximizing the services from the same vendor for a specific project.
There are several factors that decide the selection of an appropriate vendor for a clinical trial. These include:
- Vendor experience in the therapeutic area of interest to the sponsor
- Proposed project execution plan
- Approach towards the project and problem-solving plan: definitions for the process of identification of issues, drawbacks and the resolution thereof
- Human resources/training
- Organizational chart/Geographic coverage of the vendor
- Compliance, Business ethics
- Data/Computer management systems
- Cost of the trials
This first screening step is achieved by screening the request for proposal (RFP) from the potential vendors once the sponsor provides the objective of the work along with the expected timelines. RFP template can be used to compare vendor bids and should include questionnaire based format for clarifications on the compliance status, corrective and preventive measures (CAPA) and the quality management systems (QMS) and the data management systems used by the vendors. Other certifications as well as terms and conditions for third party outsourcing by the vendor have to be adequately addressed at this stage to ease the screening process and facilitate the shortlisting of potential vendors.
Team chemistry or the overall positive relationship between the sponsor and vendor project team is one of the key criteria for selection of a specific CRO. Sponsors highlight the importance of a good working relationship between the project teams overlooking from the sponsor side and the project team of the CRO as the teams often work over long periods of time, sometimes lasting years and across geographical locations. It is important that the sponsor and the vendor share same ethical values and problem-solving approach for a successful sponsor-vendor relationship.
Experience of the vendor, not only for a specific indication, but also the general experience in handling clinical trials is another criterion that weighs high on the scale while selecting a vendor.
Project execution plan: a sponsor is largely interested in a convincing project execution plan, detailing the resources, countries, and site selection process as well as identification and management of risks associated with the clinical trials. Other criteria for the selection of a vendor include vendor’s geographic location or access to specific markets of interest. Lastly, lowest price may not necessarily be important for the selection of a vendor.
On-site Audit by the sponsor: before finalizing a vendor, the sponsor generally visits the facility of the vendor to access the capabilities of the vendor and evaluate the compliance status.
Formulating a Quality Agreement: following a satisfactory audit, the sponsor puts a Quality Agreement in place ensuring that the vendor is able to achieve, maintain and demonstrate compliant operations in handling the clinical trials. When a contract with the vendor is finalized, a Statement of Work (SOW) is a legal document that defines the services to be provided, deliverables and timeline for services being performed, the scope with all the study specific details on services performed by the vendor.
As the overall responsibility of the clinical trials lies with the sponsor, vendor management is the key to execute the clinical trials within the assigned timelines. Sponsors should maintain oversight on a regular basis via meetings with vendors and internal team members and check upon the documentation to ensure that project associated problems, if any, are discovered and resolved early. For successful execution of clinical trials, it is important that the sponsor and the vendor remain in tandem when it concerns inputs or guidance as well as in identifying drawbacks and finding optimal solutions to keep the program in the right direction while meeting the timelines. A sponsor oversight plan can include the information on the level of oversight and the documentation required from the vendor for the review.
Regulatory Requirements – Compliance with ICH GCP guidelines
As in the prior version of International Council for Harmonization (ICH) E6, Section 5.2 of ICH E6 (R2)  states that the sponsors may transfer responsibilities to CROs and the responsibilities delegated to CROs should be in compliance with the ICH requirements on the behalf of the sponsor. Additionally, ICH E6 (R2) section 5.2.2 requires that “The sponsor should ensure oversight of any trial-related duties and functions carried out on its behalf, including trial-related duties and functions that are subcontracted to another party by the sponsor’s contracted CRO(s).” Ultimately, it remains the responsibility of the sponsor to ensure appropriate vendor oversight and maintain that all ICH guidelines are adhered to.
When inspecting sponsors, regulators emphasize the sponsor’s oversight of the vendor and therefore, it is important that the sponsors adopt a risk-based oversight management covering all aspects of clinical trial design, execution, recording, evaluation process, reporting and archiving of the clinical trial data. Addendum to ICH E6 (R2)  introduces the term Quality Risk Management (QRM) for the first time. Risk based oversight management is an excellent tool that accounts for tall three dimensions of data management involving patient safety, data integrity and sponsor responsibilities during a clinical trial and allows for a proactive approach towards the management of risks.
Enhanced Vendor Oversight through Collaborative Quality Risk Management
ICH E6(R2) guideline encourages sponsor oversight of the vendor by inviting sponsors to partner with CROs to identify risks associated with critical data and protocol processes and for continuous ongoing risk management throughout the time course of the study. This is aimed at encouraging the trust between the sponsor and the vendor as they address any issues with the protocol and compliance early on and actively engage in resolving the issues. Real time data sharing by the CROs can also assist in enabling sponsor oversight.
Oversight of Vendor Subcontractors
Addendum 5.2.2 also states that “The sponsor should ensure oversight of any trial-related duties and functions carried out on its behalf, including trial-related duties and functions that are subcontracted to another party by the sponsor’s contracted vendors and service providers. This change in regulatory guideline now requires the sponsor to have an oversight of the subcontractor agreements with the third-party vendor as the onus of subcontractor oversight is on the sponsor.
It implies that the third party subcontracting may also have to be approved by the sponsor and the vendor would be required to evidence that they are actively engaged in managing their subcontractors throughout the period of the study, while allowing for a more direct sponsor oversight of the subcontractors agreements between the vendor and the subcontractors and are expected to enable effective risk management in conducting of clinical trials.
Vendor Responsibilities for Data Sharing with the Sponsors
Sponsors can make the vendor accountable for their responsibilities and timely sharing of data via specific contracts and agreements defining the scope and frequency of data sharing by the vendor. These contracts should consider the following aspects concerning data collection, storage and sharing responsibilities.
- Subcontracting agreements in place in the contract with details on the vendor maintains oversight and shares the agreements with the sponsor of any further subcontracting.
Compliance with Good Clinical Practices (GCP), as it supersedes any internal processes and procedures. It is important that the sponsor realizes the impact of the work contracted to a vendor on protection of the rights of the human subject as well as the reliability of trial results.
Follow-up in case of non-compliance
Communication plans obligating the vendor to update the sponsor on the status of the project. Communication plans should specify all protocols and Standard Operating Procedures (SOPs) in an event of a potential breach of contract.
Vendor Assessment, performed prior to contracting can be used as a mechanism for continuous assessment to review the vendor’s Quality Management System (QMS). The assessment may be in form of a questionnaire, review of SOPs, a site visit or audit. Maintaining Oversight involves regular communications with the sponsor via teleconferences or regular meetings and review of the project status and completed milestones through sponsor-initiated audits at regular intervals. Vendors should also provide the sponsor access to update reports from the subcontracted vendor. One of the key elements that can elevate the level of trust between the vendor and the sponsor is the delegation of authority by allowing the vendor to become one of the authorized signatories on key trial documents involving data management. Sponsor initiated performance reviews should also be in place in case of non-compliance and addressing the mitigation to prevent future non-compliance on the vendor’s behalf.
Sponsor can demonstrate oversight by regularly reviewing and approval of documentation including but not limited to Monitoring Visit Reports (MVRs), Data Management Plans, Statistical Analysis Plans and so forth. Sponsors are encouraged to document co-monitoring visits to the vendor facilities and enlisting all the activities performed during the visit and any issues identified, with an appropriate Corrective and Preventative Action Plan (CAPA). In an event of noncompliance, sponsors should have an ‘Escalation Plan’ to report significant noncompliance issues which should also be reflected in the contract between the sponsor and external vendor. As the documentation on Trial Master File (TMF) management is often outsourced and held by the vendor, the sponsor can demonstrate oversight through inspections of the documentation files.
Quality Agreements and Monitoring
Trial quality relies upon a well-articulated investigational plan (e.g., protocol, analysis and management plans). The need for Quality Agreements can be traced to the ICH guidance documents Q9 Quality Risk Management (ICH Q9) ,which recommends supplier evaluation through audits and supplier Quality Agreements, and ICH Q10 Pharmaceutical Quality Systems (ICH Q10) ,which states that the control and review of any vendor activity is the responsibility of the sponsor.
A written ‘Quality Agreement’ is imperative aspect that covers a clinical trial by defining the roles and responsibilities for all quality elements within the QMS. An independent ‘Quality Agreement’ is considered as best practice and should be available to all personnel (on the sponsors side as well as the vendors side) involved in the conduct of a clinical trial. By building quality into clinical trials, data integrity and subject protection can be achieved efficiently. The sponsor should detail the processes and procedures obliging the vendor to promptly notify the sponsor in an event of identification and escalation of any potential serious breaches as it is the obligation of the sponsor to report it to the regulatory authorities within 7 days of identification of a serious event.
Sponsors should consider having established action plans in an event of quality tolerance deviations for risk minimization. After the initial quality audit, the evaluation process of the vendor continues with monitoring of the vendor performance with respect to the delivery requirements. It is imperative that the sponsor and the vendor agree to the predefined quality objectives and key performance indicators against predetermined performance metrics, and the manner in which these are reported. Records need to be diligently maintained where there are identified nonconformances followed by Corrective Actions / Preventive Actions (CAPA) which the sponsor has to cross-reference to the vendor’s quality record file.
The risk management utilizes available clinical and market data to identify risk minimization efforts that might be required. It includes the understanding of anticipated treatment settings, typical habits and practices involving treatment within those settings. The risk minimization plan uses the knowledge from the medical affairs and commercial organizations and integrates the insights to the planning and implementation process of clinical trial design. To this end, a risk management specialist can act as in-house person to provide guidance and advice on how to plan, implement, and evaluate risk minimization programs .
Overseeing Technology Vendors
Current data management systems are extremely complex and not all vendors have the ability to handle huge amounts of data generated during clinical trials. Sponsors expect the vendors to have the ability to manage the clinical trial data, which is often outsourced to the subcontractors . Quality and scalability of IT systems and infrastructure are a necessity for conducting a clinical trial. Advanced software such as the clinical trial management system, electronic data capturing systems, trial master files, and market intelligence systems enable sponsors to monitor project progress and quality and view data online in real time. As data management systems are extremely crucial capturing and recording data, oversight of technology service provider is also crucial to implementing a successful clinical trial.
ABOUT THE AUTHOR
Poonam Balani has a PhD in biomedical sciences from the University of Muenster, Germany. She has 14 years of experience in medical and academic writing and has worked as a researcher at the University of Tennessee Health Science Center, USA and the Institute for Bioengineering and Nanotechnology, Singapore.
1. Guideline, I.H., Integrated addendum to ICH E6 (R1): guideline for good clinical practice E6 (R2).
2. Guideline, I., Quality risk management. Q9 Current Step. 2005.
3. Guideline, I.H.T., Pharmaceutical quality system q10. Current Step, 2008. 4.
4. Morrato, E.H. and M.Y. Smith, Integrating risk minimization planning throughout the clinical development and commercialization lifecycle: an opinion on how drug development could be improved. Therapeutics and clinical risk management, 2015. 11: p. 339-348.
5. Clinical Decision Support Systems Could Be Modified To Reduce ‘Alert Fatigue’ While Still Minimizing The Risk Of Litigation. Health Affairs, 2011. 30(12): p. 2310-2317.