The problem with POS

The Problem with Probability of Success (POS) in Biotech and Pharma

Probability of Success (POS) is a widely used metric in biotech and pharma, offering a single probability that a drug candidate will successfully navigate the entire development process, from discovery through preclinical and clinical trials to regulatory approval and eventual market entry. By aggregating cumulative risks across all stages, including scientific, clinical, regulatory, and commercial factors, POS provides stakeholders with a high-level view of potential outcomes. While the metric’s simplicity and comprehensiveness make it appealing, these very qualities are also its greatest weaknesses. POS often oversimplifies the nuanced and dynamic nature of drug development, leading to misinterpretations and suboptimal decision-making.

The Definition and Calculation of POS

POS is expressed as the cumulative product of success probabilities across each stage of development:

$$
POS = P_{discovery} \cdot P_{preclinical} \cdot P_{phase\_I} \cdot P_{phase\_II} \cdot P_{phase\_III} \cdot P_{approval} \cdot P_{market}
$$

Where:

• Pdiscovery: Probability of identifying a viable drug candidate.
• Ppreclinical​: Probability of advancing from preclinical to Phase I
• Pphase I: Probability of advancing from Phase I to Phase II
• Pphase II​: Probability of advancing from Phase II to Phase III
• Pphase III: Probability of advancing from Phase III to regulatory submission
• Papproval: Probability of gaining regulatory approval
• Pmarket: Probability of successful market entry and commercial viability.

This framework aggregates risks across technical, regulatory, and commercial dimensions, resulting in a single metric intended to guide investment, development, and strategic decisions.

Critique of POS

1. Oversimplification of Complex Risks

POS attempts to distill a complex, multidimensional development process into a single number. This oversimplification often obscures critical interdependencies and the dynamic nature of risk.

• Interdependencies Between Risks:
  Risks in drug development are rarely independent. Clinical trial success, for instance, can heavily influence regulatory outcomes. Similarly, early market entry success may depend on pricing strategies informed by trial endpoints. POS calculations typically treat each stage as independent, ignoring these interconnections and oversimplifying the underlying realities.
• Dynamic Nature of Risk:
  The probability of success evolves over time as new data emerge, external conditions shift, and internal strategies adapt. A promising Phase II result may significantly increase the likelihood of success in subsequent stages, while a competitor’s market entry might decrease commercial potential. POS is often treated as static, failing to account for these dynamic factors.

2. Reliance on Historical Averages

POS calculations often rely on historical averages derived from aggregate industry data. While historical benchmarks provide useful starting points, they fail to capture the unique risks associated with innovative or first-in-class therapies.

• Black Swan Events and Paradigm Shifts:
  POS struggles to account for black swan events—rare but transformative developments that defy historical norms. For instance:
  • The success of mRNA vaccines during the COVID-19 pandemic was an outlier relative to historical averages for vaccine development.
  • Breakthrough therapies targeting novel mechanisms, such as CAR-T cell therapy, are poorly predicted by POS models grounded in historical data.
• Outdated or Misaligned Benchmarks:
  Industry benchmarks often lag behind real-time changes in regulatory pathways, therapeutic innovations, and trial designs. Metrics based on historical success rates fail to reflect the impact of fast-track designations, adaptive trial designs, or emerging therapeutic modalities.

3. Exclusion of Commercial Success Factors

While POS nominally includes commercial risks, it frequently underrepresents or oversimplifies them, focusing instead on technical and regulatory challenges.

• Pricing and Reimbursement Risks:
  Even a technically and regulatory successful drug may fail commercially due to pricing pressures or reimbursement hurdles. For instance:
  • Therapies targeting rare diseases may face challenges in securing payer approval despite high clinical efficacy.
  • Emerging markets may limit pricing flexibility, eroding projected revenues.
• Market Competition:
  POS often excludes the impact of market dynamics, such as competitor drugs launching earlier or eroding the target market. A high POS may mislead stakeholders into overlooking these commercial challenges.

4. Ambiguity in Scope and Application

The definition of POS varies significantly across stakeholders, leading to inconsistencies in its application.

• Varied Definitions:
  Some stakeholders use POS to represent only technical and regulatory success, while others include broader commercial factors such as market entry. This ambiguity complicates comparisons across studies, industries, or projects.
• Inconsistent Methodologies:
  Variations in how POS is calculated—for example, whether it includes market success or not—lead to significant differences in interpretation. Without a standardized approach, POS can mislead rather than inform decision-making.

Illustrative Example

A Comparative Example: HER2 Pathway vs. Ferroptosis as a Target

Case 1: HER2-Targeting Therapy

• Context: HER2 (human epidermal growth factor receptor 2) is a well-studied pathway implicated in breast, gastric, and other cancers. Multiple therapies targeting HER2 have been approved, including monoclonal antibodies (e.g., trastuzumab) and antibody-drug conjugates (ADCs).
• Development Rationale:
  • Established biomarkers predict patient responsiveness.
  • Robust clinical data support the pathway’s role in oncogenesis.
• Probabilities:
  • Pdiscovery=0.95 (target already validated).
  • Ppreclinical=0.90 (existing preclinical models).
  • Pphase I=0.85 (well-characterized safety profile).
  • Pphase II=0.75 (predictable efficacy in stratified populations).
  • Pphase III=0.70 (confirmation of known effects).
  • Papproval=0.95 (high regulatory familiarity).

Calculation:

$$
POS_{HER2} = 0.95 \cdot 0.90 \cdot 0.85 \cdot 0.75 \cdot 0.70 \cdot 0.95 = 0.43 \, (43\%)
$$

Case 2: Ferroptosis-Inducing Therapy

• Context: Ferroptosis is a recently discovered form of regulated cell death driven by iron-dependent lipid peroxidation. It is a novel and underexplored target in cancer, with significant potential to treat therapy-resistant tumors.
• Development Rationale:
  • Hypothesis-driven approach with limited prior validation.
  • Few preclinical models accurately mimic ferroptosis-driven pathologies.
• Probabilities:
  • Pdiscovery=0.70 (target novelty increases uncertainty).
  • Ppreclinical=0.65 (unproven models).
  • Pphase I=0.50 (limited safety data for ferroptosis inducers).
  • Pphase II=0.40 (unpredictable efficacy in human populations).
  • Pphase III=0.30 (high risk of trial failure).
  • Papproval=0.60 (regulatory unfamiliarity).

Calculation:

$$
POS_{ferroptosis} = 0.70 \cdot 0.65 \cdot 0.50 \cdot 0.40 \cdot 0.30 \cdot 0.60 = 0.0273 \, (2.73\%)
$$

Interpreting the Results

1. HER2-Targeting Therapy:
   • The calculated POS of 43% reflects the well-established nature of the HER2 pathway.
   • Strong preclinical and clinical precedent reduces uncertainty, making it an attractive investment for stakeholders seeking lower-risk opportunities.
2. Ferroptosis-Inducing Therapy:
   • The 2.73% POS highlights significant risks due to limited prior validation and high scientific uncertainty.
   • However, success in this space could transform oncology, offering therapies for patients with few options and creating new therapeutic markets.

This example underscores the limitations of POS. While it accurately reflects lower risks for HER2-targeting therapies, it penalizes innovative approaches like ferroptosis inducers, which offer disproportionate potential rewards if successful.

Conclusion

Probability of Success (POS) offers a high-level framework for assessing risk in drug development, but its oversimplification of complex risks, reliance on historical averages, exclusion of commercial dynamics, and ambiguity in scope limit its utility. In an industry defined by uncertainty, innovation, and competition, POS often fails to provide a complete picture of a drug’s potential. While useful as a starting point, POS should be complemented with more nuanced analyses that incorporate dynamic risks, market conditions, and strategic factors to ensure informed decision-making.