7. Results of the Simulation Study
7.2 Description of Results from the Base Case Simulation Scenario
the “Base Case” and one further scenario that highlights the predominant
characteristics. This chapter then continues by reporting the key findings from the wider set.
The scenarios selected as “Base Case” and “Alternative Case” were determined after seeing the simulation results but since they represent relatively realistic Phase 2 scenarios then their use should be acceptable (and useful) to researchers planning a Phase 2 clinical trial. Further details of the differences between these 2 scenarios and justification of the “Alternative Case” are provided in Section 7.3.
7.2 Description of Results from the Base Case Simulation Scenario
Based on the simulation plan in Section 6, the scenario in Table 10 describes the specific simulation that is considered as the “Base Case” and thus warrants an extended description. This scenario has been selected since it demonstrates the principle features that are present in the majority of tested scenarios whilst also highlighting the key aspects of the simulation parameters that warrant further description later in this chapter (for example, impact of model-based estimates
- 146 -
outside the range of available doses in the trial). The scenario described in Table 10 can be cross-referenced with scenario [2I] in Table 9.
Table 10 Simulation Assumptions for Base Case
Scenario Feature Value(s) Comments
Target response
rate, θ 0.5
Target response rate higher than studied in majority of literature of early phase trials and key aspect of my research (see Section 2.5) rate on lower doses but high effect at high doses (see Figure 22).
Allocate the nearest dose to the current model estimate (where current model estimate referred to as γ)
Investigates key aspect of my research (see Section 2.5).
First 3 cohorts (i.e. doses=0, 1, 2) have 3 patients each then estimation process for γ begins for remaining patients, up to maximum 40 patients
- 147 -
Scenario Feature Value(s) Comments
Dose escalation
rule Any of the available doses
Investigates performance of unrestricted dose escalation rule which is a key aspect of my research (see Section 2.5).
Number of that aims to let the trial data (largely) determine the Posterior (e.g. see Chu et al.
(2009)).
Before proceeding to summarise 1000 simulations, it is useful to describe in detail how the model-based estimation of the target dose, γ, proceeds using the EWOC design (for simulated patients 9 to 40 in this Base Case scenario of an Intermediate dose-response curve). By considering simulation 1 (of 1000), the first row of Table 11 shows the sequence of outcomes for the first 9 patients (3 cohorts of 3 patients assigned doses 0, 1 and 2) after which the first model estimation was conducted.
Figure 25 shows the marginal posterior distributions for the two parameters 𝛽0 and 𝛽1 as well as the marginal posteriors for the derived parameters 𝜌0 and γ (see Equations ( 2.8 ) and ( 2.9 )) after 9 patients. It is clear from these equations and Figure 25 that very small values of 𝛽1 will lead to very large values of γ, however, since the posterior median of γ is used as the summary measure then extreme values have almost no impact on the recommended dose for the next patient.
- 148 -
Figure 25(a) Posterior for 𝛽0 Figure 25 (b) Posterior for 𝛽1
Figure 25(c) Posterior for 𝜌0 Figure 25 (d) Posterior for γ
Figure 25 Marginal Posterior Distributions for Base Case after 9 Patients
The posterior median after 9 patients (with no responses) was γ=5.28. Recall that the true value of the dose leading to a 50% response rate is 4.1429 so zero
responses for dose=1, 2 and 3 is unsurprising. In the EWOC design with a dose allocation rule assigned using the nearest dose (i.e. using Equation ( 2.16 )) then patient 10 is allocated dose=5 (see Table 11).
Since patient 10 also did not respond then the model provides a higher estimate of γ (i.e. 9.74), so patient 11 is allocated dose=10 (and responded).
- 149 -
Table 11 Model Estimation for Simulation 1 from Base Case Scenario
Patient Dose Outcome
(0=No Response;
5.28 (First estimation performed after N=9)
10 5 0 9.74
The corresponding marginal posterior distributions after 40 patients are provided in Figure 26 and show how the marginal posterior for γ is concentrated between 4 and 5.5.
- 150 -
Figure 26(a) Posterior for 𝛽0 Figure 26(b) Posterior for 𝛽1
Figure 26(c) Posterior for 𝜌0 Figure 26(d) Posterior for γ
Figure 26 Marginal Posterior Distributions for Base Case after 40 Patients
The estimated dose response curve for this single simulation after 40 patients using the posterior median values is plotted in Figure 27 alongside the true
dose-response curve and shows how (as expected) under-estimation at the lower part of the logistic curve is compensated by over-estimation at higher doses. A “good”
estimator will, on average, minimise such discrepancies and later results will describe the properties having performed thousands of simulations.
- 151 -
Figure 27 Estimated versus True Dose Leading to Target Response Rate of 0.5 for Base Case after 40 Patients
Although it is useful to obtain an estimated dose-response curve to enable
prediction of future responses to different doses, it is also important to understand if (simulated) patients were allocated to doses at or near the true value. As such, the doses allocated and the outcomes of the 40 patients from Simulation 1 are
summarised in Figure 28 and show that no patients were allocated to dose=4 (i.e.
the dose closest to the true value of 4.1429), however, of the 22 patients allocated to dose=5, 12 did respond (i.e. response rate=0.55). Note that although the
underlying model predicts a response rate of 0.65 for dose=5 (Figure 27), the observed response rate (12/22) is within sampling variability for small sample sizes (i.e. just 2 more responses, 14/22, would give a response rate of 0.65).
- 152 -
Figure 28 Summary of Dose Allocation and Outcomes for Simulation 1 from Base Case Scenario
Comparison of Figure 27 and Figure 28 also allows us to observe that the estimated and true dose-response curves converge around the doses for which most patients were allocated, namely doses 5 to 7 (and with a mix of non-responders and
responders). Since this is just a single simulation, generalisations would be
premature, however, there is intuitive appeal of an estimator that has minimal bias near the values containing most information.
Although 40 patients may be a suitable number to recruit for the entire trial, the repeated estimation in a trial with EWOC design allows a temporal inspection of the model estimates, hence the model estimates of γ from Table 11 have also been presented in graphical form as a profile plot in Figure 29 and shows that:
• all estimates of the target dose, γ, were close but higher than the true value
• estimates of γ stabilised after approximately 20 patients
- 153 -
Given the small changes in model estimates (beyond 20) then had this been a real clinical trial, the (long memory) EWOC design potentially allows the investigator to decide if enrolment of future patients would be worthwhile. That could be done either informally (e.g. by visual inspection of the data in Table 11 and noting that estimates of γ change by <1 after each patient response) or formally (e.g. by calculating the posterior predictive distribution for subsequent patient(s)).
Figure 29 Example Profile Plot from 1 Simulated Trial for Base Case
Clearly, generalisations of the small sample properties of the EWOC estimator cannot be made on the basis of 1 simulation so this chapter continues with the description of the corresponding results for 1000 simulated clinical trials. In particular, the profile plot of 1000 simulations is shown in Figure 30.
- 154 -
Figure 30 Profile Plot from 1000 simulated trials for Base Case
The plot shows that:
• Initial estimates of the target dose γ (e.g. when N<20) varied highly i.e.
estimates ranged from 2 to 20.
• In the majority of simulations, the estimate stabilised to its long-term value by the time 20 patients had been simulated (similar results in a range of other scenarios will be discussed later)
• In one scenario, the estimate of γ at N=20 was more than double the true value (i.e. 8.6781 versus true value of 4.1429) and was still 57% higher than the true value at N=40 (i.e. 6.5284 versus true value of 4.1429). This
scenario is the blue profile line in Figure 30 that is consistently higher than all others from N=19 patients onwards.
A visual representation of the (small) bias at N=40 patients from the same 1000 simulations is found by inspecting the distribution at N=40 from Figure 30 and this is plotted in Figure 31 along with a Kernel Density Estimate (KDE).
- 155 -
Figure 31 Example of the Distribution of Estimates of the True Dose Leading to Target Response Rate of 0.5
In the example from Figure 31, the true value for γ is 4.1429 and so the mean bias is 0.0041 (i.e. 4.1470 – 4.1429). Furthermore, the mean bias for all of the potential response rates, θ=0.1 to 0.9 for the Base Case scenario from Table 10 is shown as the blue values in Figure 32 (Mean Square Error, MSE, is also shown by red values).
Positive bias for values of θ<0.5 and negative bias for θ>0.5 indicates that the slope parameter, 𝛽1 is consistently over-estimated (i.e. greater than true value of 0.7) when implementing the dose allocation rule of Equation ( 2.17 ) i.e. one which takes the nearest of the available discrete doses. Further generalisations will be made in Section 7.4.3 when discussing the larger set of simulation scenario results in Figure 41.
The practical implication of positive bias (albeit small) for low values of θ (i.e. 0.1 to 0.3) is very relevant in Phase 1 oncology trials, since this has been the area in which the EWOC design was developed and has been applied (see Section 2.4.3). In
particular, positive bias for target toxicity rates between 0.2 and 0.35 could P50 Posterior Estimates of Target Dose for Response Rate Theta=0.5
1000 Simulated Trials (using doses {0 to 16 by 1} Maximum N=40 each trial) Prior for Beta0 ~Un(-16,16), Beta1 ~Un(0.01,5)
Distribution after 40 patients (with KDE superimposed)
DATASEED=12345 for Initial cohorts then DATASEED varied. Output code: P_AV0_16BY1_I_NR_BUN_P50_N40_50 Dose Allocation rule: Gamma Nearest DOSE
True Dose-Response: logistic Intermediate: Beta0=-2.9, Beta1=0.7 Implies True Gamma=4.1429 at Theta=0.5
True Value
Posterior Estimate of Target Dose, Gamma
- 156 -
translate to the model recommending doses higher than the true value. If such (higher) doses are subsequently used in later (and likely, larger) trials then the observed toxicity rate in those later trials might be higher than the investigators predicted based on the Phase 1 study. It is thus recommended that when planning clinical trials, simulation is used to characterise the potential for bias according to the expected dose-response curve (and variants thereof) as well as other study design parameters (see Section 7.8 for other recommendations from the
simulations).
Figure 32 Example of Mean Bias and MSE for various Target Response Rates
- 157 -
As expected, the value of MSE is minimised when the bias is close to zero (see Equation ( 6.7 )). The corresponding values of the mean standardised bias for θ=0.1 to 0.9, respectively, were 162%, 126%, 73%, 31%, 1%, -13%, -36%, -54% and -59%
thus only values of response rate, θ, between 0.4 and 0.7 were within the ±40%
rule-of-thumb of Collins et al. (2001).
This chapter continues by describing in detail the impact of changing one aspect of the Base Case that addresses one of the open questions listed in Section 2.5.