Example

Consider a non-randomized cardiovascular device study. In this study, we collected data from a single-arm trial of an experimental embolic protection device for patients with small vessels. We also obtained data from historical control patients who met the inclusion and exclusion criteria of the single-arm trial of the experimental device. The primary objective of this non-randomized study was to assess non-inferiority via risk difference of the experimental device to the active control with respect to major adverse cardiovascular event (MACE). The experimental device is considered non-inferior to active control when the former is no more than 4% higher than the latter with respect to rate of MACE. This leads to the following hypothesis test: H0: 𝑝𝑐 − 𝑝𝐸 ≤ −0.04 vs. H₁: 𝑝_𝐶− 𝑝_𝐸 > −0.04. Since subjects in this study were assigned to experimental or

active control device in a non-randomized manner, we expected differences between treatment groups with respect to distribution of baseline characteristics. To account for this, the primary analysis was conducted on the propensity-score matched sample using nearest neighbor matching with a caliper width of 0.2 of the standard deviation of the logit of the propensity score. Each experimental subject was matched to 1 or 2 control subjects without replacement. The following are the baseline characteristics used to generate the propensity score: age (years), diameter stenosis (%), diabetes status (yes/no), lesion length (mm), reference vessel diameter (mm), and sex (male/female). The

propensity matched sample consisted of 111 subjects treated with the experimental device and 213 treated with the control device. About 1.9 controls were matched to each experimental subject. The unweighted estimate of the risk difference (𝑅𝐷̂ = 𝑝̂_𝐶− 𝑝̂_𝐸) was 0.0442 (or 4.42%). In order to account for the variable ratio matching, we calculated a weighted estimate of the risk difference (𝑅𝐷̂∗_{), where each control subject’s weight is} the inverse of the number of control subjects matched to the same experimental subject, while the weight for the experimental subject would be 1. The weighted estimate of the risk difference was 𝑅𝐷̂∗ _{= 0.0450 (or 4.50%). When using a method for independent} samples, i.e. Farrington-Manning’s (FM) method, the estimated standard error of the risk difference was 𝑆𝐸𝐹𝑀= 0.034096 and the test statistic value was

𝑍_𝐹𝑀 = 𝑅𝐷̂ − 𝛿0 𝑆𝐸_𝐹𝑀 =

0.0442 − (−0.04)

When using a method for correlated matched samples, i.e. the adjusted Farrington- Manning’s (AFM) method, the estimated standard error of the risk difference was 𝑆𝐸𝐴𝐹𝑀= 0.031332 and the test statistic value was

𝑍𝐴𝐹𝑀 = 𝑅𝐷̂∗− 𝛿₀ 𝑆𝐸𝐴𝐹𝑀 =0.045 − (−0.04) 0.031332 = 2.71 where 𝑆𝐸_𝐴𝐹𝑀 = √𝑉𝑎𝑟(𝑝̂𝐶) ∗ 𝑉𝐼𝐹𝐶 + 𝑉𝑎𝑟(𝑝̂𝐸) − 2𝐶𝑜𝑣(𝑝̂𝐶, 𝑝̂𝐸).

The unadjusted variance in the control group was 𝑉𝑎𝑟(𝑝̂𝐶)=0.00029648, variance

inflation factor for control group was 𝑉𝐼𝐹𝐶=0.916938, variance in the experimental group was 𝑉𝑎𝑟(𝑝̂𝐸)=0.00086605, and covariance 𝐶𝑜𝑣(𝑝̂𝐶, 𝑝̂𝐸)=0.000078108. The estimated standard error based on the method for correlated matched samples (i.e. AFM) was smaller than the estimated standard error based on the method assuming independent samples (i.e. FM). Both test statistics were greater than the critical value of 1.96. Therefore, based on both test statistics, we reject the null hypothesis and conclude that the experimental device is non-inferior to the active control. The adjusted test statistic (AFM) was larger than the unadjusted test statistic (FM).

2.6. Discussion

We compared a method for independent samples (i.e. Farrington-Manning method) to a method for correlated matched samples (i.e. adjusted Farrington-Manning method) when assessing non-inferiority via risk differences in one-to-many propensity- score matched samples. When all or almost all the experimental treated subjects are well- matched to one or more controls with similar propensity scores, we found that compared

to using the Farrington-Manning method, the use of the adjusted Farrington-Manning method resulted in: empirical type I error rates that were closer to the nominal level, higher empirical power, empirical coverage rates of 95% CI that were closer to the advertised 95% rate, and estimated standard errors that were more similar to the sampling standard deviation of the estimated risk difference.

The results from our Monte Carlo simulation study suggests that a method for correlated matched samples is preferable than a method for independent samples when assessing non-inferiority in well-matched samples that contain almost all of the

experimental treated subjects. For some scenarios, the original Farrington-Manning test was too conservative (i.e. significantly less than the nominal level), especially when baseline covariates were strongly related to the outcome. Using the adjusted Farrington- Manning method may increase the precision of the estimate of the risk difference and thus increase the empirical power. With respect to empirical type I error rates and power, differences between the FM and AFM methods were greater when the covariates were strongly related to the outcome, when the variability of the number of controls matched to each experimental increased, or when the sample size of the control group was not large enough to find "good" (with respect to propensity score) matches for the

experimental subjects.

Using one-to-many matching, instead of one-to-one matching, may increase power and precision, especially when using a method that accounts for correlated

matched samples (i.e. AFM method). However, if one is unable to successfully match all (or almost all) of the experimental subjects to at least one control subject, then one should

consider another approach (e.g. stratification, regression adjustment, etc.) to reduce confounding. Type I error rates for the AFM tended to be significantly inflated when some of the experimental treated subjects were excluded in the final matched sample. Not including all (or almost all) of the experimental treated subjects in the matched sample resulted in bias estimates of the average treatment effect for the treated (ATT). Widening the caliper may allow more experimental treated subjects to be matched. However, widening the caliper, or not using a caliper at all, may result in experimental subjects to be matched with dissimilar control subjects with respect to propensity scores, resulting in bias estimates of the treatment effect.

ATT is the measure of effect that is estimated when propensity-score matching is used (Imbens, 2004). The true ATT is defined as the average of the within-pair (i.e. 1-to- 1) differences of the outcome under the experimental and the potential outcome under the control for each experimental treated subject (Austin, Optimal caliper widths for

propensity-score matching when estimating differences in means and differences in proportions in observational studies, 2011). Therefore, we recommend using the weighted RD to estimate the ATT when matching one or more controls to each experimental treated subject (i.e. 1-to-≤M matching), where each control subjects’ weight is the inverse of the number of control subjects matched to the same experimental subject. The weights are in place to make sure the sample is representative of the

population of interest. Using a method for independent samples (i.e. FM method) does not account for the one-to-many variable-ratio matching, which can result in bias estimates of the risk difference.

The current study complements prior published research. Earlier studies found that in many settings, methods for matched samples tended to result in improved inference compared to methods for independent samples when used for analyzing propensity-score matched samples. In prior studies, type I error rates and empirical coverage rates of confidence intervals were studied for differences in means, relative risks, odds ratios, hazard ratios, and/or risk differences (Austin, Type I error rates, coverage of confidence intervals, and variance estimation in propensity-score matched analyses, 2009; Austin, Comparing paired vs non-paired statistical methods of analyses when making inferences about absolute risk reductions in propensity-score matched samples, 2011). All these studies only examined scenarios with one-to-one matching. These prior studies did not examine inferences about non-inferiority in one-to-many propensity-score matching. Also, these prior studies did not study the empirical power of these methods.

In conclusion, if all or almost all the experimental treated subjects are

successfully matched to one or more controls with similar propensity scores, then we recommend that the adjusted Farrington-Manning method be used when assessing non- inferiority via risk difference.

3. ASSESSING NON-INFERIORITY VIA RISK DIFFERENCE IN