5.4.1
Behavioral Experiment
Kapelner and Chandler (2010) (Chapter 3 of this document) ran experiments using the Amazon Mechanical Turk platform, a global outsourcing website for small one-off tasks that can be completed anonymously on the Internet. They focused on mea- suring subjects’ stated preference for a beer price when the beer came from different
purchasing locations (an online replication of Thaler, 1985’s demonstration of the “framing effect,” a cognitive bias). The treatment involved subtle text manipula-
tions: the same beer came from either afancy resort or a run-down grocery store. In
their control wing (n = 168), no tricks were employed to ensure the subjects were
paying attention to the text. Thus, in this wing, the subtle text manipulation did not seem to affect the subjects’ stated beer prices. The effect may have been real, but the data was either too noisy or there was insufficient sample size to find it. We demonstrate here that if our sequential matching procedure was employed, the effect estimator would have been more efficient.
For matching, we first used most of the covariates found in the original dataset: age, gender, level of earnings, number of weekly hours spent doing one-off tasks, level of multitasking when performing tasks, stated motivation level, passing the “instructional manipulation check” (Oppenheimer et al., 2009) and a survey gauging the subject’s “need for cognition” (Cacioppo and Petty, 1982).
We note that R2 under OLS was about 18.7%. We then run two off-the-shelf
machine learning algorithms that are designed to find interactions and non-linearities
in the response function. The in-sample pseudo-R2 using Chipman et al. (2010)’s
Bayesian Additive Regressive Trees (BART) was 42.4% and Breiman (2001b)’s Ran- dom Forests (RF) was 70.4%. Although this is not a formal test, it is pretty com- pelling evidence that the covariates do not combine strictly linearly to inform beer price. Thus, as demonstrated in figure A.17a and column 4 of table 5.2, our method should be more powerful and more efficient than using previous dynamic allocation
strategies with a classic estimator. The results for 200 simulations at λ = 0.10 are
shown in table 5.4a. Many of the covariates are binary. Thus, the variance-covariance
matrix was not invertible in line 5 of algorithm 1 for many of the early iterations, so
we used the Moore-Penrose generalized inverse instead.
average average approx. sample purported n actualn efficiency size reduction
(a) 50 37.8 1.84 45.7% 100 71.9 1.23 16.9% 168 (all) 116.1 1.06 5.4% (b) 50 34.9 2.01 50.1% 100 67.8 1.60 37.3% 168 (all) 112.1 1.57 36.3%
Table 5.4: Results for 200 simulations of the sequential matching procedure over
many values of n and λ = 0.10. (a) all covariates matched on (b) four cherry-picked
covariates are matched on (OLS has an R2 = 20.8%, BART, 32.1% and RF, 26.5%).
Note that many of the covariates were binary, and thus the variance-covariance matrix
was not invertible in line 5 of algorithm 1 for many of the early iterations of these
simulations, so we used the Moore-Penrose generalized inverse.
OLS regression on the full dataset: age, level of earnings, level of multitasking when performing tasks and one question from the survey gauging the subject’s “need for cognition.” The results are found in table 5.4b. Note that the efficiencies are higher
and do not drop off as quickly whennincreases. Thus, matching onrelevant covariates
yields a performance enhancement in our procedure.
5.4.2
Clinical Trial
We now examine sequential clinical trial data from Foster et al. (2010), a twelve- week, multicenter, double-blind, placebo-controlled clinical trial studying whether amitriptyline, an anti-depressant drug, can effectively treat painful bladder syndrome. The study measured many outcomes, including change in pain after 12 weeks (dif- ference in Likert scale scores). The confidence interval for the ATE between pill and
placebo for this outcome measure was [−1.00,0.30] with apvalue of 0.29 (Table 2, row 1, page 1856). Such results indicate the effect may have been real but there wasn’t enough power to detect it due to low sample size or a high error variance.
For matching, we first use most of the covariates found in the original dataset: age, gender, race (White / Hispanic), level of education, level of employment, living with a partner, presence of sexually transmitted diseases and urinary tract infection, as well as baseline measures of pain, urination frequency and urgency, quality of life, anxiety and depression as well as syndrome symptom levels.
R2 was about 25.2% in an OLS regression. We then run two off-the-shelf machine
learning algorithms that are designed to find interactions and non-linearities in the
response function. The in-sample pseudo-R2 using Chipman et al. (2010)’s Bayesian
Additive Regressive Trees (BART) was 42.6% and Breiman (2001b)’s Random Forests (RF) was 82.4%. Although this is not a formal test, it is pretty compelling evidence that the covariates do not combine strictly linearly to inform change in pain after 12 weeks. Thus, as demonstrated in Figure A.17a and column 4 of Table 5.2, our method should be more powerful and more efficient than using previous dynamic allocation strategies with a classic estimator.
We simulate the subjects being dynamically allocated using the sequential match- ing procedure by first assuming the entering subjects do not exhibit any time trend; this will allow us to permute their order. During the iterative procedure, all subjects assigned to the reservoir keep whichever assignment they had during the experiment. During matching, if the subject happened to have been assigned the treatment which the sequential matching procedure allocated, they are kept in the subject pool; if not, they are discarded (this is illustrated in Figure 5.2). Thus, during our simulations,
we result in a subset of the data we began with. Note that we only show results
for the classic estimator versus the modified estimator in Equation 5.4, not the OLS modified estimator whose results we suspect to be similar.
nsim: 1 ...o.xx....o....xxx.ooxx.o.xxooxooxx.xo. (37) nsim: 2 ...xxx..x.xxoo.xxx.o.o.oo..ox.xxxxx.o (35) nsim: 3 ...x.x..oo.xxxo.x.xxxxxxxo.x.ox.oox.o (34) nsim: 4 ...o...o...ox.o.o.xo.ox.o....xxooox.oox.xo (42) nsim: 5 ...x.xoo.o.o..xxoxx.o.x.x.oo.o.xoxx.. (39) nsim: 6 ...x..xx.x..xx.oo.xoxoxxx.ooxo.o.xxxo (35) nsim: 7 ...x.o..oo..oxx.x..o.oxo.xxox.o.xoxxxoxo (37) nsim: 8 ...xx.oxoxxo..x.xxx.x..oxoo.o.xx.x..xox.o (34) nsim: 9 ...o...xxoo.xo.ooo...o.ooxox.oxx..o.x (42)
Figure 5.2: Running an n = 50 subset of historical data through the sequential
procedure. The dots represent a subject being placed into the reservoir. The “o”
signifies that the subject was matched and that their treatment allocation wasopposite
of their matching partner. The “x” signifies that the subject was matched but their
treatment allocation was thesame as their matching partner, resulting in the subject
being discarded. The number in parenthesis at the end of the line is the sample size retained of the purported 50.
The results for 200 simulations are shown in Table 5.5a. We also match on the top four covariates which are the most significant in an OLS of the full dataset: living with a partner, baseline pain, frequency of pain and syndrome symptom levels. The results are found in Table 5.5b. Note that these efficiencies are higher when compared
to matching on all covariates and they do not suffer the steep drop off as n increases.
5.5
Discussion
Estimation in sequential experiments, of which many are clinical trials, can have higher power and efficiency if the covariate information is leveraged. We present a dynamic allocation of treatment and control that matches subjects on-the-fly via a novel algorithm and present modified estimators of classic approaches: average difference, linear regression, and permutation testing. We simulate under different scenarios and illustrate higher power in scenarios where competing methods cannot
average average approx. sample
purportedn actual n efficiency size reduction
(a) 50 38.9 1.30 23.0% 100 75.2 1.10 9.2% 150 111.3 1.05 4.9% 224 (all) 165.5 1.07 6.7% (b) 50 38.0 1.27 21.2% 100 72.9 1.23 18.8% 150 108.6 1.15 13.2% 224 (all) 160.3 1.13 11.3%
Table 5.5: Results for 200 simulations of the sequential matching procedure over
many values of n and λ = 0.10. (a) all covariates matched on (b) four cherry-picked
covariates are matched on (OLS has an R2 = 18.9%, BART, 35.1% and RF, 45.0%).
Note that many of the covariates were binary, and thus the variance-covariance matrix
was not invertible in line 5 of algorithm 1 for many of the early iterations of these
simulations, so we used the Moore-Penrose generalized inverse.
make proper use of covariate information. We underperform only in the case of low sample size when the covariate model is linear or non-existent. In simulations with real clinical data, we find the efficiency of our method over complete randomization increases as the covariates become more important. This is most likely due to the fact that real-world response functions, such as this one in the clinical setting, combine covariates non-linearly, and this is when our procedure is most advantageous.
We note that “analysis assumptions may be compromised due to the ‘pseudo’- random allocation” (Scott et al., 2002) and would like to address this criticism which can be made about our procedure. Note that in Table 5.3, the size of the tests under stratification and minimization are less than 5%. One should not use the classic
estimator in these cases because one implicitly tried to balance on the covariates, but then did not include the covariates in the model (as seen by the poor sizes using the classic model in scenario NL using the classic model in scenario LI in Table 5.3). Simon (1979) and Senn (2000) have very good discussions about this issue and recommend using regression adjustment (as seen in the competitors in scenario
LI, linear response model in the Table 5.3 for n > 50). As for exact testing under
dynamic allocation, Kalish and Begg (1987) and many others warn that permutation distributions in stratification and minimization are incorrect unless the investigator permuted the treatment allocation according to how the allocation was determined by the covariates. This is not straightforward in practice for stratification and even less straightforward for minimization.
In contrast, the sizes of the test for our approach seem to be correct in Table
5.3, especially when making use of the low-sample T approximation. Usage of our
classic estimator seems ill-advised for the same reasons that the classic estimator is not recommended under stratification and minimization. However, using linear re- gression on the covariates is also ill-advised when the model is non-linear or otherwise does not satisfy the OLS model assumptions (Freedman, 2008) although Rubin (1979) finds that covariance adjustment of matched pair differences is robust to model mis- specification. Thus, since our permutation test performs well in the non-linear case, properly sized and can avoid many of the above issues, we recommend the permutation test in practice. We permuted according to the structure of our dynamic matching allocation, thus our permutation tests are valid.
5.5.1
Further Developments
We view this contribution as a step forward in covariate-adaptive randomization in sequential experiments but it is far from complete. We list extensions below which
we believe are in difficulty order.
Sequential Analysis Although we assume fixed n in our construction, it is rela-
tively straightforward to adapt to a fully or group sequential design whose methods can be found in Jennison and Turnbull (2000). It would be hard to tabulate val- ues when our estimator has unknown convergence properties, thus it would probably have to be done by waiting until the estimator most likely converged, and then using standard sequential analysis software.
Multi-armed designs This paper is devoted to studies where there are only two
arms. If the study is multi-armed, the approach can be modified. Upon matching, the newly matched subject could be randomly assigned to a random treatment (an arm other than the treatment of the reservoir subject) or perhaps, alternatively, assigned to one of the other arms to best achieve balance. Analagous estimators can then be created for contrasts of interest.
Non-continuous outcomes We believe with adjustment of the estimators, our
method can apply beyond continuous responses to binary, ordinal, or count responses.
Imbalance protection Complete randomization when allocating subjects in the
reservoir may result in treatment imbalances in the reservoir after the experiment is completed. In clinical trials, block permutations are popularly employed to avoid imbalance. In our case, blocks for reservoir allocation would not work since, upon matching, the block sequences would be broken. A solution would be to use a biased coin allocation scheme (Efron, 1971) or an urn model. This would involve modification of the exact test of Section 5.2.3.3 to reflect the restricted randomization scheme.
Multi-centered designs Clinical trial recruitment is frequently spread across mul-
in two ways to be of use in multiple centers. First, if large enrollment is expected in each center, the procedure can run independently in each center and resulting es- timates from each center can be combined upon study completion. Second, if small enrollment is expected in each center, the subjects can be matched across centers and the least squares estimator of section 5.2.3.2 can be used to adjust for center-center variation.
Better matching distance We feel that the most significant improvement would
be better matching. Mahalanobis distance is logical, but prone to strange behavior with departures from the normality assumption. Another natural extension is to bootstrap the distribution of the nominal metric in Equation 5.2 as to not rely on
probabilities from the scaled F distribution. Also, practitioners may want to weight
the variables in the matching as well as force some variables to always match (see Rosenbaum, 2010, chapter 8).
On-the-fly variable selection Of course, our procedure suffers from the central
issue of all matching: selecting the variables to match on. A poor choice makes a big difference as evidenced by the simulations on clinical trial data (Table 5.5a vs. 5.5b).
There may be a way to iteratively match and differentially weight by covariates that
are found to be important, so the set of perceived important covariates is updated during the sequential experiment.
Reservoir optimization We have begun to consider how largenRgrows asymptot-
ically as a function ofn0 andλ(Section 5.2.4). To understand the optimal tradeoff of
n0 andλto maximize estimator efficiency as functions ofn,p, the variance-covariance
matrix of the covariates, and how strong the signal of f is to the noise will involve
a lot of theory. Also, perhaps a variable rule for λ would be effective: if the sample
the beginning of the experiment, but then become less conservative as time passes.
Waiting until n0 is almost half n can be another strategy.
Supplementary Materials
Original data and source code for reproduction can be found atgithub.com/kapelner/
sequential_matching_simulations_2.
Acknowledgements
We wish to thank Larry Brown, Dean Foster, John Horton, Stephen Kapelner, Katherine Propert, Paul Rosenbaum, Andrea Troxel, and the reviewers for helpful suggestions and Hugh MacMullan for help with grid computing. Adam Kapelner also acknowledges the National Science Foundation for his graduate research fellowship.
6
Estimating the Number of Objects in Images
∗Abstract
We develop a method to estimate the count of objects or features in an image such as tallying the number of birds in a photograph or the the number of cells in a microscopic image. Our approach has two novel steps. We first develop software that records the labelings of many naive workers via Amazon’s Mechanical Turk. We then view each worker’s training as a “capture” in a set of capture-recapture experiments. We use statistical learning to eliminate falsely-trained objects and then take a Bayesian approach and use a Gibbs sampler to estimate the true number of objects.
6.1
Introduction
Counting and locating specific features in an image has many applications such as cell counting, harvest estimation, and crowd counting. Standard methods in counting (how many words did Shakespeare known? (Efron and Thisted, 1976) and animal
abundance) use models where the frequencies of counts form sufficient statistics. How- ever, with large amounts of noisy data, such models have to be refined. In this paper, we develop a two-stage solution to the counting and localization of objects in images. Some images feature objects that are easy to locate, thus the count is obvious; other images are difficult featuring ambiguous and obfuscated objects, this is where our solution can be applied.
In section 6.2 we give an introduction to object identification in the field of com- puter vision and our previous research that served as the inspiration for this project.
In section 6.3 we talk about the engineering effort of our new tool, DistributeEyes,
that uses crowd labor to label images. The use of this new tool has many applications, but we focus here on improving the estimate of the number of objects in an image.
In section 6.4 we talk about statistical methods that use data from DistributeEyes
to estimate the number of objects in images using Bayesian capture-recapture meth- ods. In section 6.5 we run an experiment with a wide variety of images, and test our count estimation method on the experimental data. We are also able to present a quality control analysis and present observations about spatial and temporal patterns in workers’ training points. We conclude and talk about future directions in the final section 6.6.