• No results found

How to collect high quality segmentations: use human or computer drawn object boundaries?


Academic year: 2021

Share "How to collect high quality segmentations: use human or computer drawn object boundaries?"


Loading.... (view fulltext now)

Full text


Boston University



Computer Science CAS: Computer Science: Technical Reports


How to collect high quality

segmentations: use human or

computer drawn object


Gurari, Danna; Wu, Zheng; Isenberg, Brett; Zhang, Chentian; Purwada, Alberto;

Wong, Joyce; Betke, Margrit. "How to Collect High Quality Segmentations: Use

Human or Computer Drawn Object Boundaries?", Technical Report

BUCS-TR-2013-020, Computer Science Department, Boston University, December

23, 2013. [Available from: http://hdl.handle.net/2144/11428]



Boston University Computer Science Technical Report No. BUCS-TR-2013-20

How to Collect High Quality Segmentations: Use Human or Computer Drawn

Object Boundaries?

Danna Gurari, Zheng Wu, Brett Isenberg, Chentian Zhang, Alberto Purwada, Joyce Y. Wong, Margrit Betke


High quality segmentations must be captured consis-tently for applications such as biomedical image anal-ysis. While human drawn segmentations are often col-lected because they provide a consistent level of qual-ity, computer drawn segmentations can be collected effi-ciently and inexpensively. In this paper, we examine how to leverage available human and computer resources to consistently create high quality segmentations. We pro-pose a quality control methodology. We demonstrate how to apply this approach using crowdsourced and do-main expert votes for the “best” segmentation from a collection of human and computer drawn segmentations for 70 objects from a public dataset and 274 objects from biomedical images. We publicly share the library of biomedical images which includes 1,879 manual annota-tions of the boundaries of 274 objects. We found for the 344 objects that no single segmentation source was ferred and that human annotations are not always pre-ferred over computer annotations. Our results led us to suggest a new segmentation approach that uses machine learning to predict the optimal segmentation source and a modified segmentation evaluation approach.

1. Introduction

The ubiquitous use of cameras and the advance in imag-ing technology for medical and biotech applications have resulted in an explosion of the number of images to be an-alyzed. While demarcating the boundaries of objects in the images is commonly important for tasks such as tracking, classification, and shape or behavior analysis, we observe with public datasets that imperfect segmentations can lead to state-of-the-art algorithm performance [1,2,3]. In con-trast, obtaining segmentations that are high quality consis-tently and efficiently is critical for many biomedical appli-cations and algorithm validation.

Segmentation can be a time-consuming bottleneck.

Figure 1. It is unclear which single segmentation collection method will successfully demarcate objects of interest due to the variety of object appearances with respect to intensity, size, and shape; weak edges separating objects from the background; and noisy/cluttered backgrounds.

Careful attention may be necessary to separate complex ob-ject shapes from other obob-jects and/or the background, and domain expertise may be required to understand how object boundaries should be delineated (Fig.1).

Much related work is devoted to obtaining high-quality segmentations. Algorithm development and comparison are performed with the goal of establishing an optimal algo-rithm that works well in general [4,5,6]. Manual annota-tion studies investigate ways to reduce the inter and intra-annotator variability that arises whether collecting segmen-tation drawings from domain experts or the general pub-lic [7]. Ensemble methods combine multiple segmentations created by humans, computers, or a mixture of both to ob-tain a better segmentation [7,8]. The many options neces-sitate a search for the best suited way to produce segmen-tations. Obtaining object boundaries from humans is often preferred because they provide consistent quality, however this process can be time-consuming and expensive.


ing object boundaries from computers is often preferred be-cause they perform the task inexpensively and quickly, how-ever this approach can be unreliable because segmentation quality may vary significantly. Our work examines how to leverage available human and computer resources to consis-tently obtain high quality segmentations.

Our work is partially motivated by the goal of crowd-sourcing the segmentation problem. Recent advances with collecting hand drawn segmentations from large groups of people make it plausible for manual annotations to be a scalable solution to the segmentation problem [11]. This begs the question of whether we prefer human or computer drawn segmentations. Additionally, quality control is an important step in crowdsourcing for filtering out erroneous results coming from unskilled or ill-intentioned humans [9], computers pretending to be humans [10], or confusion aris-ing from subjective problems [9]. Our work provides guid-ance for how to perform quality control and when to prefer human or computer drawings.

Our work is also motivated by a concern about the stan-dard approach for segmentation algorithm evaluation. Al-gorithm generated segmentations are typically compared to references drawn by humans [15,12,13,6,14]. Although this approach was likely motivated by the belief that a hu-man’s visual experience in the world is sufficient training for a person to infer the boundary of an object in an image, it is not clear why a person’s ability to perceive an optimal segmentation should match that person’s ability to draw the segmentation. Assuming algorithms draw worse than hu-mans is akin to assuming that a computer algorithm could not beat a human in a game of chess. Our work explores the impact of this assumption on algorithm benchmarking image sets.

Lastly, we were motivated to fill a gap with public seg-mentation datasets that either address applications where segmentation quality is not critical [12, 13], suffer from scope ambiguity (i.e., different annotators draw segmen-tations at differing granularities) [12], or provide a sin-gle manual annotation per object for algorithm valida-tion [6,14] despite known limitations of annotator variabil-ity.

The key contributions of this paper are:

• A methodology for establishing high quality segmen-tations consistently using consensus voting to perform quality control.

• An empirical study applying the proposed method-ology to 344 objects coming from seven real world datasets for which both human and computer drawn boundaries are candidate optimal segmentations, and analyses on the voting outcomes regarding which sources are preferred.

Table 1. Description of six datasets and annotations in BU-BIL.

ID Imaging Modality # Objects # Annotations 1 Phase Contrast 35 350 2 Phase Contrast 47 470 3 Phase Contrast 34 340 4 MRI 35 350 5 Fluorescence 65 195 6 Fluorescence 58 174

• A discussion of how to apply the proposed method-ology to crowdsource high quality segmentations and improve the segmentation evaluation approach. • A new image library of biomedical images with a

col-lection of expert-drawn and computer-drawn segmen-tations per each of the 274 objects.

The design of our experiments relates to those in human perception experiments for evaluating image compression algorithms [16,17] and comparing the realism of different graphical methods [18,19]. In such work, the testing envi-ronment is often controlled to minimize the psychophysical variations influencing human decisions (i.e., lighting and display quality) [20], ideal representations are available for comparison (e.g., uncompressed image or hand-drawn il-lustrations), and rankings are collected from observers who order a set of images based on specified criteria. Our exper-iments, in contrast, address the segmentation problem, do not constrain the testing environment, lack an ideal refer-ence for comparison, and are based on votes collected from observers choosing the best segmentation.

2. Biomedical Image Library (BU-BIL)

We collected images that were recorded for biology and biomedical research studies at SAGE for which high-quality image segmentations were required. We call this collec-tion the Boston University Biomedical Image Library (BU-BIL). We solicited manual annotations from domain experts and produced computer-generated annotations using vari-ous algorithms. This resulted in a new library of images and annotations that is publicly available at

http://www.cs.bu.edu/∼betke/Biomedical ImageSegmentation.

Image Collection. Our image library includes 235 im-ages, organized in six datasets that represent three imaging modalities and six object types, as summarized in Table1. We instructed the providers of the datasets to choose im-ages that capture the various environmental conditions and imaging noise that arose in their studies. We asked them to select objects from those images that reflect the natural diversity of shape and appearances that these objects can


exhibit. The outcome was 274 annotated objects. We ver-ified by visual inspection that the image library includes a variety of object appearances, backgrounds, and properties distinguishing objects from the background (Figure1).

Collection of Expert-drawn Annotations. We col-lected multiple annotations for each of the 274 objects in our image library as follows. A total of eight domain experts participated as annotators. Some of the annotators were also the creators of the image data. They had a vested interest in the quality of the segmentations they produced because they needed accurate object boundaries for their biomedical re-search studies.

The annotators created segmentations using two com-puter annotation tools, ImageJ [21] and Amira [22]. ImageJ takes as input user specified points and connects them se-quentially with straight lines to produce a 2D segmentation. Amira takes as input user brush strokes to produce a 2D binary mask indicating all pixels in an object. All domain experts had experience with biomedical images and ImageJ. We instructed the annotators to identify the object re-gions using their own judgement. For datasets 1–4, five of the domain experts drew each object region twice, once with each tool. As a result, there were ten annotations for each object. For datasets 5 and 6, the other three domain experts drew the object region only with the annotation tool ImageJ resulting in three segmentation annotations for each object. To avoid ambiguities with detecting the correct object, we extracted image subregions so that there is exactly one dominant object in the foreground. We used one set of man-ual annotations to detect for each image the object location and then grew its bounding box by padding its width in each direction by 25% of the bounding box width and padding its height in each direction by 25% of the bounding box height. Collection of Computer-drawn Annotations. We cre-ated three computer-produced annotations for each of the 274 objects in our image library using publicly available im-plementations of Otsu thresholding[23], the Lankton level sets method [24,27], and the Shi level sets method [25,27]. To run the two level sets method fully automatically, we created the initial contour from which the boundary evolves with the Theriault method [26]. We applied post-processing to fill holes and keep only the largest object in the image. We chose these algorithms to emphasize recent state-of-the-art methods, which are level sets based methods, while in-cluding a commonly used algorithm for fluorescence im-ages, Otsu thresholding.

3. Proposed Methodology for Quality Control

Consistency in collecting high-quality annotations is im-portant in many applications. We here propose a quality control methodology for this purpose that involves a num-ber of independent participants who evaluate the same im-age annotations. The goals are to find (I) the annotation

Figure 2. Proposed quality control methodology. For a given im-age, one collects s annotations, then collects n votes indicating which annotation is the best, and finally recommends an annota-tion to use as the gold standard or declares a tie.

for each image that best serves as a gold standard and (II) for all images tested, the distribution of annotation sources that produced the selected gold standard. We consider the general case that these annotations may be region/object de-tections, segmentations, or class labels, etc. We do not im-pose any restrictions on the source of the annotations or on the participants – both groups may involve computers or hu-mans (the same or different domain experts or non-experts). To address goal (I), we propose to set up a quality con-trol experiment for each image in the library that involves a series of three steps summarized in the flowchart in Fig-ure2: Annotation Step (1): The image is annotated by s algorithms, domain experts, or non-experts. Voting Step (2): The annotations are inspected by n algorithms, domain experts, or non-experts which/who each vote for the best annotation. Decision Step (3): A decision mechanism in-terprets the votes of the s annotations based on the results of the n inspections and recommends an annotation to be selected as a gold standard or declares a tie. A simple ex-ample of a decision process is majority voting. The output of the experiment, for each image, is the recommended gold standard or a recommendation for further inspection due to a tie. To address goal (II), we propose to repeat the experi-ment for each image in the library and tally the number of times each annotation source produced the gold standard.

3.1. Segmentation Quality Analysis Software

To support the Voting Step in the proposed quality control methodology (Fig.2), human-computer interaction software is needed to present image annotations to partici-pants of the experiment for inspection, comparison, and vot-ing. In this paper, we are concerned with evaluating image segmentations, so we chose the publicly available software SAGE[7] for participants of our experiments to inspect and vote between image segmentations. SAGE can present si-multaneously as many human or computer segmentations as the user chooses to evaluate. Each segmented region is shown overlaying the original image. If the user wants to compare two segmentations A and B quantitatively, the user can use SAGE to compute their similarity using the over-lap ratio, which is a standard segmentation evaluation mea-sure [15]. This measure calculates the ratio|A∩B||A∪B|by count-ing the number of pixels in common (i.e., A ∩ B) from all pixels included in A and B (i.e., A ∪ B).


3.2. Experimental Design Methodology

In this paper, we are particularly interested in which is the “highest quality” segmentation and whether the source of a “best” annotation is a human or a computer. We there-fore use the methodology introduced above for the case where the annotation sources are separated into two groups depending on the goal: segmentation of interest A and seg-mentations not of interest 6A or humans H and computers C. Using the H versus C scenario as an example, the bino-mial coefficient nk can be used to count the number of dif-ferent ways k selections of H can occur in a sequence of n independent inspections. If there are as many human-drawn annotations as there are computer-generated, the likelihood that any one of these ways are selected by chance is 0.5n. If the ratio of the number of human-drawn annotations to the number of computer-drawn annotations C is p, the likeli-hood that k selections of H occur by chance is pkand n − k selections of C occur by chance (1 − p)n−k. The binomial distribution

f (k; n, p) = ( n! k!(n − k)!)p

k(1 − p)n−k

(1) is therefore the appropriate distribution to express the prob-ability that k of n annotation sources in group H are se-lected by chance. The frequency with which voters in an experiment select a particular k-combination of annotation sources in group H can be used to decide whether a selection occurs by chance or intention. This same approach would be applied to decide whether to trust the majority vote win-ner annotation A as a gold standard segmentation.

The distribution in Eq.1is important to consider when designing experiments because it reveals the statistical im-portance of voting outcomes and enables educated decisions regarding whether to trust or be “suspicious” of voting out-comes. For example, when collecting votes through crowd-sourcing, voters may not necessarily be trustworthy and so one may not want to trust all majority vote outcomes. Sim-ilarly, disagreement between trusted voters may be cases to set aside for follow-up if such decisions are life critical, as can be the case for medical applications.

4. Quality Control Experiments

We perform quality control using segmentations care-fully drawn by hand and segmentations drawn by top per-forming algorithms. We used the experiments to examine the outcome one may expect for human versus computer performance for everyday and biomedical images.

4.1. Baseline Experiments

We designed the first experiment to examine the baseline difficulty for humans to generate high quality annotations, by using black and white synthetic images where the true segmentation is known. We also discuss challenges that are

introduced by performing quality control on real world im-ages instead of binary imim-ages.

Experimental methodology: We chose 15 synthetic black-and-white images containing a single object from the MPEG-7 classification dataset [28] (Figure1). We in-structed a professional medical graphics illustrator to draw the object boundary for as many images as possible in one hour with the goal to draw a better boundary than a com-puter. The illustrator created the segmentations using a touchpad with pen in the software Adobe Photoshop. The computer drawn segmentations were created using Otsu thresholding [23].

Results: The illustrator annotated three images in one hour. The overlap ratio for the human-drawn annotations of images showing an apple, cup, and fish was 0.7771, 0.9974, and 0.9974 respectively and for the computer-generated an-notations was 1.0 for each of the 15 images.

Discussion: Results show that the computer generated higher quality segmentations than the human. An interest-ing question is why the human performed worse since there were no perceptual questions of which pixels are part of the object versus background. We learned that the illustrator used a paintbrush tool for the apple image. Possible causes of error included the thickness of the drawing tool and hand jitter. For the other two images, we learned that the illus-trator zoomed in to observe the individual pixels and then marked each pixel along the boundary. These results exem-plify the level of image quality obtained when a highly qual-ified expert dedicates large amounts of time and painstaking attention to the task of accurately capturing object bound-aries. If a highly qualified expert does not even provide perfect segmentation results in this baseline experiment, it is unlikely that non-experts could draw perfect outlines in the same time frame. This suggests that human-drawn ob-ject boundaries in more complicated experiments will al-ways be imperfect, whether coming from domain experts or crowdsourcing. The results of this experiment motivate us to learn when, in practice, to prefer humans or computers to draw segmentations.

This experiment does not capture all the challenges that arise when performing quality control for real world im-ages. First, an image pixel may represent both an object and background as a result of digitization making it ambiguous whether to assign the pixel as object or background. More-over, even if this were not the case, it is often impossible to know the true segmentations for natural images. Finally, even if it was possible to know the true segmentations, a human’s decisions may be influenced by psychophysical factors such as the display settings or the environmental lighting. The aforementioned concerns expose the fact that segmentation evaluation relies on human decisions to deter-mine what pixels count as foreground/background and so can be subjective. We address this concern in subsequent


experiments by applying the methodology proposed in Sec-tion3using consensus from multiple voters to identify the “best” segmentation.

4.2. Experiment on Everyday Images

We implemented the quality control methodology using a disproportionate number of human and computer drawn segmentations while crowdsourcing votes. We used a pub-lic dataset collected by the Weizmann Institute [13] which highlights the generality of our findings. The 70 images we selected typically show one object of interest that dominates the image, for example, a bird, tree, or balloon as shown in Figure 1. The outline of the object of interest is thus the coarse-grained granularity of the desired segmentation.

Experimental methodology: For each image, we asked five individuals from the general public to vote on the mentation best representing each object region from six seg-mentation options shown simultaneously. One segmenta-tion was a manual annotasegmenta-tion provided by the Weizmann Institute for dataset benchmarking. We created the other five segmentations using five publicly available algorithms. We used Hough Transform [29], Bernard level sets meth-ods [30,27], Li level sets method [31,27], Shi level sets method [25,27], and Caselles level sets method [32,27], followed by post-processing to fill holes and keep only the largest object. For the level sets methods, we created the initial contours from which the boundary evolves using the Theriault method [26]. All voters were presented the orig-inal images in the same order. To prevent biases of voters possibly learning the algorithmic or manual source of the segmentations, the order of segmentations presented by the user interface was randomized for each object.

The votes were used for quality control to establish high quality segmentations and to analyze when human or com-puter drawn segmentations are better. For quality control, we first determined whether to be suspicious or trusting of the voting outcomes, using Eq. 1, since voters may arbi-trarily vote without consideration of which segmentation is the best. We calculated the probability of observing k = {0, 1, ..., 5} human votes for a particular segmentation, where n = 5 voters and p = 1/6 probability of picking that segmentation by chance. Then, we selected a probability as the threshold that determines if we conclude the outcome is suspicious (i.e., arose by chance) or trusted. We analyzed the voting outcome for three different thresholds. For ana-lyzing when human or computer drawn segmentations are better, we counted the number of voting outcomes based on the number of votes for a human drawn segmentation (H) versus a computer drawn segmentation (C).

Results: Voting time for the five individuals ranged be-tween 15 and 30 minutes each.

For quality control, we tallied the voting results based on the number of votes for each of the segmentation sources.

Figure 3. The number of “trusted” voting outcomes for each of the six sets of annotations for 70 images in the Weizmann image library for three scenarios: trust majority vote (i.e., 2+ votes), 3+ voting agreements, and agreement between all 5 votes.

From the 70 images, the majority vote arose by consensus from all five voters for 44 images, four voters for 12 ages, three voters for 12 images, and two voters for two im-ages. We found no cases of no consensus. Using Eq.1, we calculated that outcomes with at least k = 2 voting agree-ments are likely to arise by chance <20% of the time, at least k = 3 voting agreements are likely to arise by chance <4% of the time, and k = 5 voting agreements are likely to arise by chance <1% of the time. We report the distribution of voted high quality segmentation sources when using each of these three thresholds to separate suspicious from trusted outcomes in Figure3. Example images showing when hu-man or computer drawn segmentations were preferred are shown in Figure4.

Discussion: We observed that computer-drawn segmen-tations are perceived by the majority as higher quality than hand-drawn gold standard segmentations used for algorithm validation for 9 out of the 70 images (i.e., nearly 15% of im-ages). One may be concerned that such results arise because voters chose between a disproportionate number of human and computer drawn segmentations and so outcomes were biased. However, we demonstrate by using equation1 to interpret the results statistically that, even as we apply more stringent rules to distinguish between suspicious and trusted voting outcomes with the most extreme scenario being to trust only unanimous agreement, we continue to observe this preference for a combination of computer and human drawn segmentations. We infer from these results that fu-ture work should predict when to use humans and comput-ers since successfully doing so for similar everyday images and annotations would lead to higher quality segmentations while reducing human involvement for drawing.

We suggest using equation1to guide the quality control experimental design by choosing a threshold to distinguish between suspicious and trusted voting outcomes and then determining the minimum number of voting agreements


Figure 4. Qualitative human versus computer voting results. Each row shows the image followed by segmentations created by a hu-man, Bernard level sets algorithm, Li level sets algorithm, Shi level sets algorithm, Caselles level sets algorithm, and Hough Trans-form. Darker gray level shadings indicate more votes for the seg-mentation. The voting outcomes are a unanimous win for the hu-man in row 1, a unanimous win for the computer in row 2, a ma-jority vote win for the computer in row 3, and a mama-jority vote win for the human in row 4.

(i.e., k) that leads to trusted outcomes. We observe a trade-off between minimizing the chance for error and minimiz-ing the number of suspicious outcomes. Applications where object shape is critical, such as when shape influences med-ical diagnoses, should use larger levels of voting agreement to minimize the chance of making a mistake. Applications where object shape is not as critical, such as for tracking or training a machine learning algorithm, should use smaller levels of voting agreement to minimize the number of sus-picious results. Future work will examine how to handle suspicious outcomes.

An interesting question is what to infer about segmenta-tion quality when using voter consensus to perform quality control. Voting forces the possibly many observed imper-fections throughout a segmentation to be collapsed into a single assessment. We observed that the winning segmenta-tion could appear both flawed and accurate (Figure4, rows 1 and 2). Similarly, we observed that voting disagreement arose when multiple segmentations appeared accurate or all segmentations appeared flawed and different judgements were made when voting for the best (Figure4, rows 3 and 4). Human drawn segmentations provided a consistent level of quality while computer drawn segmentations could pro-vide a higher level of quality when they were successful. Future work will involve a larger-scale study of the factors that influence the voting outcomes in order to provide feed-back for how to handle ties and advise how humans and computers could produce higher quality annotations.

4.3. Experiment on Biomedical Images

We designed the third experiment to analyze the impact of performing quality control when human and computer drawn boundaries are considered for applications involving

domain experts whose studies are based on shape analysis. Experimental methodology: We recruited three highly-trained domain experts for our experiment. For each of the 274 images of objects of interest in the BU-BIL, we simul-taneously showed six segmentation options that were ob-tained for the BU-BIL. We selected three of the six options to be human expert annotations and the other three options the computer-drawn boundaries, in order to have a balanced number of options representing both human and computer sources. The three experts recruited for this experiment were instructed to vote, for each image in the library, which segmentation option they thought best represented the ob-ject region from the six choices. As in the previous exper-iment, all voters were presented the original images in the same order. The order of segmentations presented by the user interface was randomized for each each image to pre-vent biases of voters possibly learning the algorithmic or manual source of the segmentations.

These votes were used for quality control to establish gold standard segmentations and to analyze when human or computer drawn segmentations are better. For quality con-trol, we use the majority vote to determine the gold standard and, when there was no consensus, a human drawn segmen-tation was chosen when an option was available. For ana-lyzing when human or computer drawn segmentations are better, we counted the number of voting outcomes based on the number of votes for a human drawn segmentation (H) versus a computer drawn segmentation (C).

Results: The three voters reported that voting took ap-proximately 55 minutes, 2 hours, and 5 hours.

For quality control, we tallied the voting results based on the number of votes for each of the segmentation sources. From the 274 images, the gold standard was selected by consensus from all three voters for 50 images, two voters for 161 images, and no consensus for 63 images. From the 211 instances of consensus, the distribution of gold standards coming from the six sets of annotations is 84 for the first set of manual annotations, 67 for the second set of manual annotations, 47 for the third set of manual annotations, 1 for Otsu Thresholding, 10 for Lankton level sets method, and 2 for Shi level sets method.

We also tallied the voting results based on the number of votes for a human drawn segmentation (H) versus a com-puter drawn segmentation (C). From the four possible vot-ing outcomes (votvot-ing order is ignored), we found the fol-lowing: 184 HHH, 67 CHH, 19 CCH and 4 CCC. We show quantitative voting outcomes for each dataset in Figure5. We observe that computer drawn segmentations are accept-able annotations based on imaging modality for over half of the MRI images (i.e., 20/35 ≈ 57%), nearly half of the fluorescence images (i.e., 57/123 ≈ 46%), and nearly 9% of the phase contrast images (i.e., 13/151). This is based on the assumption that all voters are trusted and so a single


Figure 5. The number of votes from three voters for each dataset in the BU-BIL based on the number of votes for human (H) and computer (C) drawn segmentations.

vote suggests a particular segmentation is an acceptable op-tion. If considering a majority vote, computer drawings are preferred for approximately 31% of the MRI images (i.e., 11/35), 8% of the fluorescence images (i.e.,10/123), and un-der 2% of the phase contrast images (i.e., 2/151).

Discussion: We observe that consensus voting led to gold standard segmentations for 77% of the images with no preference for a single annotator or algorithm. More-over, we observe that computer generated segmentations account for 5% of the images. Although a smaller percent-age than for the previous experiment, this result supports the observation that higher quality segmentations are ob-tained when considering both human and computer drawn segmentations. We believe there would be a higher pref-erence for computer drawn segmentations when collecting segmentations for applications since we used multiple an-notations collected for benchmarking purposes by highly trained experts.

We infer from the results that a promising research direc-tion for similar fluorescence and MRI images would be to build a classifier to automatically predict when to use com-puter drawings since it would reduce hand drawing time by approximately 50% for similar images and annotations. We infer future work for phase contrast images would be better targeted towards designing semi- and fully-automated algo-rithms since not much benefit would be reaped from divid-ing the annotation effort between humans and computers.

5. Conclusions

We proposed a quality control methodology based on consensus voting to determine the optimal source to use to solve a problem. Experiments applying this methodology to the image segmentation problem showed that overall quality improved for 344 objects, whether using domain experts or crowdsourcing to vote for the best source from a collection of computer and human drawn segmentations. Analyses showed that a single segmentation source was not always preferred and human drawn segmentations were not always preferred. We suggest to use a quality control step to estab-lish gold standard segmentations. Possible future research

directions include analyzing the strengths and weaknesses of humans and computers for creating segmentations and building classifiers to predict from the raw image when to use which segmentation source.


The authors gratefully acknowledge funding from the Na-tional Science Foundation (IIS-0910908).


[1] A. Torralba, R. Fergis, and W. T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Trans. Pattern Anal. Mach. Intell., 30(11): 1958-1970, 2008.1

[2] M. Rohrbach, S. Amin, M. Andriluka, and B. Schiele. A database for fine grained activity detection of cooking activ-ities. Proc CVPR IEEE, 11194-1201, 2012.1

[3] M. Everingham, L. V. Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The Pascal Visual Object Classes (VOC) Challenge. Int J Comput Vis, 88(2):303–338, 2010.1

[4] L. He et al. A comparative study of deformable contour methods on medical image segmentation Image Vision Com-put, 26(2):141-163, 2008. 1

[5] A. J. Noble and D. Boukerroui. Ultrasound image segmen-tation: a survey. IEEE Trans on Med Imag, 25(8):987-1010, 2006.1

[6] L. P. Coelho, A. Shariff, and R. F. Murphy. Nuclear Seg-mentation in Microscope Cell Images: A Hand Segmented Dataset and Comparison of Algorithms I S Biomed Imaging (ISBI), 518–521, 2009.1,2

[7] D. Gurari et al. SAGE: An Approach and Implementation Empowering Quick and Reliable Quantitative Analysis of Segmentation Quality. IEEE Workshop on Applications In Computer Vision (WACV), 475–481, 2013.1,3

[8] S. Ghosh, J. Pfeiffer, and J. Mulligan. A framework for rec-onciling multiple weak segmentations of an image. IEEE Workshop on Applications In Computer Vision (WACV), pages 1–8, 2009.1

[9] J. Deng, J. Krause, and L. Fei-Fei. Fine-Grained Crowd-sourcing for Fine-Grained Recognition. Proc CVPR IEEE, 2013.2

[10] L. V. Ahn., B. Maurer, C. McMillen, D. Abraham, and M. Blum. reCAPTCHA: Human-based Character Recogni-tion Via Web Security Measures. Science, 321(5895):1465– 1468, 2008.2

[11] B. M. Good and A. I. Su. Crowdsourcing for Bioinformatics. Bioinformatics, 29:1925–1933, 2013.2

[12] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Eco-logical Statistics. Proc. 8th IEEE I Conf Comp Vis, 2:416– 423, 2001.2


[13] S. Alpert, M. Galun, R. Basri, and A. Brandt. Image Seg-mentation by Probabilistic Bottom-Up Aggregation and Cue Integration. Proc CVPR IEEE, 2007.2,5

[14] V. Ljosa, K. L. Sokolnicki, and A. E. Carpenter. Annotated high-throughput microscopy image sets for validation. Na-ture Methods, 9(7):637, 2012.2

[15] Y. J. Zhang. A survey on evaluation methods for image seg-mentation. Pattern Recognit., 29(8):1335–1346, 1996. 2,


[16] C. S. Stein, A. B. Watson, and L. E. Hitchner. Psychophys-ical rating of image compression techniques. Proc. SPIE, 1077:198–208, 1989.2

[17] M. P. Eckert and A. P. Bradley. Perceptual quality met-rics applied to still image compression. Signal Processing, 70(3):177–200, 1998.2

[18] R. McDonnell, M. Breidt, and H. H. Bulthoff. Render me real? investigating the effect of render style on the perception of animated virtual humans. TOG, 31(4):91, 2012.2

[19] T. Isenberg, P. Neumann, S. Carpendale, M. C. Sousa, and J. A. Jorge. Non-photorealistic rendering in context: an ob-servational study. In Proc of the 4th international sympo-sium on non-photorealistic animation and rendering, 115-126, 2006.2

[20] S. Winkler. Issues in vision modeling for perceptual video quality assessment. Signal Processing, 78(2):231–252, 1999.2

[21] W. Rasband. ImageJ, 1997–2012. U.S. National Institutes of Health, Bethesda, Maryland, USA.3

[22] Amira, software platform for visualizing, manipulating, and understanding life science and bio-medical data. Retrieved August 17, 2012, from http://amira.com.3

[23] N. Otsu. A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man, Cybern. B, Cybern., 9(1):62–66, 1979.3,4

[24] S. Lankton and A. Tannenbaum. Localizing region-based active contours. IEEE Trans. Image Process., 17(11):2029– 2039, 2008.3

[25] Y. Shi and W. C. Karl. A real-time algorithm for the ap-proximation of level-set based curve. evolution. IEEE Trans. Image Process., 17(5):645–656, 2008.3,5

[26] D. Theriault, M. Walker, J. Wong, and M. Betke. Cell morphology classification and clutter mitigation in phase-contrast microscopy images using machine learning. Mach. Vis. Appl., 23(4):659-673, 2011.3,5

[27] T. Dietenbeck, M. Alessandrini, D. Friboulet, and O. Bernard. Creaseg: A free software for the evaluation of image segmentation algorithms based on level-set. IEEE Im-age Proc (ICIP), pIm-ages 665–668, 2010.3,5

[28] MPEG-7 Core Experiment CE-Shape-1 Dataset. Retrieved August 2013, from http://www.dabi.temple.edu/ ˜shape/MPEG7/dataset.html.4

[29] D. Ballard. Generalizing the Hough transform to detect arbi-trary shapes. Pattern Recognit., 13(2):111–122, 1981.5

[30] O. Bernard, D. Friboulet, P. Thevenaz, and M. Unser. Vari-ational b-spline level-set: A linear filtering approach for fast deformable model evolution. IEEE Trans. Image Process., 18(6):1179–1191, 2010.5

[31] C. Li, C. Y. Kao, J. C. Gore, and Z. Ding. Minimization of region-scalable fitting energy for image segmentation. IEEE Trans. Image Process., 17(10):1940–1949, 2008.5

[32] V. Caselles, R. Kimmel, and G. Sapiro. Geodesic Active Contours IEEE Trans. Image Process., 22(1):61–79, 1997.5


Related documents

Moreover, numerical solution and corresponding contour plot of the present method for the same problem parameters obtained with unstructured grid points are illustrated in Fig...

sive  mothers,  those  with  U.S.-born  parents  had  greater  self-esteem 

Currently, payment transaction services from Mobile Banking OCBC NISP still use basic banking service features, namely Purchase and Biller Transactions, Payment, Virtual

This is the aim of the current thesis to extend literature on Albanian consumers and more specifically their perceptions of food outlets, more specifically of

Additionally, hypoxia induced accumulation of primiRNAs 200b/c in all cell lines compared to normoxia condition ( Fig. Similarly, hypoxia induced reduction of their mature forms.

Anne’s advice Try these exercises when you have plenty of time on your hands and don’t need to climax urgently. Slow intercourse like this can be viewed as a kind of hobby

Throughout this pro­ cess, we will see trivial integrals that can be evaluated using basic techniques of integration (such as integration by parts), however we will also

The study was significant to university curriculum developers, instruction department leaders, and high school teachers in rural settings, including the local educational setting