International Challenges and Datasets

Below we will give a brief overview of the most famous datasets and international challenges that see them involved. We will only analyze datasets in the object detection field, as this deep learning project falls into such field. Dedicated datasets in the retail systems analytics field will be treated later in the chapters that describe the system model.

Every year new datasets over specific contexts are created; new international challenges arise and researchers from all over the world challenge each other to design cutting-edge targeted solu- tions. The major international challenges in computer vision are mainly focused on topics such as image classification, object detection and semantic segmentation. Parallel to the main tasks, new challenges arise that integrate the same tasks in different contexts. This is the case for instance segmentation, image captioning and many other challenges.

Currently, the main datasets that see the greatest number of international challenges derive from a succession of datasets in the computer vision field started in 1998 with the famous MNIST dataset for handwritten digits recognition [42]. After MNIST, other datasets, such as ImageNet [40] and PASCAL VOC [16], respectively in the image classification and object detection fields, have been created. These have subsequently integrated other types of challenges and ImageNet

24 2.4. Hardware, Frameworks and Datasets

Figure 2.25: Images and annotations from Microsoft COCO.

Name Description Size Class Tasks Format MNIST Handwritten digits recognition 60K 10 Image classification 28 × 28Gray Fashion MNIST Fashion products recognition 60K 10 Image classification 28 × 28Gray CIFAR-10 Small images, various classes 60K 10 Image classification 32 × 32RGB CIFAR-100 Small images, various classes 60K 100 Image classification 32 × 32RGB Caltech-101 Small images, various classes 9K 101 Image classification Small RGB Caltech-256 Small images, various classes 30K 256 Image classification Small RGB PASCAL VOC Various themes 11.5K 20 Image classification,

object detection, in- stance segmentation, action classification

Small RGB

ImageNet Visual Recognition at scale 14M 21K Image classification, object detection RGB Ms COCO Complex everyday scenes of com-

mon objects in natural context 330K 171 Object detection, se-mantic segmentation, instance segmentation, keypoint detection

RGB

Open Images Unified image classification, ob- ject detection, and visual rela- tionship detection at scale

9M 19.8K Image classification, object detection, visual relationship detection

RGB

SUN Very large scene and object

recognition database 131K 4.7K Object detection, in-stance segmentation RGB Cityscapes Semantic understanding of ur-

ban street scenes 25K 30 Object detection, in-stance segmentation RGB DAVIS Densely annotated video seg-

mentation 10.5K - Object detection, in-stance segmentation, video segmentation

RGB

LabelMe Dynamic dataset of annotated

natural scenes 187K - Object detection, in-stance segmentation RGB Visual Genome Connecting language and vision

with dense image annotations 108K 33.9K Object detection, im-age captioning RGB

Table 2.2: Famous deep learning image classification and object detection datasets [11]. currently in addition to image classification is used for object detection while PASCAL VOC has also been able to integrate instance segmentation. Starting from these datasets, in 2014 Microsoft COCO – Common Objects in COntext – has been created [46]. This has resized the concepts of instance segmentation started by PASCAL VOC at scale and is currently the main reference dataset together with ImageNet all over the world.

some of the recent declinations of these algorithms.

This chapter contains a technical overview of all the main themes in the object detection land- scape. The most performing object detection algorithms as well as activities ranging from clas- sification to semantic segmentation that often act as an integration to the proposed algorithms will be treated. The contents of this chapter will be the core innovation engine of subsequent developments in the retail products analytics area proposed by our work.

3.1

An Object Detection Review

Object detection [39] is currently one of the main tasks of computer vision. This is the natural evolution of image classification, another very well known task in the computer vision field that provides for the assignment of a specific label to an image to characterize its content. Object detection differs from the latter because it aims to determine, for an arbitrary image, both the label and the bounding box that defines each entity in the scene. Besides object detection, other tasks have assumed great importance nowadays. These are semantic segmentation and instance segmentation which provide to make per-pixel predictions to identify for a generic image respec- tively all the pixels belonging to a given class and all the pixels belonging to a given class instance. Other declinations see the extension of these tasks from the 2D world to the 3D world, introducing also advanced domain-specific challenges as video streaming understanding.

Given the semantic gap between what a human being is able to understand and what a computer is able to process, a series of issues are defined that make these tasks extremely challenging. In particular, computer vision algorithms traditionally have troubles dealing with [39] viewpoint variation, different illumination and deformation conditions, occlusion, intraclass variation and many other challenges. Therefore actually there is no obvious way to hard-code algorithms to solve the previous tasks except in a restricted domain.

Object detection – and, more generally, computer vision algorithms – have evolved over time from heuristic analytics techniques based on empirically defined rules to reach the modern data-driven approach [39]. The latter aims to train a classifier on a set of reference images and annotations called dataset and currently is very successful in terms of results achieved. The following constitutes a basic yet standard-compliant overview of the evolution of object detection approaches.

Geometric invariants. This class of algorithms exploits invariant geometric characteristics of classes of distinct geometric objects such as circles and rectangles to identify specific patterns in the images. Hough Transform and Generalized Hough Transform algorithms are part of this algorithmic class [36]. The latter in particular constitute a template matching approach

26 3.1. An Object Detection Review

Figure 3.1: Image classification [20]. Figure 3.2: Object detection [20].

Figure 3.3: Semantic segmentation [20]. Figure 3.4: Instance segmentation [20].

to determine which objects belong to the same class through an evaluation procedure of each candidate in a parametric space.

Feature based. Starting from an image it is possible to identify special points called keypoints through features detectors, as well as to associate these points based on the surrounding area through features descriptors. The detector provides a stable keypoints identification, insensitive to light changes and geometrically accurate. The descriptor, based on information from color, texture and orientation, instead provides a vector representation of a specific part of an image from the keypoints identified by the detector. The main application of the previous algorithms consists in comparing specific patterns in different images through a matcher. SIFT [49], SURF [5], HOG [14]: these names are all part of a famous pool of algorithms for object detection based on features.

Bag of features. The bag of features approach [41] is inspired by the famous bag of words method and combines both the data-driven approach and the feature-based one. The initial step of this approach involves learning a visual vocabulary based on features extracted from feature- based algorithms. Subsequently, it is possible to perform the classification and the detection by appropriately aggregating the visual vocabulary identified in the image through further data elaborations.

Deep learning. The data-driven approach has been developed over the years attaining increas- ingly higher goals. This was initially developed as a classification problem based on the concept of proposals and subsequently it has recently been formulated in an alternative way as a regression problem [39]. The first category includes the two-stage region-based approaches that perform entities detection on the basis of candidate image portions called regions. The second solution instead belong to single-stage approaches that simplify the can-

Figure 3.5: R-CNN pipeline [23].

didates proposal problem and directly execute the inferences on image areas defined a priori. Both solutions are currently used and constitute the state of the art of object detection algorithms.

The following sections contain a review of the most promising object detection algorithms adopt- ing the deep learning approach [39]. The collection does not aim to be exhaustive but contains the main results that have led the evolution towards modern approaches. A fundamental reflection point in the study of these algorithms is the experimental evaluation [27]. This is the main tool for performance evaluation and provides useful information on the design of the most innovative approaches. Experimental evaluation will be discussed in detail in the object detection field in Sec- tion 5.1 but the following sections will still provide key results and insights for efficient algorithms design. In particular, as a metric to evaluate performance, mAP – mean Average Precision – is often used to give a measure of the number of relevant objects retrieved in the test set images by the algorithms with respect to the ground truth. This is often calculated with respect to the IoU – Intersection over Union – threshold (where if not specified it means 50%) which indicates the percentage of surface of an object to be retrieved from the algorithm so that the latter’s inferences can be considered positive.

In some circumstances, the ablation experiments will be briefly discussed to provide useful key insights for understanding the various architectures. These experiments, have been originally used in the field of psychology to study the effects of removing parts of the animal’s brains on their behavior [73]. Actually especially in the deep learning landscape ablation studies have been adopted to gain better understanding of the networks behaviour by systematically removing its parts [23].