automatic image annotation

With the advent of various multimedia devices such as digital cameras, mobile phones etc.. The number of images has increased dramatically. Thus, Effective methods are required for organizing, searching and browsing these images. Many search engines use text-based searching methods for retrieving images. Indexing images based on its semantic content will improve the image search quality. However, as it is impossible to manually annotate all images, Automatic image annotation (AIA) might be a promising solution. The goal of Automatic image annotation is to assign meaningful keywords to an image by automatically checking the semantics of the image. Automatic image annotation is essential to label a huge collection of unlabeled photos. Traditional methods of image annotation are not adequate as the amount of images to be indexed is huge, which makes it impractical and error prone. Thus to enhance annotation performance, optimization techniques are used. Optimization is a commonly encountered mathematical problem in all engineering disciplines. It means finding the best possible solution. Feature weighting is a technique used to approximate the optimal degree of influence of individual features. In this survey various methods to annotate images are discussed. Also, various approaches for optimum feature selection are interpreted and their limitations are discussed.

This paper presents MRSMO, a high performance SVM algorithm for automatic image annotation, using Google’s MapReduce [22], a distributed computing framework that facilitates data intensive processing. MRSMO is built on the Sequent Minimal Optimization (SMO) algorithm [21] and implemented using the Hadoop implementation [23] of MapReduce. The framework facilitates a number of important functions such as partitioning the input data, scheduling the program’s execution across a cluster of participating nodes, handling node failures, and managing the required network communications. This allows programmers with limited experience of parallel and distributed computing to easily utilize the resources of a large distributed system [22]. MRSMO efficiently performs the expensive computation and updating steps related to the optimality condition vector in parallel. In our approach we partition the training data according to the dataset size as well as the number of MapReduce mappers to be employed. In the Hadoop context, it is fundamental to identify the appropriate task dimension and establish a balance between framework overhead and increased load balancing and parallelism. The strategy for influencing the number of mappers and map tasks is based on the number of dataset fragments or chunks as well as respective sizes. We maintain classification accuracy close to global SVM optimization solvers by adopting simple yet effective strategies for global weight vector in linear SVM and global support vector computation in non-linear cases.

One method is content based image retrieval (CBIR) in which image is retrieved based on low level features like shape, color and texture. In this method user needs to apply query image in CBIR and similar images based on sample query image is retrieve by the system. But there is a semantic gap between low level visual feature and high level semantic concept that are used by the user. In manual image annotation, images are annotated manually by the user, so that images can be retrieved as easy as retrieving text document. This method is accurate but it is also inefficient because of the manual assignment of keywords to image, which is cumbersome and time consuming process. To overcome such problems of manual image annotation and to bridge the semantic gap, research in this area shifted to Automatic Image Annotation [1].

Abstract: Automatic tag selection for image is an important for many image related applications. The proposed approach explore the feasibility of automatic tag selection for News image in a knowledge-lean way and it consists of two components, namely extracting image content and rendering it in natural language. By using MixLDA represent the visual and textual modalities jointly as a probabilistic distribution over a set of topics. Automatic image annotation model take these distributions into account and finding the most likely keywords for an image and its associated documents. For caption generation, extractive and abstractive caption generation models are used to render the extracted image contents in natural language without rely on rich knowledge resources or sentence templates. Experimental results shown that the generated keywords and captions are relevant to the specific content of an image and its associated articles.

Nowadays image annotation & retrieval have been a very popular area for research. Many researchers were attracted by the benefits of Web image context in the past. As a result, a variety of context extraction methods, ranging from simple heuristics-based approaches to complex DOM and vision-based extractors have been proposed. In image retrieval and tagging, text annotation act as an important role. Approach [9], [7] uses contextual knowledge for automatic image annotation, whereas [9] uses CMRM techniques and [7] uses DOM based technique. While this technique are simple and fast but are prone to errors, i.e. when the text is overlapping.

The above experimental results in table 2 show that our method outperforms the Co-occurrence model [4] in the average precision and recall. Since our model uses the blob-tokens to represent the contents of the image regions and converts the task of automatic image annotation to a process of translating information from visual lan- guage (blob-tokens) to textual language (keywords). So Maximum Entropy Model is a natural and effective choice for our task, which has been successfully applied to the dyadic data in which observations are made from two finite sets of objects. But disad- vantages also exist. There are two fold problems to be considered. First, since Maxi- mum Entropy is constrained by the equation p ( ) ( ) f = p ~ f , which assumes that the expected value of output of the stochastic model should be the same as the expected value of the training sample. However, due to the unbalanced distribution of key- words frequency in the training subset of Corel data, this assumption will lead to an undesirable problem that common words with high frequency are usually associated with too many irrelevant blob-tokens, whereas uncommon words with low frequency have little change to be selected as annotations for any image regions, consider word “sun” and “apple” , since both words may be related to regions with “red” color and “round” shape, but it is difficult to make a decision between the word “sun” and “ap- ple”. However, since “sun” is a common word as compared to “apple” in the lexical set, the word “sun” will definitely used as the annotation for these kind of regions. To address this kind of problems, our future work will mainly focus on the more sophis- ticated language model to improve the statistics between image features and key- words. Second, the effects of segmentation may also affect the annotation perform- ance. As we know, semantic image segmentation algorithm is a challenging and com- plex problem, current segmentation algorithm based on the low-level visual features may break up the objects in the images, that is to say, segmented regions do not defi- nitely correspond to semantic objects or semantic concepts, which may cause the Maximum Entropy Model to derive a wrong decision given an unseen image.

This paper presents a novel image annotation framework for domains with large numbers of images. Automatic image annotation is such a domain, by which a computer system automatically assigns metadata in the form of captioning or keywords to a digital image. This application of computer vision technique is used in image retrieval system to organize and locate images of interest from a database. Many techniques have been proposed for image annotation in the last decade that has given reasonable performance on standard datasets In this work, we propose a new model for image annotation known as JSVM which treats annotation as a retrieval problem. In this work, we introduce an JSVM model for image annotation that treats annotation as a retrieval problem. The proposed technique utilizes low level image features and a simple combination of basic distances using JEC to find the nearest neighbors of a given image; the keywords are then assigned using SVM approach which aims to explore the combination of three different methods. First, the initial annotation of the data using flat wise and axis wise methods, and that takes the hierarchy into consideration by classifying consecutively its instances through position wise method. Finally, we make use of pair wise majority voting between methods by simply summing strings in order to produce a final annotation. The result of the proposed technique shows that this technique outperforms the current state of art methods on the standard datasets.

Automatic image annotation is a popular task in computer vision. The earliest approaches are closely related to image classification (Vailaya et al., 2001; Smeulders et al., 2000), where pictures are assigned a set of simple descriptions such as indoor, out- door, landscape, people, animal. A binary classifier is trained for each concept, sometimes in a “one vs all” setting. The focus here is mostly on image pro- cessing and good feature selection (e.g., colour, tex- ture, contours) rather than the annotation task itself. Recently, much progress has been made on the image annotation task thanks to three factors. The availability of the Corel database, the use of unsu- pervised methods and new insights from the related fields of natural language processing and informa- tion retrieval. The co-occurrence model (Mori et al., 1999) collects co-occurrence counts between words and image features and uses them to predict anno- tations for new images. Duygulu et al. (2002) im- prove on this model by treating image regions and keywords as a bi-text and using the EM algorithm to construct an image region-word dictionary.

set [14]. The conducted works have often used the hybrid models in order to define a common distribution on image attributes and tags [15, 16]. [17] Addresses automatic image annotation through transferring the tag to a semantic space. Accordingly, the semantic gap problem can be reduced by constructing a semantic space which includes a combination of visual and textual information. In [18] a voting directional graph- based framework is presented for retrieving related tags. [19] Presents a model called ML- RANK, which aims at ranking the tags related to an image based on visual similarities and semantic relations. [20] Given that some images have unclear content and their detection is somewhat difficult, other images and metadata are likely to exist in the neighborhood of them, which help to identify and recognize the content of the desired image. [21] Describes the problem of learning the relationship between labels as a basis for improving their descriptive power. Specifically, it defines a supervised neighbor voting stage that labels relations obtained by visual neighbors. The structure of this system consists of two main parts: 1- the formulation of relationship between tags and 2- ranking oriented learning. [22] Uses a two-step algorithm to retrieve tags. First, the initial score of the associated tag is calculated for each of tag using kernel density estimation (Gaussian), and second, a random motion on the tag graph is done by which the edges of a related tag are weighted based on similarity. [23] Proposes a framework for multi-query expansion to retrieve semantically robust landmarks of a user’s query using Latent Dirichlet Allocation (LDA) technique for incomplete and poorly queries from the user. In [24] uses a data-driven approach as well as knowledge extracted from the generated tags by users, available resources on the web, images uploaded by users and visual similarity of key frames to refine video tags in order to increase the number of initial tags provided by users.

With the popularization of digital cameras and other digital devices, the number of images of the network has increased exponentially [1], and image retrieval technology has become a hot research topic. According to the different retrieval methods, image retrieval technology can be divided into two categories: text-based image retrieval (TBIR); content- based image retrieval (CBIR) [2–4]. The advantage of TBIR is convenient, and users can query and get the relevant results by searching the relevant keywords. However it requires manual annotation of images, the workload is very massive. CBIR searches similar images based on the visual characteristics of images. Although there are many works about CBIR [5–10], the semantic gap still exists because the images are annotated based on their low-level features such as color and texture. Many studies combine semantic infor- mation to improve content-based image retrieval techniques, and semantic information is usually composed of textual keywords that describe the semantic attributes of images. Because manually annotating semantic information is a very time-consuming and laborious work, automatic image annotation has become an increasingly crucial problem in image retrieval [11–19].

Abstract—In this paper, we propose a new approach for automatic image annotation (AIA) in order to automatically and efficiently assign linguistic concepts to visual data such as digital images, based on both numeric and semantic features. The presented method first computes multi-layered active contours. The first-layer active contour corresponds to the main object or foreground, while the next-layers active contours delineate the object’s subparts. Then, visual features are extracted within the regions segmented by these active contours and are mapped into semantic notions. Next, decision trees are trained based on these attributes, and the image is semantically annotated using the resulting decision rules. Experiments carried out on several standards datasets have demonstrated the reliability and the computational effectiveness of our AIA system.

Adaptive Privacy Policy Prediction (A3P) [8] system is introduced by Anna Cinzia Squicciarini. Personalized policies can be automatically generated by this system. It makes use of the uploaded images by users and a hierarchical image classification is done. Images content and metadata is handled by the A3P system .It consists of two components: A3P Core and A3P Social. The image will be first sent to the A3P-core, when the user uploads the image. The A3P-core classifies the image and determines whether there is a need to invoke the A3P-social.When meta data information is unavailable it is difficult to generate accurate privacy policy. This is the disadvantage of this system. Privacy violation as well as inaccurate classification will be the after effect of manual creation of meta data log information. Automatic Image Annotation (AIA) helps to overcome the problem with meta data information.

Abstract—Automatic image annotation is the process by which a computer system automatically assigns metadata in the form of captioning or keywords to an image. This application of computer vision techniques is used in image retrieval systems to organize and locate images of interest from database. Most of the existing work such as Latent Semantic Indexing and Markovian Semantic Indexing (MSI) has been presented for online image retrieval system. By extending these methods, the proposed system uses Semantic annotated Markovian Semantic Indexing (SMSI) for retrieving the images. The proposed system automatically annotates the images using hidden Markov model. To annotate images, concepts are represented as states by using Hidden Markov model. The parameters of the model are estimated from a set of manually annotated (training) images. Each image in a large test collection is then automatically annotated with the a posteriori probability of concepts present in it. After annotating these images, semantic retrieval of images can be done. This can be done by using Natural Language processing tool namely WordNet and also measuring semantic similarity of annotated images in the database using MSI.

Automatic image annotation is a challenging field with a far reaching effect. As the world moves towards becoming more and more dependent on digital technologies every day, use of machine to automatically annotate images can be proved as demanding in many fields of image processing. Automatic Image Annotation reduces the gap between low level image features and high level image semantics. Utilization of Speeded Up Robust Features (SURF) in automatic image annotation is very appealing due to the fact that SURF is scale and rotation invariant detector and descriptor and is much faster than any other schemes. Unlike other methods SURF features use the entire image instead of segmented blocks of image. That is why annotation of images by using SURF can be considered as more accurate. In this paper, a SVM based image annotation approach is proposed that uses SURF features of image for annotation purpose. The experiments suggest that the method proposed is much more efficient than other methods.

Automatic image annotation using Support Vector Machine The mechanism of SVM classifier works by finding a hyper plane from a training set of samples to separate them. Feature vector and class label is associated with each training sample. An SVM is basically a binary classifier. The output of the classifier is the semantic concept(s) which is used for image annotation [1] The aim is to define a Hyper plane which partitions the set of examples such that all the points with the same label are on the same side of the hyper plane [12][1].Chapelle et al [12] use the above mentioned basic framework to train 14 SVM classifiers for 14 image level concepts. Images are represented with HSV histogram..To train an SVM for a particular concept, training images belonging to that concept are regarded as positive instance while the others are regarded as negative instance. Therefore, each trained classifier can be regarded as one vs. all ’classifier. During testing, each classifier

In section 4 we have seen that the salience mea- sure helps in determining what persons are visible in the image. We have used the fact that face detection in images is relatively easily and can thus supply a cut-off value for the ranked person names. In the present state-of-the-art, we are not able to exploit a similar fact when detecting all types of entities. We will thus use the salience measure in a different way. We compute the salience of every entity, and we assume that only the entities with a salience score above a threshold of 0.5 will appear in the image. We see that this method drastically improves preci- sion to 66.03%, but also lowers recall until 54.26%. We now create a last model where we combine both the visualness and the salience measures. We want to calculate the probability of the occurrence of an entity e im in the image, given a text t, P (e im |t).

Compared with baselines, the proposed WSG algorithm matches much higher accuracies of 0.71, 0.64, and 0.38on the MSRC, COREL- 100, and VOC-07 dataset respectively. Since the BSVM classifier is trained at the image level and tested at the patch level, it performs worst. It shows that cross-level label inference is not trivial, and straightforward propagating labels from images to patches is not applicable. A more sophisticated method is required to weakly impose image labels upon their descendent patches.

C. Lakshmi Devasena is presently doing PhD in Karpagam University, Coimbatore, Tamilnadu, India. She has completed M.Phil. (computer science) in the year 2008 from Bharathidasan University and MCA degree in 1997 from RVS College of Engineering and Technology, Dindigul (Madurai Kamaraj University) and B.Sc (Mathematics) in 1994 from Gandhigram Rural Institute. She has Eight years of teaching experience and Two years of industrial experience. Her area of research interest is Image processing and Data mining. She has published 10 papers in International Journals and Presented 18 papers in National and international conferences. At Present, she is working as Lecturer in Karpagam University, Coimbatore.

There is a great deal of importance on the textural and image means of repre- sentation, as we would like to be able to extract as much detailed information as possible for the learning process. Various approaches have been suggested such as colour moments and Gabor texture descriptors[17] as well as scale invariant in- terest points[10] and affine invariant interest point detector [11]. Scale Invariant Feature Transformation (SIFT) have been introduced by [9] and have been shown to be superior to other descriptors[12]. This is due to the fact that the SIFT de- scriptors are designed to be invariant to small shifts in position of the salient (i.e. prominent) region. SIFT transforms the image data into scale invariant coordi- nates relative to local features. The underlying idea of SIFT is to extract distinc- tive invariant features from an image such that it could be used to perform reli- able matching between different views of an object or scene. Since we are aiming to learn the association of a keyword to an object, which could appear in different angels and scenes, we find SIFT ideal for the image representation.

(CNN) (Krizhevsky et al., 2012) in the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC12) (Deng et al., 2012a). Donahue et al. (2013) report that features extracted from the acti- vation of a deep CNN trained in a fully supervised fashion can also be re-purposed to novel generic tasks that differ significantly from the original task. Inspired by Donahue et al. (2013), we extract such acti- vation as feature for ImageNet images that correspond to the 1,372 synsets, and train binary classifiers to detect the presence of the synsets in the images of Flickr8k. More specifically, we use as our training set the 1,571,576 ImageNet images in the 1,372 synsets, where a random sample of 5,000 images serves as negative examples, and as our test set the 8,091 images in Flickr8k. For each image in both sets, we extracted activation of a pre-trained CNN model as its feature. The model is a reference implementation of the structure proposed in Krizhevsky et al. (2012) with minor modifications, and is made publicly available through the Caffe project (Jia, 2013). It is shown in Donahue et al. (2013) that the activation of layer 6 of the CNN performs the best for novel tasks. Our study on a toy example with 10 ImageNet synsets however suggests that the activation of layer 7 has a small edge. Once the 4,096 dimensional activation of layer 7 is extracted for both training and test sets, 1,372 binary classifiers are trained and applied using LIBSVM (Chang and Lin, 2011), which give probability estimates for the test images. For each image, the 1,372 classifiers are then ranked in order of their probability estimates.