Does Multimodality Help Human and Machine for Translation and Image Captioning?

Does Multimodality Help Human and Machine for Translation and Image

Captioning?

Ozan Caglayan1,3_{, Walid Aransa}1_{, Yaxing Wang}2_{, Marc Masana}2_,

Mercedes Garc´ıa-Mart´ınez1_{, Fethi Bougares}1_{, Lo¨ıc Barrault}1 _{and Joost van de Weijer}2 1_{LIUM, University of Le Mans,}2 _{CVC, Universitat Autonoma de Barcelona,}

3_{Galatasaray University}

1_{[email protected]}

2_{{joost,mmasana,yaxing}@cvc.uab.es}

3_{[email protected]}

Abstract

This paper presents the systems developed by LIUM and CVC for the WMT16 Mul-timodal Machine Translation challenge. We explored various comparative meth-ods, namely phrase-based systems and at-tentional recurrent neural networks mod-els trained using monomodal or multi-modal data. We also performed a hu-man evaluation in order to estimate the usefulness of multimodal data for human machine translation and image description generation. Our systems obtained the best results for both tasks according to the auto-matic evaluation metrics BLEU and ME-TEOR.

1 Introduction

Recently, deep learning has greatly impacted the natural language processing field as well as com-puter vision. Machine translation (MT) with deep neural networks (DNN), proposed by (Kalchbren-ner and Blunsom, 2013; Sutskever et al., 2014) and (Bahdanau et al., 2014) competed successfully in the last year’s WMT evaluation campaign (Bo-jar et al., 2015).

In the same trend, generating descriptions from images using DNNs has been proposed by (Elliott et al., 2015). Several attempts have been made to incorporate features from different modalities in order to help the automatic system to better model the task at hand (Elliott et al., 2015; Kiros et al., 2014b; Kiros et al., 2014a).

This paper describes the systems developed by LIUM and CVC who participated in the two pro-posed tasks for the WMT 2016 Multimodal Ma-chine Translation evaluation campaign: Multi-modal machine translation (Task 1) and multi-modal image description (Task 2).

The remainder of this paper is structured in two parts: The first part (section 2) describes the archi-tecture of the four systems (two monomodal and two multimodal) submitted for Task 1. The stan-dard phrase-based SMT systems based on Moses are described in section 2.1 while the neural MT systems are described in section 2.2 (monomodal) and section 3.2 (multimodal). The second part (section 3) contains the description of the two systems submitted for Task 2: The first one is a monomodal neural MT system similar to the one presented in section 2.2, and the second one is a multimodal neural machine translation (MNMT) with shared attention mechanism.

In order to evaluate the feasibility of the multi-modal approach, we also asked humans to perform the two tasks of this evaluation campaign. Results show that the additional English description sen-tences improved performance while the straight-forward translation of the sentence without the im-age did not provide as good results. The results of these experiments are presented in section 4.

2 Multimodal Machine Translation

This task consists in translating an English sen-tence that describes an image into German, given the English sentence itself and the image that it describes.

2.1 Phrase-based System

Our baseline system for task 1 is developed fol-lowing the standard phrase-based Moses pipeline as described in (Koehn et al., 2007), SRILM (Stolcke, 2002), KenLM (Heafield, 2011), and GIZA++ (Och and Ney, 2003). This system is trained using the data provided by the organizers and tuned using MERT (Och, 2003) to maximize BLEU(Papineni et al., 2002) and METEOR(Lavie

and Agarwal, 2007) scores on the validation set.

We also used Continuous Space Language Model1_{(CSLM) (Schwenk, 2010) with the}

auxil-iary features support as proposed by (Aransa et al., 2015). This CSLM architecture allows us to use sentence-level features for each line in the train-ing data (i.e. all n-grams in the same sentence will have the same auxiliary features). By this means, better context specific LM estimations can be ob-tained.

We used four additional scores to rerank 1000-best outputs of our baseline system: The first two scores are obtained from two separate CSLM(s) trained on the target side (i.e. German) of the par-allel training corpus and each one of the following auxiliary features:

• VGG19-FC7 image features:The auxiliary feature used in the first CSLM are the image features provided by the organizers which are extracted from the FC7 layer (relu7) of the VGG-19 network (Simonyan and Zisserman, 2014). This allows us to train a multimodal CSLM that uses additional context learned from the image features.

• Source side sentence representation vec-tors: We used the method described in (Le and Mikolov, 2014) to compute continuous space representation vector for each source (i.e. English) sentence that will be provided to the second CSLM as auxiliary feature. The idea behind this is to condition our target lan-guage model on the source side as additional context.

The two other scores used for n-best reranking are the log probability computed by our NMT sys-tem that will be described in the following sec-tion and the score obtained by a Recurrent Neural Network Language Model (RNNLM) (Mikolov et al., 2010). The weights of the original moses fea-tures and our additional feafea-tures were optimized to maximize the BLEU score on the validation set.

2.2 Neural MT System

The fundamental model that we experimented2

is an attention based encoder-decoder approach (Bahdanau et al., 2014) except some notable changes in the recurrent decoder called Condi-tional GRU.

1_{github.com/hschwenk/cslm-toolkit} 2_{github.com/nyu-dl/dl4mt-tutorial}

We define by X and Y, a source sentence of length N and a target sentence of length M

re-spectively. Each source and target word is rep-resented with an embedding vector of dimension

EX andEY respectively:

X= (x1, x2, ..., xN), xi∈REX (1)

Y = (y1, y2, ..., yM), yj ∈REY (2)

A bidirectional recurrent encoder reads an input sequenceXin forwards and backwards to produce

two sets of hidden states based on the current in-put and the previous hidden state. An annotation vectorhi for eachposition iis then obtained by concatenating the produced hidden states.

An attention mechanism, implemented as a sim-ple fully-connected feed-forward neural network, accepts the hidden statehtof the decoder’s recur-rent layer and one input annotation at a time, to produce the attention coefficients. A softmax acti-vation is applied on those attention coefficients to obtain the attention weights used to generate the weighted annotation vector for timet. The initial

hidden stateh0 of the decoder is determined by a

feed-forward layer receiving the mean annotation vector.

We use Gated Recurrent Unit (GRU) (Chung et al., 2014) activation function for both recurrent en-coders and deen-coders.

2.2.1 Training

We picked the following hyperparameters for all NMT systems both for Task 1 and Task 2. All embedding and recurrent layers have a dimen-sionality of 620 and 1000 respectively. We used Adam as the stochastic optimizer with a mini-batch size of 32, Xavier weight initialization (Glo-rot and Bengio, 2010) and L2 regularization with

λ = 0.0001 except the monomodal Task 1 sys-tem for which the choices were Adadelta, sam-pling fromN(0,0.01)and L2 regularization with

λ= 0.0005respectively.

The performance of the network is evaluated on the validation split using BLEU after each 1000 minibatch updates and the training is stopped if BLEU does not improve for 20 evaluation periods. The training times were 16 and 26 hours respec-tively for monomodal and multimodal systems on a Tesla K40 GPU.

2.3 Data

Phrase-based and NMT systems for Task 1 are trained using the dataset provided by the organiz-ers and described in Table 1. This dataset con-sists of 29K parallel sentences (direct translations of image descriptions from English to German) for training, 1014 for validation and finally 1000 for the test set. We preprocessed the dataset using the punctuation normalization, tokenization and low-ercasing scripts from Moses. In order to general-ize better over the compound structs in German, we trained and applied a compound splitter3

(Sen-nrich and Haddow, 2015) over the German vocab-ulary of training and validation sets. This reduces the target vocabulary from 18670 to 15820 unique tokens. During translation generation, the splitted compounds are stitched back together.

Side Vocabulary Words

English 10211 377K

German 15820 369K

Table 1: Training Data for Task 1.

2.4 Results and Analysis

The results of our phrase-based baseline and the four submitted systems are presented in Table 2. TheBL+4Featuressystem is the rescoring of the baseline 1000-best output using all the features de-scribed in 2.1 whileBL+3Featuresis the same but excluding FC7 image features. Overall, we were able to improve test set scores by around 0.4 and 0.8 on METEOR and BLEU respectively over a strong phrase-based baseline using auxiliary fea-tures.

Regarding the NMT systems, the monomodal NMT achieved a comparative BLEU score of 32.50 on the test set compared to 33.45 of the phrase-based baseline. The multimodal NMT sys-tem that will be described in section 3.2, obtained relatively lower scores when trained using Task 1’s data.

3 Multimodal Image Description Generation

The objective of Task 2 is to produce German de-scriptions of images given the image itself and one or more English descriptions as input.

3_{github.com/rsennrich/wmt2014-scripts}

3.1 Visual Data Representation

To describe the image content we make use of Convolutional Neural Networks (CNN). In a breakthrough work, Krizhevsky et al. (Krizhevsky et al., 2012) convincingly show that CNNs yield a far superior image representation compared to previously used hand-crafted image features. Based on this success an intensified research ef-fort started to further improve the representations based on CNNs. The work of Simonyan and Zisserman (Simonyan and Zisserman, 2014) im-proved the network by breaking up large convolu-tional features into multiple layers of small con-volutional features, which allowed to train a much deeper network. The organizers provide these fea-tures to all participants. More precisely they pro-vide the features from the fifth convolutional layer, and the features from the second fully connected layer of VGG-19. Recently, Residual Networks (ResNet) have been proposed (He et al., 2015). These networks learn residual functions which are constructed by adding skip layers (or projection layers) to the network. These skip layers pre-vent the vanishing gradient problem, and allow for much deeper networks (over hundred layers) to be trained.

To select the optimal layer for image representa-tion we performed an image classificarepresenta-tion task on a subsection of images from SUN scenes (Xiao et al., 2010). We extract the features from the various layers of ResNet-50 and evaluate the classifica-tion performance (Figure 1). The results increase during the first layers but stabilize from Block-4 on. Based on these results and considering that a higher spatial resolution is better, we have se-lected layer ’res4fx’ (end of Block-4, after ReLU) for the experiments on multimodal MT. We also compared the features from different networks on the task of image description generation with the system of Xu et al. (Xu et al., 2015). The results for generating English descriptions (Table 3) show a clear performance improvement from VGG-19 to ResNet-50, but comparable results are obtained when going to ResNet-152. Therefore, given the increase in computational cost, we have decided to use ResNet-50 features for our submission.

3.2 Multimodal NMT System

System Description Validation Set Test Set

METEOR (norm) BLEU METEOR (norm) BLEU Phrase-based Baseline (BL) 53.71 (58.43) 35.61 52.83 (57.37) 33.45 BL+3Features 54.29 (58.99) 36.52 53.19 (57.76) 34.31 BL+4Features 54.40 (59.08) 36.63 53.18 (57.76) 34.28 Monomodal NMT 51.07 (54.87) 35.93 49.20 (53.10) 32.50 Multimodal NMT 44.55 (47.97) 28.06 45.04 (48.52) 27.82

Table 2: BLEU and METEOR scores on detokenized outputs of baseline and submitted Task 1 systems. The METEOR scores in parenthesis are computed with-normparameter.

block2 block3 block4 block5

80 85 90 95 100

ResNet Layers

A

ccurac

y

Figure 1: Classification accuracy on a subset of SUN scenes (Xiao et al., 2010) for ResNet-50: The colored groups represent the building blocks while the bars inside are the stacked blocks (He et al., 2015).

Network BLEU-1 BLEU-2 BLEU-3 BLEU-4

VGG-19 58.2 31.4 18.5 11.3

ResNet-50 68.4 45.2 30.9 21.1

ResNet-152 68.3 44.9 30.7 21.1

Table 3: BLEU scores for various deep features on the image description generation task using the system of Xu et al. (Xu et al., 2015).

The model involves two GRU layers and an at-tention mechanism. The first GRU layer computes an intermediate representations0_j as follows:

s0_j = (1₋z0_j)s0_j+z_j0 sj−1 (3)

s0_j = tanh(W0E[yj−1] +rj0 (U 0

sj−1)) (4)

r0_j = σ(W_r0E[yj−1] +Ur0sj−1) (5)

z0_j = σ(W_z0E[yj−1] +Uz0sj−1) (6)

where E is the target word embedding, s0_j is the

hidden state, r0_j and z_j0 are the reset and update

gate activations. W0,U_r0,W_r0,U_r0,W_z0 andU_z0 are

the parameters to be learned.

A shared attention layer similar to (Firat et al., 2016) that consists of a fully-connected feed-forward network is used to compute a set of modality specific attention coefficients emod

ij at

each timestepj:

emod_ij =Uatttanh(Wcatthmodi +Watts 0

j) (7)

The attention weight between source modality contextiand target wordjis computed by apply-ing a softmax onemod_ij :

αij =

exp(etxt ij )

PN

k=1exp(etxtkj)

(8)

βij =

exp(eimg_ij )

P196

k=1exp(e

img kj )

(9)

The final multimodal context vectorcjis obtained as follows:

cj = tanh( N

X

i=1

αijhtxti +

196

X

i=1

βijhimg_i ) (10)

The second GRU generatessjfrom the interme-diate representations0_j and the context vectorcj as follows:

sj = (1−zj)sj +zjs 0

j (11)

s_j = tanh(W cj+rj(U s0j)) (12)

rj = σ(Wrcj +Urs 0

j) (13)

zj = σ(Wzcj+Uzs 0

j) (14)

wheres0_j is the hidden state,rj andzjare the reset and update gate activations. W,Ur, Wr,Ur, Wz andUzare the parameters to be learned.

Finally, in order to compute the target word, the following formulations are applied:

oj =Lotanh(E[yj−1] +Lssj+Lccj) (15)

P(yj|yj−1, sj, cj) =Softmax(oj) (16)

x₁ x₂ x₃ x_N (N, 1, 1000)

RNN Forward

x₃ x₂ x₁

(N, 1, 1000) RNN Backward

(N, 1, 2000) Phrase Context Vectors

c_N

ResNet-50 CNN 14 x 14 x 1024 Feature Maps

(196, 1, 2000) Image Context Vectors * W_img= ...

...

Target Embeddings

Softmax

GRU

*Wc_att

*W_att

Context Fusion

Φ (Σ)

Φ (Σ)

Φ (Σ)

GRU

*U_att

Σβ_i* ..

Σα_i* ..

Φ (Σ)

(196, 1, 1024) Image Context

Vectors

*L_S x_N

*L_C Predicted Target Word

c₁

Multimodal Conditional GRU

*L_O

Figure 2: The architecture of the multimodal NMT system. The boxes with∗ refers to a linear trans-formation whileΦ(Σ) means atanhapplied over the sum of the inputs. The figure depicts a running

instance of the network over a single example.

3.2.1 Generation

Since we are provided 5 source descriptions for each image in order to generate a single German description, we let the NMT generate a German description for each source and pick the one with the highest probability and preferably without an UNK token.

3.3 Data

The organizers provided an extended version of the Flickr30K Entities dataset (Elliott et al., 2016) which contains 5 independently crowd-sourced German descriptions for each image in addition to the 5 English descriptions originally found in the dataset. It is possible to use this dataset either by considering the cross product of 5 source and 5 tar-get descriptions (a total of 25 description pairs for each image) or by only taking the 5 pairwise de-scriptions leading to 725K and 145K training pairs respectively. We decided to use the smaller subset of 145K sentences.

Side Vocabulary Words

English 16802 1.5M

German 10000 1.3M

Table 4: Training Data for Task 2.

The preprocessing is exactly the same as Task 1 except that we only kept sentence pairs with sen-tence lengths∈[3,50]and with a ratio of at most 3. This results in a final training dataset of 131K

sentences (Table 4). We picked the most frequent 10K German words and replaced the rest with an UNK token for the target side. Note that com-pound splitting was not done for this task.

3.4 Results and Analysis

System Validation Test

METEOR BLEU METEOR BLEU

Monomodal 36.3 24.0 35.1 23.8

Multimodal 34.4 19.3 32.3 19.2

Table 5: BLEU and METEOR scores of our NMT based submissions for Task 2.

As we can see in Table 5, the multimodal system does not surpass monomodal NMT system. Sev-eral explanations can clarify this behavior. First, the architecture is not well suited for integrating image and text representations. This is possible as we did not explore all the possibilities to bene-fit from both modalities. Another explanation is that the image context contain too much irrele-vant information which cannot be discriminated by the lone attention mechanism. This would need a deeper analysis of the attention weights in order to be answered.

4 Human multimodal translation and/or description

Method BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR

Image + sentences 54.30 35.95 23.28 15.06 39.16

Image only 51.26 34.74 22.63 15.01 38.06

Sentence only 39.37 23.27 13.73 8.40 32.98

Our system 60.61 44.35 31.65 21.95 33.59

Table 6: BLEU and METEOR scores for human translation/description generation experiments.

where we replace the computer algorithm with hu-man participants. The two modalities are the five English description sentences, and the image. The output is a single description sentence in German. The experiment asks the participants for the fol-lowing tasks:

• Given both the image and the English de-scriptions: ’Describe the image in one sen-tence in German. You can get help from the English sentences provided.’

• Given only the image: ’Describe the image in one sentence in German.’

• Given only one English sentence: ’Translate

the English sentence into German.’

The experiment was performed by 16 native Ger-man speakers proficient in English with age rang-ing from 23 to 54 (comrang-ing from Austria, Germany and Switzerland, of which 10 are female and 6 male). The experiment is performed on the first 80 sentences of the validation set. Participants per-formed 10 repetitions for each task, and not re-peating the same image across tasks. The results of the experiments are presented in Table 6. For humans, the English description sentences help to obtain better performance. Removing the image altogether and providing only a single English de-scription sentence results in a significant drop. We were surprised to observe such a drop, whereas we expected good translations to obtain competitive results. In addition, we have provided the results of our submission on the same subset of images; humans clearly obtain better performance using METEOR metrics, but our approach is clearly out-performing on the BLEU metrics. The participants were not trained on the train set before performing the tasks, which could be one of the reasons for the difference. Furthermore, given the lower per-formance of only translating one of the English de-scription sentences on both metrics, it could pos-sibly be caused by existing biases in the data set.

5 Conclusion and Discussion

We have presented the systems developed by LIUM and CVC for the WMT16 Multimodal Ma-chine Translation challenge. Results show that in-tegrating image features into a multimodal neural MT system with shared attention mechanism does not yet surpass the performance obtained with a monomodal system using only text input. How-ever, our multimodal systems do improve upon an image captioning system (which was expected). The phrase-based system can benefit from rescor-ing with multimodal neural language model as well as rescoring with a neural MT system.

We have also presented the results of a human evaluation performing the same tasks as proposed in the challenge. The results are rather clear: im-age captioning can benefit from multimodality.

Acknowledgments

This work was supported by the Chist-ERA project M2CR4_{. We kindly thank KyungHyun Cho}

and Orhan Firat for providing the DL4MT tuto-rial as open source and Kelvin Xu for the arctic-captions5_system.

References

Walid Aransa, Holger Schwenk, and Loic Barrault. 2015. Improving continuous space language models using auxiliary features. InProceedings of the 12th International Workshop on Spoken Language Trans-lation, pages 151–158, Da Nang, Vietnam, Decem-ber.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.

Ondˇrej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 workshop on statistical machine translation. InProceedings of the Tenth Workshop on Statistical Machine Translation, pages 1–46, Lisbon, Portugal, September. Association for Computational Linguis-tics.

Junyoung Chung, C¸aglar G¨ulc¸ehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence model-ing.CoRR, abs/1412.3555.

4_{m2cr.univ-lemans.fr}

Desmond Elliott, Stella Frank, and Eva Hasler. 2015. Multi-language image description with neural se-quence models. CoRR, abs/1510.04709.

D. Elliott, S. Frank, K. Sima’an, and L. Specia. 2016. Multi30K: Multilingual English-German Image De-scriptions. CoRR, abs/1605.00459.

Orhan Firat, Kyunghyun Cho, and Yoshua Ben-gio. 2016. Multi-way, multilingual neural ma-chine translation with a shared attention mechanism.

arXiv preprint arXiv:1601.01073.

Xavier Glorot and Yoshua Bengio. 2010. Understand-ing the difficulty of trainUnderstand-ing deep feedforward neu-ral networks. InIn Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS’10). Society for Artificial Intelligence and Statistics.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recog-nition.arXiv preprint arXiv:1512.03385.

Kenneth Heafield. 2011. KenLM: faster and smaller language model queries. In Proceedings of the EMNLP 2011 Sixth Workshop on Statistical Ma-chine Translation, pages 187–197, Edinburgh, Scot-land, United Kingdom, July.

Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. Seattle, October. As-sociation for Computational Linguistics.

Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014a. Multimodal neural language models. In Tony Jebara and Eric P. Xing, editors,Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 595–603. JMLR Work-shop and Conference Proceedings.

Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. 2014b. Unifying visual-semantic em-beddings with multimodal neural language models.

CoRR, abs/1411.2539.

Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In

Meeting of the Association for Computational Lin-guistics, pages 177–180.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hin-ton. 2012. Imagenet classification with deep con-volutional neural networks. In Advances in neural information processing systems, pages 1097–1105. Alon Lavie and Abhaya Agarwal. 2007. Meteor: an

automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceed-ings of the Second Workshop on Statistical Machine Translation, StatMT ’07, pages 228–231, Strouds-burg, PA, USA. Association for Computational Lin-guistics.

Quoc V Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. arXiv preprint arXiv:1405.4053.

Tomas Mikolov, Martin Karafi´at, Lukas Burget, Jan Cernock`y, and Sanjeev Khudanpur. 2010. Recur-rent neural network based language model. In IN-TERSPEECH, volume 2, page 3.

Franz Josef Och and Hermann Ney. 2003. A sys-tematic comparison of various statistical alignment models. Comput. Linguist., 29:19–51, March.

Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. InProceedings of the 41st Annual Meeting on Association for Computa-tional Linguistics - Volume 1, ACL ’03, pages 160– 167, Stroudsburg, PA, USA. Association for Com-putational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic eval-uation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Com-putational Linguistics, ACL ’02, pages 311–318, Stroudsburg, PA, USA.

Holger Schwenk. 2010. Continuous space lan-guage models for statistical machine translation. In

The Prague Bulletin of Mathematical Linguistics, (93):137–146.

Rico Sennrich and Barry Haddow. 2015. A joint dependency model of morphological and syntactic structure for statistical machine translation. In Pro-ceedings of the 2015 Conference on Empirical Meth-ods in Natural Language Processing, pages 114– 121. Association for Computational Linguistics.

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition.arXiv preprint arXiv:1409.1556.

Andreas Stolcke. 2002. Srilm - an extensible language modeling toolkit. Proceedings of the 7th Interna-tional Conference on Spoken Language Processing (ICSLP 2002), pages 901–904.

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural net-works. CoRR, abs/1409.3215.

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. 2010. Sun database: Large-scale scene recognition from abbey to zoo. In