2019 International Conference on Artificial Intelligence and Computing Science (ICAICS 2019) ISBN: 978-1-60595-615-2
Self-supervised Adversarial Hashing Cross-modal Retrieval with
Generative Models Based on Attention Mechanism
Jian-qiong XIAO
*, Zhi-yong ZHOU and Xiao-qing ZHOU
Educational Information Technical Center, China West Normal University, China
*Corresponding author
Keywords: Attention mechanism, DEEP hashing learning, Adversarial network, Generative models.
Abstract. In this paper, a self-supervised adversarial network retrieval model (SAHCM) based on attention mechanism is proposed, which takes attention mechanism and adversarial network fusion generation process to cross-modal representation, and has the ability to capture global semantic information and local information of text and image video data at the same time. In order to learn a common representation space for multimode data, experiments show that the proposed method can accurately match image and text description in complex content, reduce the storage space of cross-modal retrieval and improve the efficiency of computation time. And the effect of state-of-the-art is obtained in the cross-modal retrieval experiment on MSCOCO dataset.
Introduction
With the development of smart mobile devices and the Internet, multimedia data such as text, pictures, video and audio are growing rapidly, and these different modal data are often used to describe the same event or content. In the face of such large and interconnected multimode data, users are eager to be able to use one of the modes (such as text) to quickly and effectively retrieve other modal resources (such as images, videos, etc.) associated with them in a large amount of multimedia data, that is, cross-modal Retrieval. Cross-modal retrieval overcomes the limitation of retrieving the same modal data in single mode retrieval, realizes the mutual retrieval of different modal data, makes the retrieval flexible and convenient, and is closer to the user's needs. In recent years, the research and application based on cross-modal have been developed rapidly and paid great attention to. The essence of cross-modal application is to transform the two different modes of text-vision to realize the migration between modes, the key is how to model the relationship between different modes, the difficulty is to cross bridging the semantic gap.
Since the 2000, experts and scholars at home and abroad have carried out a lot of research work around the text-visual cross-modal retrieval, and have made many important achievements, but there are still some potential problems, which need to be further studied by us. From the perspective of deep hash learning, this paper studies how to use the global and local characteristics of image video, combined with attention mechanism and adversarial network, to study the effective coding extraction of visual information features, establish the general representation of cross-modal data, and realize the accurate understanding and analysis of video image content. In this paper, an efficient cross-modal self-supervised adversarial hashing and generation model is proposed, which aims to reduce the retrieval granularity and improve the retrieval accuracy and efficiency.
Related Work
Feng et al. [3] in the cross-modal retrieval task for the first time, the deep learning cross-modal model (CORR-AE) based on Auto Encoder is proposed, and the main idea is to use Auto Encoder encoder and decoder to learn the correlation and feature representation respectively. The reconstruction error of the minimized single modal self-encoder and the sum of the correlation errors of different modal representations are optimized, and an integrated framework of the correlation learning between the single modal representation learning and the modal is realized, which is used for reference and improvement by many subsequent researchers. In order to obtain global information and Sequence feature information of visual information, Kiros et al. [4] map images and text to common subspace with CNN network and LSTM respectively.
Deep hash learning is a combination of deep learning and hash learning, which can learn video image representation and hash coding at the same time, and obtains better results than traditional hash algorithm and depth learning. In 2014,Xia et al. first combine deep learning with hashes, they proposed the CNNH algorithm [5] in Aaai, which divides the hash function learning process into two steps. 2015 lai [6] proposed a DNNH algorithm on CVPR, which optimizes the objective function based on the image ternary group sorting information as the monitoring information, which proves that the deep convolution neural network can be used to simultaneously carry out feature extraction and hash learning within a framework.
In recent years, generative adversarial Networks(GAN) has made a splash on computer vision and natural language processing tasks. GAN contains a generation model and a discriminant pattern, which optimizes the model through game confrontation of two models, optimizes the parameters of one of the models first, updates the parameters of the other, and then alternately iterates to maximize the error of the other party. Wang [7]et al. successfully applied the optimization idea of game countermeasure to cross-modal retrieval, and proposed a novel and antagonistic cross-modal retrieval (ACMR) method. Experimental results show that the performance of this iterative optimization cross-modal retrieval model based on game thinking has been greatly improved, which opens up a new gate for the study of cross-modal retrieval.
Modal
When people query image video based on text sentences, they may first filter clearly unrelated image videos based on rough overall semantic information, however, to ultimately locate the most relevant videos, they largely need to rely on the keywords and video keyframes that are queried. Therefore, in the process of cross-modal correlation representation learning, it is necessary to consider both the global semantic information of the data and the significant local key information, so as to fully depict the local fine similarity and hierarchical similarity of text and visual information, across the “semantic gap” between different modes.
[image:2.612.179.390.545.654.2]Cross-Modal Correlation Representation
Figure 1. KGRU model.
the rate of model training, so the GRU model is selected in our model. To obtain locally significant information about the input data, we added a key gate to the GRU, as shown in Figure 1, which controls how much input information is output at each moment without changing the way the hidden state is updated.
In the figure above, R, Z and K are reset gate, update gate and Key gate.
KGRU has two outputs, namely, a State feature output ht and a local key feature output Ot. Its update mechanism is defined as follows:
The model can not only capture the intrinsic correlation between sequences (between words and words, between video frames and video frames), obtain the global semantic information of each modal, but also capture the local key information of the input data.
Self-Supervised Adversarial Hashing Cross-Modal Retrieval with Generative Models(SAHCM)
[image:3.612.113.506.432.542.2]Firstly, the KGRU model is used to obtain the text-visual cross-modal representation at the global semantic level, then the text-video and video-text two generation models are introduced to carry out the cross modal feature representation at the local level, and secondly, two countermeasure networks are used to maximize the semantic correlation and representation consistency between different modes. The cross-modal anti-hash generation model of multi-scale feature fusion based on attention mechanism is constructed, as shown in Figure 2.
Figure 2. The proposed Self-Supervised Adversarial Hashing Cross-Modal Retrieval with Generative Models (SAHCM).
The model mainly includes three modules: The cross-modal feature representation part (the entire upper region), the video-text generation feature learning part (solid line part) and the text-video generation countermeasure learning part (dotted part).
Finally, the semantic information obtained is used as the monitoring information to guide the characteristics of different modes and the process of hash learning.
Objective Function
where, α is the edge threshold,
Experiments
In order to verify the effectiveness of the proposed SAHCM model in cross-modal retrieval tasks, we experimented with two video-text-based cross-media retrieval tasks. The specific overview is as follows:
(1) text-to image: by giving a natural language query, the given candidate image is sorted; (2)image-to-text: Given an image query, the predefined candidate text sentences are sorted.
Implementation Details
(1) picture processing
We use a pre-trained CNN model to extract its eigenvectors. We use the ResNet-152 model that is trained on a ImageNet dataset that contains 1000 classes, and the output of the ResNet-152 last pooling layer as the visual eigenvector of the image, which has a dimension of 2048, and then enters it into KGRU.
(2) text processing
We first convert all words to lowercase and delete all punctuation marks. Words that appear less than 5 times in a training set are replaced with a special “UNK” symbol, and the maximum length of a sentence is set to 20 if the sentence is too long, ignoring the words that follow.
(3) model config
We set the size of the hidden state of the KGRU for image and sentence encoding to 1024, and the dimension of the word embedding is set to 300. In addition, we use Skip-gram Word2vec, which is trained on the Flickr label, to initialize We. For the model of sentence retrieval graphics, the image and sentence side branches each use a fully connected layer. In the Image Square branch, the size of the full-connection layer weight matrix is 3072×2048. In the sentence Square branch, the size of the full-connection layer weight matrix is 1524×2048. As a result, two branch networks project image and sentence features into 2048-D public spaces.
For the model of image retrieval sentence, the image side branch has two fully connected layers, and its weight matrix size is 3072x2048 and 2048x2048. And the sentence to the side branch also has two full connection layer, its size is 1024x2048 and 2048x2048.
(4)Dataset and Overparameter setting
We evaluate our approach on the MSCOCO dataset, which contains 113,287 training images with five caption seach,5,000 images for validation and 5,000 images for testing.
Based on experience, the initial learning rate is set to η=0.0005, attenuation weights for γ=0.9 and constant e==10-8. To mitigate the model over-fitting problem, set the dropout=0.5 on all hidden layers of the model. The method of updating the learning rate is as follows: once 5 consecutive epoch, the performance of the model on the validation set does not increase, we divide the learning rate by 2. If the performance on the validation set does not improve after 10 consecutive epoch, the training of the model is stopped. We set the maximum number of epoch for training to 50.
Experimental Result
Table 1. Cross-modal retrieval results on MSCOCO 1K-image test set.
As shown in table 1, the results verify the validity of the retrieval model based on the attention mechanism KGRU model for image/sentence expression. The countermeasure generation model has the ability to capture both global semantic information and local information of text and image video data, and can accurately match image and text description in complex content.
Conclusion
In this paper, we have proposed a novel cross-modal representation learning framework, it has the ability to capture the global semantic information and local information of text and image video data at the same time by taking the attention mechanism and the GAN fusion generation process into the cross-modal representation, which incorporate the visual-to-text and the text-to-visual generative models into the conventional cross-modal feature embedding. Experiments have shown that this method can accurately match image video and text description in complex content.
Acknowledgment
This research was supported by the Innovation Team of China West Normal University (No.CXTD2017-6) and Excellence Fund of China West Normal University (No.17Y183).
References
[1] Amir A., Basu S., Iyengar G., et al. A multi-modal system for the retrieval of semantic video events [J]. Computer Vision & Image Understanding, 2004, 96(2): 216-236
[2] Feng, Wang, Li R. Cross-modal retrieval with correspondence auto-encoder [C]. In International Conference on Multimedia. ACM, Orlando, FL, USA, 2014: 7-16．
[3] Ryan Kiros, Ruslan Salakhutdinov, Richard S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models [J]. arXiv preprint arXiv: 1411.2539, 2014.
[4] Xia R., Pan Y., Lai H., et al. Supervised Hashing for Image Retrieval via Image Representation Learning [C]//Proceedings of the AAAI Conference on Artificial Intelligence. Qu´ebec City, Canada: AAAI, 2014: 2156–2162.
