Weakly Supervised Deep Hyperspherical Quantization for Image Retrieval

(1)

Weakly Supervised Deep Hyperspherical Quantization for Image Retrieval

Jinpeng Wang ^1,2* , Bin Chen ^1*† , Qiang Zhang ³ , Zaiqiao Meng ⁴ , Shangsong Liang ^2† , Shutao Xia ¹

1 Tsinghua Shenzhen International Graduate School, Tsinghua University

2 School of Computer Science and Engineering, Sun Yat-sen University

3 University College London

4 University of Cambridge

[email protected], [email protected], [email protected], [email protected], [email protected], [email protected]

Abstract

Deep quantization methods have shown high efficiency on large-scale image retrieval. However, current models heavily rely on ground-truth information, hindering the application of quantization in label-hungry scenarios. A more realistic demand is to learn from inexhaustible uploaded images that are associated with informal tags provided by amateur users.

Though such sketchy tags do not obviously reveal the labels, they actually contain useful semantic information for super- vising deep quantization. To this end, we propose Weakly- Supervised Deep Hyperspherical Quantization (WSDHQ), which is the first work to learn deep quantization from weakly tagged images. Specifically, 1) we use word embeddings to represent the tags and enhance their semantic information based on a tag correlation graph. 2) To better preserve seman- tic information in quantization codes and reduce quantization error, we jointly learn semantics-preserving embeddings and supervised quantizer on hypersphere by employing a well- designed fusion layer and tailor-made loss functions. Exten- sive experiments show that WSDHQ can achieve state-of-art performance on weakly-supervised compact coding.

With the explosive growth of media data on the web, many retrieval tasks need to handle large-scale and high- dimensional data. Due to high computational efficiency and low memory overhead, learning to hash (Wang et al. 2018), as a technique in Approximate Nearest Neighbor (ANN) search, has been applied in many applications. Briefly, the goal of hashing is to transform high-dimensional data into compact binary codes while preserving semantic informa- tion. Based on the ways of measuring distance between en- coded data, hashing methods can be roughly categorized into two types. 1) Binary hashing (Gionis et al. 1999; Salakhut- dinov and Hinton 2009) transforms high-dimensional data into hash codes in Hamming space such that distances are computed with fast bitwise operators. 2) Quantization (Je- gou, Douze, and Schmid 2010; Ge et al. 2013; Babenko and Lempitsky 2014; Zhang, Du, and Wang 2014; Yang, Chen, and Xia 2020) divides high-dimensional data space into dis- joint cells and approximately represents each point by its cell

* Equal contribution.

† Corresponding authors.

Copyright © 2021, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

animal grass

Image T ags Labels !

We attempt to use weak tags instead of ground-truth labels to supervise quantization learning…

Various Semantic Information Tag Sparsity Weak Semantics of a Single Tag

#friends #cute #nature #kitty #kiss

#friendship #cat #wildlife #pair

#kisses #deer #fawn #bambi #smooch

Figure 1: An example from NUS-WIDE dataset to illustrate the problem of weakly supervised quantization using tags.

centroid. Since the pairwise distances between data points are pre-computed by inter-centroid distances and stored in a lookup table, the search speed is accelerated.

Recently, deep learning has been integrated into hash models and yields superior performance over the shallow methods (Zhao et al. 2015; Huang, Chen, and Pan 2019; Jin et al. 2020). Empirically, deep quantization methods have more powerful representation capability than deep binary hashing methods for ANN search (Cao et al. 2016, 2017; Liu et al. 2018; Chen and Cheung 2019; Yuan et al. 2020). Nev- ertheless, there are two concerns about existing deep quanti- zation methods. 1) Data hunger: existing deep quantization methods heavily rely on high-quality supervision. However, despite the advent of large-scale annotated datasets such as ImageNet (Deng et al. 2009), the lack of well-labeled data in specific domains remains a critical bottleneck of deep learning. Collecting massive data with exact labels is usu- ally labor-intensive and expensive, which hinders the de- ployment of deep quantization methods in practical large- scale applications, e.g. search engines and social media. 2) High norm variance of deep features: deep networks often produce representation vectors with relatively high variance on the norm, which adversely degrades the quality of quan- tization (Wu et al. 2017; Eghbali and Tahvildari 2019).

To overcome the heavy dependence of manually anno- tated data, we consider taking the freely available web im- ages with impure tags as training data, and study the novel

The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI- 21)

(2)

problem of weakly supervised deep quantization, as illus- trated in Fig. 1. Different from existing deep supervised quantization methods that leverage clear labels as supervi- sion, we try to tackle the following challenges: 1) Tag spar- sity: the tag words are nearly unrestricted, so many synony- mous tags share similar meanings; the total number of tags could be huge. 2) Weak semantics of a single tag: a single tag can be semantically vague, leading to confusion in learning.

3) Various semantics in one image: an image may contain multiple concepts. Besides, to further improve deep quanti- zation, we also attempt to reduce the high norm variance of deep embeddings in the quantization model.

In this paper, we propose Weakly-Supervised Deep Hyperspherical Quantization (WSDHQ) for learning to quantize with weak tags. To our best knowledge, WSDHQ is the first work to address the problem of weakly-supervised deep quantization without using ground-truth labels. It ex- plores the possibility of disconnecting the deep quantization evolution from the scaling of human-annotated datasets, given free and inexhaustible web and social media data. We make the following contributions in WSDHQ:

1) On the specific task of image quantization, we are the first to consider enhancing the weak supervision of tags.

Concretely, we build a tag embedding correlation graph, to effectively enhance tag semantics and reduce sparsity.

2) To reduce the error of deep quantization, we remove the norm variance of the deep features by applying ` 2 nor- malization and maps visual representations onto a se- mantic hypersphere spanned by tag embeddings.

3) We further improve the ability of quantization model to better preserve semantic information into quantization codes by designing a novel adaptive cosine margin loss and a novel supervised cosine quantization loss, that di- rects the training of our model in an end-to-end manner.

4) Extensive experiments show that WSDHQ yields state- of-art retrieval results in weakly-supervised scenario.

Related Work

Deep Quantization. Deep quantization methods cooperat- ing with CNNs (Krizhevsky, Sutskever, and Hinton 2012; He et al. 2016) have shown superior performance against tradi- tional non-deep methods that use hand-crafted features (Je- gou, Douze, and Schmid 2010; Ge et al. 2013; Kalantidis and Avrithis 2014; Babenko and Lempitsky 2014; Zhang, Du, and Wang 2014; Martinez et al. 2016, 2018). The goal of deep supervised quantization is to learn compact codes through deep networks that are faithful to given semantic information such as pointwise (Cao et al. 2017; Eghbali and Tahvildari 2019; Yuan et al. 2020), pairwise (Cao et al. 2016;

Chen and Cheung 2019) or triplet labels (Yu et al. 2018;

Liu et al. 2018). Despite the promising performance, ex- isting methods largely rely on high-quality labels to learn satisfactory models. It limits the application of deep quan- tization on many real-world scenarios, where a lot of data is available without adequate ground-truths. Different from previous works, we attempt to solve the novel problem of weakly-supervised deep quantization in this paper.

Tackle Norm Variance for Quantization. It has been re- vealed that many adopted deep networks produce represen- tations with relatively large norm variance, which leads to greater quantization error and unexpected performance de- generation (Wu et al. 2017; Eghbali and Tahvildari 2019). To reduce the norm variance, MSQ (Wu et al. 2017) quantized the data norms using an additional scalar quantizer before applying product quantization (PQ). Nevertheless, it is hard to balance the budget between two quantizers, and PQ is in- ferior to other quantization methods due to its orthogonality assumption on codebooks (Babenko and Lempitsky 2014).

DSQ (Eghbali and Tahvildari 2019) removed the norm vari- ance for deep representations by quantizing them on a unit- norm sphere, and learned to close the Euclidean distance be- tween each image embedding and its unique class center.

In this paper, we employ a transformation layer with ` 2

normalization as DSQ did, to embed deep image represen- tations onto a semantic hypersphere spanned by tag em- beddings. Different from DSQ, we learn quantization in a multi-semantics (i.e., multi-label and multi-class) settings by jointly optimizing two novel cosine losses, i.e., adaptive co- sine margin loss and supervised cosine quantization loss.

Weakly-supervised Hashing. There has been less attention paid to weakly-supervised hashing, where meta-data (weak tags) attached to web images is freely available to be an in- exhaustible source of weak supervision (Gomez et al. 2018).

To our knowledge, there has been no weakly-supervised quantization method for ANN search until now. Although there have been a few initial attempts (Zhang et al. 2016;

Tang and Li 2018; Guan et al. 2018; Gattupalli, Zhuo, and Li 2019; Cui et al. 2020) in weakly-supervised binary hash- ing, they either rely on pure labels to rectify the model or directly train the model using raw tags without effec- tive semantic enhancement, which leaves a lot of room for improvement. Zhang et al. (2016) used collaborative filter- ing to predict tag-label associations, where labels were re- quested. Guan et al. (2018) proposed a two-stage frame- work consisting of weakly supervised pre-training and fine- tuning using ground-truth labels. Technically, the above two methods are not ideal to be truly weakly-supervised hash- ing, because of the dependence on ground-truth information.

Weakly Supervised Multimodal Hashing (WMH) (Tang and Li 2018) is the first holistically weakly-supervised hashing method, which constructs binary matrix of tags and for- mulates hashing as an eigenvalue problem. It tends to be vulnerable because WMH directly uses weak tags as bi- nary labels. Weakly Supervised Deep Hashing using Tag Embeddings (WDHT) (Gattupalli, Zhuo, and Li 2019) em- ploys Word2Vec (Mikolov et al. 2013) embeddings for tags.

Although noises and vagueness are partially alleviated in WDHT, the tags of each image are roughly represented as the average vector of tag embeddings by simple weighting strategies, which is too rough to grasp various semantics.

Our WSDHQ is the first weakly supervised deep quanti-

zation method. Different from existing weakly supervised

binary hashing methods, there are two improvements: 1)

WSDHQ investigates the semantic relations of weak tags,

based on which, it further enhances tag semantics and re-

duces similar tags confusion so as to improve the supervi-

(3)

sion. 2) WSDHQ preserves more fine-grained semantic rep- resentations during training, which helps to get better per- formance. We will discuss the effects of these two strategies (i.e., loss designs) in the ablation experiment.

Proposed Approach

In this section, we first formulate the research problem and give a brief overview of our approach. Then we present the supervision enhancement and the quantization on hyper- sphere. Finally, we introduce the overall learning algorithm.

Problem Formulation

In a weakly-supervised image retrieval scenario, we are given a training set of N images with attached tag sets {x ⁿ , Y ⁿ } ^N n=1 , where each image is represented as a P - dimensional vector x n ∈ R ^P and is associated with a set of textual tags Y ⁿ ⊂ Y, Y is a set containing all tags.

While the web images with tags are easy to obtain from search engines or social media, the raw tags are not re- liable enough to be ground-truth labels. Thus the goal of weakly-supervised deep quantization is to learn a composi- tional quantizer q : x ∈ R ^P 7→ b ∈ {0, 1} ^B , which encodes each point x into a compact B-bit binary code b by preserv- ing the weak semantics from tags via DNNs.

Overview of Our Approach

This paper enables efficient image retrieval by present- ing Weakly Supervised Deep Hyperspherical Quantization (WSDHQ) in an end-to-end deep learning architecture, as shown in Fig. 2. WSDHQ consists of five main components:

1) A standard CNN f , e.g. AlexNet (Krizhevsky, Sutskever, and Hinton 2012), to extract deep features of images. 2) A word embedding model, e.g. Word2Vec (Mikolov et al.

2013), to represent each tag y i ∈ Y as an embedding t ⁱ , while T and T ⁿ are tag embedding sets w.r.t. raw tag sets Y and Y ⁿ . 3) A correlation graph G, to enhance the semantics of tags and merge synonymous tags. We denote semantically enhanced tag sets via G as e T and e T ⁿ . 4) A transform layer g, to embed deep features v n as r n on a hypersphere, which is spanned by the normalized tags S. 5) A semantic hyper- sphere, on which the norm variances of data points are re- duced to zeros. With the help of two well-customized cosine losses, our joint process of semantics-preserving learning for image embedding r n and quantization on the hypersphere yield better compact codes for ANN search.

Enhance Weak Supervision with Tag Correlation When using attached tags as supervision, we first consider the issues of weak semantics of single tags and tag sparsity.

Semantic Correlation Graph. We first extract the pre- trained Word2Vec embedding t i ∈ R ^D for each informative tag y i ∈ Y, where D is the dimension of textual embedding space, then form the tag embedding sets T and {T ⁿ } ^N n=1

w.r.t. Y and {Y ⁿ } ^N n=1 . Let T ∈ R ^{D×|T |} be the tag embed- ding matrix. Next we obtain a k-nearest neighbor set NN ^k (i) by cosine similarity for each tag t i . A semantic correlation

graph G is then constructed on the total tag set T by an ad- jacency matrix A = (a ij ) ∈ {0, 1} ^{|T |×|T |} , where

a ij =

( 1, if j ∈ NN ^k (i) and _kt ^t ^> ⁱ ^t ^j

i kkt j k ≥ τ, or just i = j, 0, otherwise,

and τ is the correlation threshold that determines whether a neighbor tag is related to an anchor tag in semantics. Note that each row a i in A indicates the semantic similarities be- tween t i and other tags on G.

Semantic Enhancement on Graph. Since the neighbor in- formation from G can effectively enhance the semantic rep- resentations of single tags and alleviate their semantic bi- ases, we can obtain an semantics-enhanced tag embedding matrix e T = T ˜ A ^> by aggregating the embeddings from its neighbor tags, where ˜ A is the row-wise normalization of A.

Reduce Sparse Tags. After semantic enhancement, the rep- resentations of similar tags are expected to move closer and aggregate in small regions, so those sparse and redundant tags can be further detected and merged. The merging pro- cess follows classic density-based clustering algorithm (Es- ter et al. 1996). Any t ⁰ _i , t ⁰ _j on G are density-reachable to each other if d(t ⁰ _i , t ⁰ _j ) < , where is the merging thresh- old and d( ·, ·) is a distance metric, e.g. the ` ² distance. Each time we pick up one unprocessed point t ⁰ _i from G in order and compute its density-reachable set R(i, ) = {t ⁰ j | t ⁰ j is density-reachable from t ⁰ _i }. Once |R(i, )| > 1, we merge and replace all the elements in R(i, ) by the average point.

Finally, we obtain refreshed embedding sets e T and { e T ⁿ } ^N n=1

by removing same embeddings in each tag set.

Quantization on Semantic Hypersphere

We adopt a standard CNN f : R ^P 7→ R ^V for extracting V -dimensional deep features v n = f (x n ) of image x n . S = {t ⁰ i / kt ⁰ i k ² } ^{| e} i=1 ^{T |} and S ⁿ = {t ⁰ i / kt ⁰ i k ² } ^{| e} i=1 ^T ⁿ ^| are the nor- malized total tag set and the normalized tag set of image x n , which span the D-dimensional hyperspherical seman- tic space. We set up a transform layer g : R ^V 7→ R ^D with ` 2 normalization to remove norm variance of deep features v n and map v n onto the semantic hypersphere as r n = g(v n ) = _kσ(W ^σ(W ^g ^v ⁿ ⁾

g v n )k 2 , where σ( ·) is the activation, e.g. the hyperbolic tangent (tanh), and W g is the parameter matrix of g. For brevity, we use h = f ◦ g to denote the fusion of network f and layer g such that r n = h(x n ).

Semantics-preserving Learning on Hypersphere. We pro- pose an adaptive cosine margin loss L ⁿ to enable semantics- preserving learning on hypersphere, namely

L ⁿ = X

s ⁺ _i ∈S n

X

s ⁻ _j ∈N n

h

∆ ij − cos θ (s ⁺ _i ,r n ) + cos θ _(s − j ,r n )

i

+ , where [ ·] + = max(0, ·), cos θ ^(v,v ⁰ ⁾ = hv, v ⁰ i, s.t. kvk ² = kv ⁰ k ² = 1, N ⁿ is the negative semantic set of x n and ∆ is the cosine margin. This metric loss encourages the image embedding r n to move closer to positive semantic embed- dings while pushing it from negative embeddings.

Ideally, we would like to utilize all negative semantic em-

beddings of x n , i.e., N ⁿ = S\S ⁿ for learning, but this can

(4)

Image T ags

#friends #cute #nature #kitty

#kiss #friendship #cat

#wildlife #pair #kisses #deer

#fawn #bambi #smooch

semantic hypersphere

+ +

-

✓ (r n ,s j )

<latexit sha1_base64="mrzhTa+ZJD58tJD29ly0x16fcy8=">AAAC8XicjVHLSsNAFD3GV31XXboJFkFBS+LGLotuXEkFq4KtIUnHNjYvZiZCKf0Jd+7ErT/gVn9C/AP9C++MEaoiOiGZc8+952TuXC8NAyEt62XEGB0bn5gsTE3PzM7NLxQXl45FknGf1f0kTPip5woWBjGry0CG7DTlzI28kJ143T2VP7liXARJfCR7KWtGbjsOLgLflUQ5xc2G7DDpOv31hpeELdGLaDO5E28Ox8K5PN/aGDjFklW29DJ/AjsHpWpl6qAKoJYUn9FACwl8ZIjAEEMSDuFC0HMGGxZS4proE8cJBTrPMMA0aTOqYlThEtulb5uis5yNKVaeQqt9+ktILyeliTXSJFTHCau/mTqfaWfF/ubd157qbD3avdwrIlaiQ+xfus/K/+pULxIXqOgeAuop1Yzqzs9dMn0r6uTmUFeSHFLiFG5RnhP2tfLznk2tEbp3dbeuzr/qSsWq2M9rM7ypU9KA7e/j/AmOt8u2VbYPadK7+FgFrGAV6zTPHVSxjxrq5H2NBzziyRDGjXFr3H2UGiO5ZhlflnH/DrFxoqE=</latexit><latexit sha1_base64="PbmqxkscL6KVLQS1dPN1e5BtMok=">AAAC8XicjVHLSsNAFD3GV31XXboJiqCgJRHEgpuiG1eiYFWwNSTpqLF5MTMRSulPuHMnrgR/wK3uXYt/oH/hnTGCWkQnJHPuufeczJ3rpWEgpGW99Bi9ff0Dg4Wh4ZHRsfGJ4uTUvkgy7rOqn4QJP/RcwcIgZlUZyJAdppy5kReyA6+5qfIHF4yLIIn3ZCtl9cg9jYOTwHclUU5xqSbPmHSd9kLNS8KGaEW0mdyJl77Gwjk/Xl7sOMU5q2TpZXYDOwdzlfLQ9vrT6u1OUnxGDQ0k8JEhAkMMSTiEC0HPEWxYSImro00cJxToPEMHw6TNqIpRhUtsk76nFB3lbEyx8hRa7dNfQno5KU3MkyahOk5Y/c3U+Uw7K/Y377b2VGdr0e7lXhGxEmfE/qX7rPyvTvUicYKy7iGgnlLNqO783CXTt6JObn7pSpJDSpzCDcpzwr5Wft6zqTVC967u1tX5V12pWBX7eW2GN3VKGrD9c5zdYH+lZFsle5cmvYGPVcAMZrFA81xDBVvYQZW8L3GPBzwawrgyro2bj1KjJ9dM49sy7t4BTgKkRg==</latexit><latexit sha1_base64="PbmqxkscL6KVLQS1dPN1e5BtMok=">AAAC8XicjVHLSsNAFD3GV31XXboJiqCgJRHEgpuiG1eiYFWwNSTpqLF5MTMRSulPuHMnrgR/wK3uXYt/oH/hnTGCWkQnJHPuufeczJ3rpWEgpGW99Bi9ff0Dg4Wh4ZHRsfGJ4uTUvkgy7rOqn4QJP/RcwcIgZlUZyJAdppy5kReyA6+5qfIHF4yLIIn3ZCtl9cg9jYOTwHclUU5xqSbPmHSd9kLNS8KGaEW0mdyJl77Gwjk/Xl7sOMU5q2TpZXYDOwdzlfLQ9vrT6u1OUnxGDQ0k8JEhAkMMSTiEC0HPEWxYSImro00cJxToPEMHw6TNqIpRhUtsk76nFB3lbEyx8hRa7dNfQno5KU3MkyahOk5Y/c3U+Uw7K/Y377b2VGdr0e7lXhGxEmfE/qX7rPyvTvUicYKy7iGgnlLNqO783CXTt6JObn7pSpJDSpzCDcpzwr5Wft6zqTVC967u1tX5V12pWBX7eW2GN3VKGrD9c5zdYH+lZFsle5cmvYGPVcAMZrFA81xDBVvYQZW8L3GPBzwawrgyro2bj1KjJ9dM49sy7t4BTgKkRg==</latexit><latexit sha1_base64="XOwIaG6bTucLdsWcCgVpSKpdUY0=">AAAC8XicjVHLSsNAFD3Gd31VXboJFqFCLYkbXRbduKxgtdBqSNKxjc2LmYlQij/hzp249Qfc6k+If6B/4Z0xQlVEJyRz7rn3nMyd66VhIKRlvYwZ4xOTU9Mzs4W5+YXFpeLyyrFIMu6zhp+ECW96rmBhELOGDGTImilnbuSF7MTr76v8ySXjIkjiIzlI2WnkduPgPPBdSZRTrLRlj0nXGZbbXhJ2xCCizeROXBmNhXNxtrV55RRLVtXSy/wJ7ByUkK96UnxGGx0k8JEhAkMMSTiEC0FPCzYspMSdYkgcJxToPMMVCqTNqIpRhUtsn75dilo5G1OsPIVW+/SXkF5OShMbpEmojhNWfzN1PtPOiv3Ne6g91dkGtHu5V0SsRI/Yv3Sflf/VqV4kzrGrewiop1Qzqjs/d8n0raiTmyNdSXJIiVO4Q3lO2NfKz3s2tUbo3tXdujr/qisVq2I/r83wpk5JA7a/j/MnON6u2lbVPrRKtb181DNYwzrKNM8d1HCAOhrkfY0HPOLJEMaNcWvcfZQaY7lmFV+Wcf8OWj+hkg==</latexit>

✓ (r n ,s ⁺ i )

<latexit sha1_base64="1pALxwYLiT45pZpdDQLGfw23KCo=">AAAC8XicjVHLSsNAFD3G97vq0k1QhIpSEhF0KbpxZwWrQqshSac6mBczE6EUP8CtO3fi1h9wqz8h/oH+hXfGFKoiOiGZc8+952Tu3CCLuFSO89pn9Q8MDg2PjI6NT0xOTZdmZg9lmouQ1cI0SsVx4EsW8YTVFFcRO84E8+MgYkfBxY7OH10yIXmaHKh2xk5i/yzhLR76iiivtNpQ50z5XqfcCNKoKdsxbbbwktXeWHr8dGX5yistOhXHLPsncAuwuLV+vXcHoJqWXtBAEylC5IjBkEARjuBD0lOHCwcZcSfoECcIcZNnuMIYaXOqYlThE3tB3zOK6gWbUKw9pVGH9JeIXkFKG0ukSalOENZ/s00+N86a/c27Yzz12dq0B4VXTKzCObF/6bqV/9XpXhRa2DQ9cOopM4zuLixccnMr+uR2T1eKHDLiNG5SXhAOjbJ7z7bRSNO7vlvf5N9MpWZ1HBa1Od71KWnA7vdx/gSHaxXXqbj7NOltfK4RzGMBZZrnBrawiypq5H2DRzzh2ZLWrXVn3X+WWn2FZg5flvXwAZJAo3s=</latexit><latexit sha1_base64="TrK60X/FYsk2RoKQpVMGmSK06ig=">AAAC8XicjVHLLgRBFD3aa7wHSwsdIiFk0i0SlsLGzkgMEkOnu6dQmX6lqloymVha2NpJLMTWD9jyE+IP+Au3Sks8IlSnu849957TdesGWcSlcpznDquzq7unt9TXPzA4NDxSHh3bkWkuQlYL0ygVe4EvWcQTVlNcRWwvE8yPg4jtBs11nd89ZULyNNlWrYwdxP5xwo946CuivPJCXZ0w5Xvt2XqQRg3ZimmzhZcsfI6lxw/n58688rRTccyyfwK3ANOrSxeb15PnT9W0/IQ6GkgRIkcMhgSKcAQfkp59uHCQEXeANnGCEDd5hjP0kzanKkYVPrFN+h5TtF+wCcXaUxp1SH+J6BWktDFDmpTqBGH9N9vkc+Os2d+828ZTn61Fe1B4xcQqnBD7l+6j8r863YvCEVZMD5x6ygyjuwsLl9zcij65/akrRQ4ZcRo3KC8Ih0b5cc+20UjTu75b3+RfTKVmdRwWtTle9SlpwO73cf4EO4sV16m4WzTpNbyvEiYwhVma5zJWsYEqauR9iXs84NGS1pV1Y92+l1odhWYcX5Z19wbI66Ty</latexit><latexit sha1_base64="TrK60X/FYsk2RoKQpVMGmSK06ig=">AAAC8XicjVHLLgRBFD3aa7wHSwsdIiFk0i0SlsLGzkgMEkOnu6dQmX6lqloymVha2NpJLMTWD9jyE+IP+Au3Sks8IlSnu849957TdesGWcSlcpznDquzq7unt9TXPzA4NDxSHh3bkWkuQlYL0ygVe4EvWcQTVlNcRWwvE8yPg4jtBs11nd89ZULyNNlWrYwdxP5xwo946CuivPJCXZ0w5Xvt2XqQRg3ZimmzhZcsfI6lxw/n58688rRTccyyfwK3ANOrSxeb15PnT9W0/IQ6GkgRIkcMhgSKcAQfkp59uHCQEXeANnGCEDd5hjP0kzanKkYVPrFN+h5TtF+wCcXaUxp1SH+J6BWktDFDmpTqBGH9N9vkc+Os2d+828ZTn61Fe1B4xcQqnBD7l+6j8r863YvCEVZMD5x6ygyjuwsLl9zcij65/akrRQ4ZcRo3KC8Ih0b5cc+20UjTu75b3+RfTKVmdRwWtTle9SlpwO73cf4EO4sV16m4WzTpNbyvEiYwhVma5zJWsYEqauR9iXs84NGS1pV1Y92+l1odhWYcX5Z19wbI66Ty</latexit><latexit sha1_base64="9ME0PTuV3SYbrluhItXUjuzrTN0=">AAAC8XicjVHLSsNAFD3GV31XXboJFkFRSuJGl6IblxVsK7QaknRsB/NiZiKU4k+4cydu/QG3+hPiH+hfeGeMoBbRCcmce+49J3PnBlnEpXKclxFrdGx8YrI0NT0zOze/UF5casg0FyGrh2mUipPAlyziCasrriJ2kgnmx0HEmsHFgc43L5mQPE2OVT9jp7HfTfg5D31FlFfeaqseU743WG8HadSR/Zg2W3jJ1tdYevxsc+PKK1ecqmOWPQzcAlRQrFpafkYbHaQIkSMGQwJFOIIPSU8LLhxkxJ1iQJwgxE2e4QrTpM2pilGFT+wFfbsUtQo2oVh7SqMO6S8RvYKUNtZIk1KdIKz/Zpt8bpw1+5v3wHjqs/VpDwqvmFiFHrF/6T4r/6vTvSicY9f0wKmnzDC6u7Bwyc2t6JPbX7pS5JARp3GH8oJwaJSf92wbjTS967v1Tf7VVGpWx2FRm+NNn5IG7P4c5zBobFddp+oeOZW9/WLUJaxgFes0zx3s4RA11Mn7Gg94xJMlrRvr1rr7KLVGCs0yvi3r/h1TF6GP</latexit>

✓ _(s + i ,s _j )

<latexit sha1_base64="2qewf4kNjueiAFImmzY32VT6wCo=">AAAC83icjVHLSgMxFD0dX63Pqks3gyIoaplxo8uiG1dSwVahrWVmGm3svJhkhFL8C3fuxK0/4Fb/QfwD/Qtv0hGsIpphkpNz7znJzXVjnwtpWa85Y2R0bHwiX5icmp6ZnSvOL9RElCYeq3qRHyWnriOYz0NWlVz67DROmBO4Pjtxu/sqfnLFEsGj8Fj2YtYMnIuQn3PPkUS1iqWG7DDptPprDTfy26IX0GKKFj/b2BxmLs+21q9bxRWrZOlh/gR2BlbKu4XDMoBKVHxBA21E8JAiAEMISdiHA0FfHTYsxMQ10ScuIcR1nOEak6RNKYtRhkNsl+YL2tUzNqS98hRa7dEpPv0JKU2skiaivISwOs3U8VQ7K/Y37772VHfr0epmXgGxEh1i/9J9Zv5Xp2qROMeuroFTTbFmVHVe5pLqV1E3N79UJckhJk7hNsUTwp5Wfr6zqTVC167e1tHxN52pWLX3stwU7+qW1GD7ezt/gtp2ybZK9hF1eg+DkccSlrFG/dxBGQeooEreN3jEE56N1Lg17oz7QaqRyzSLGBrGwwdXY6M6</latexit><latexit sha1_base64="rRgFAR3awjv6ai0zNrR1yFz2iV0=">AAAC83icjVHLSsQwFD3W93vUpZviIIyoQyuIghvRjStRcFSYF20no9G+aFJhGPwLd+7EhRt/wK1uXYt/oH/hTeyAOoimNDk5956T3Fw39rmQlvXaY/T29Q8MDg2PjI6NT0zmpqYPRZQmHit5kR8lx64jmM9DVpJc+uw4TpgTuD47cs+3VfzogiWCR+GBbMWsGjgnIW9yz5FE1XPFijxl0qm3CxU38huiFdBiijqvLS59Z85qywuX9VzeKlp6mN3AzkB+c314d+N59W4vyr2gggYieEgRgCGEJOzDgaCvDBsWYuKqaBOXEOI6znCJEdKmlMUowyH2nOYT2pUzNqS98hRa7dEpPv0JKU3MkyaivISwOs3U8VQ7K/Y377b2VHdr0epmXgGxEqfE/qXrZP5Xp2qRaGJd18Cpplgzqjovc0n1q6ibm1+qkuQQE6dwg+IJYU8rO+9sao3Qtau3dXT8TWcqVu29LDfFu7olNdj+2c5ucLhStK2ivU+d3sLnGMIs5lCgfq5hEzvYQ4m8r/CARzwZqXFt3Bi3n6lGT6aZwbdh3H8A8+Wk3w==</latexit><latexit sha1_base64="rRgFAR3awjv6ai0zNrR1yFz2iV0=">AAAC83icjVHLSsQwFD3W93vUpZviIIyoQyuIghvRjStRcFSYF20no9G+aFJhGPwLd+7EhRt/wK1uXYt/oH/hTeyAOoimNDk5956T3Fw39rmQlvXaY/T29Q8MDg2PjI6NT0zmpqYPRZQmHit5kR8lx64jmM9DVpJc+uw4TpgTuD47cs+3VfzogiWCR+GBbMWsGjgnIW9yz5FE1XPFijxl0qm3CxU38huiFdBiijqvLS59Z85qywuX9VzeKlp6mN3AzkB+c314d+N59W4vyr2gggYieEgRgCGEJOzDgaCvDBsWYuKqaBOXEOI6znCJEdKmlMUowyH2nOYT2pUzNqS98hRa7dEpPv0JKU3MkyaivISwOs3U8VQ7K/Y377b2VHdr0epmXgGxEqfE/qXrZP5Xp2qRaGJd18Cpplgzqjovc0n1q6ibm1+qkuQQE6dwg+IJYU8rO+9sao3Qtau3dXT8TWcqVu29LDfFu7olNdj+2c5ucLhStK2ivU+d3sLnGMIs5lCgfq5hEzvYQ4m8r/CARzwZqXFt3Bi3n6lGT6aZwbdh3H8A8+Wk3w==</latexit><latexit sha1_base64="IhwXWf1fUV/vkZQOFMcgM8/VuIY=">AAAC83icjVHLSgMxFD0dX7W+qi7dDBZBUcuMG12KblxWsA9obZmZRo2dF5OMUIp/4c6duPUH3Oo/iH+gf+FNHMFaRDNMcnLuPSe5uW7scyEt6zVnjI1PTE7lpwszs3PzC8XFpZqI0sRjVS/yo6ThOoL5PGRVyaXPGnHCnMD1Wd3tHap4/YolgkfhiezH7DRwzkN+xj1HEtUpllvygkmnM1hvuZHfFf2AFlN0eHtza5i5bG9vXHeKJats6WGOAjsDJWSjEhVf0EIXETykCMAQQhL24UDQ14QNCzFxpxgQlxDiOs5wjQJpU8pilOEQ26P5nHbNjA1przyFVnt0ik9/QkoTa6SJKC8hrE4zdTzVzor9zXugPdXd+rS6mVdArMQFsX/pvjL/q1O1SJxhT9fAqaZYM6o6L3NJ9auom5vfqpLkEBOncJfiCWFPK7/e2dQaoWtXb+vo+JvOVKzae1luind1S2qw/bOdo6C2U7atsn1slfYPslbnsYJVrFM/d7GPI1RQJe8bPOIJz0Zq3Bp3xv1nqpHLNMsYGsbDBwAxois=</latexit>

s ⁺ _i

<latexit sha1_base64="yCKd93NZNbvVCd0d+HWci6E6x/k=">AAAC1nicjVHLSsNAFD2Nr/pOdekmWAVBKIkbXRbduKxgH9DWkqSjDs2LzEQpRXfi1h9wq58k/oEu/AfvTCP4QHRCZs6ce8+ZuXO9JOBC2vZzwZiYnJqeKc7OzS8sLi2bpZWGiLPUZ3U/DuK05bmCBTxidcllwFpJytzQC1jTGxyoePOCpYLH0bEcJqwbumcRP+W+K4nqmaWOFwd9MQxpsUSPn2z3zLJdsfWwfgInB+XqRvhWBVCLzSd00EcMHxlCMESQhAO4EPS14cBGQlwXI+JSQlzHGa4wR9qMshhluMQOaD6jXTtnI9orT6HVPp0S0J+S0sImaWLKSwmr0ywdz7SzYn/zHmlPdbchrV7uFRIrcU7sX7qPzP/qVC0Sp9jTNXCqKdGMqs7PXTL9Kurm1qeqJDkkxCncp3hK2NfKj3e2tEbo2tXbujr+ojMVq/Z+npvhVd2SGux8b+dP0NipOHbFOaJO72M8iljDOraon7uo4hA11Mn7Evd4wKPRMq6NG+N2nGoUcs0qvgzj7h01rphH</latexit><latexit sha1_base64="ZLWuWMAoad+Bj1J6m9qEYmhKhU8=">AAAC1nicjVHLSgMxFD0dX7W+Wt3pZvABglBm3OiyKIK4qmAfULXMTKMG58Uko5SiO3HrD7jVP1L8A134D96kU/CBaIZJTs695yQ31419LqRlveSMoeGR0bH8eGFicmp6pliarYsoTTxW8yI/SpquI5jPQ1aTXPqsGSfMCVyfNdzzbRVvXLBE8Cg8kN2YHQXOachPuOdIotrF0qEb+R3RDWgxRZsfr7WLS1bZ0sP8CewMLFWWg/e9+aedalR8xiE6iOAhRQCGEJKwDweCvhZsWIiJO0KPuIQQ13GGKxRIm1IWowyH2HOaT2nXytiQ9spTaLVHp/j0J6Q0sUKaiPISwuo0U8dT7azY37x72lPdrUurm3kFxEqcEfuXbpD5X52qReIEm7oGTjXFmlHVeZlLql9F3dz8VJUkh5g4hTsUTwh7Wjl4Z1NrhK5dva2j4686U7Fq72W5Kd7ULanB9vd2/gT19bJtle196vQW+iOPBSxilfq5gQp2UUWNvC9xjwc8Gk3j2rgxbvupRi7TzOHLMO4+APIDmYY=</latexit><latexit sha1_base64="ZLWuWMAoad+Bj1J6m9qEYmhKhU8=">AAAC1nicjVHLSgMxFD0dX7W+Wt3pZvABglBm3OiyKIK4qmAfULXMTKMG58Uko5SiO3HrD7jVP1L8A134D96kU/CBaIZJTs695yQ31419LqRlveSMoeGR0bH8eGFicmp6pliarYsoTTxW8yI/SpquI5jPQ1aTXPqsGSfMCVyfNdzzbRVvXLBE8Cg8kN2YHQXOachPuOdIotrF0qEb+R3RDWgxRZsfr7WLS1bZ0sP8CewMLFWWg/e9+aedalR8xiE6iOAhRQCGEJKwDweCvhZsWIiJO0KPuIQQ13GGKxRIm1IWowyH2HOaT2nXytiQ9spTaLVHp/j0J6Q0sUKaiPISwuo0U8dT7azY37x72lPdrUurm3kFxEqcEfuXbpD5X52qReIEm7oGTjXFmlHVeZlLql9F3dz8VJUkh5g4hTsUTwh7Wjl4Z1NrhK5dva2j4686U7Fq72W5Kd7ULanB9vd2/gT19bJtle196vQW+iOPBSxilfq5gQp2UUWNvC9xjwc8Gk3j2rgxbvupRi7TzOHLMO4+APIDmYY=</latexit><latexit sha1_base64="GfYEy66MmNyqHe2eUz5BUQdRqCk=">AAAC1nicjVHLSgMxFD0dX7W+Wl26GSyCIJQZN7osunFZwT6grWVmmtbQeTHJKKXoTtz6A271k8Q/0L/wJk5BLaIZJjk5956T3Fw39rmQlvWaM+bmFxaX8suFldW19Y1iabMhojTxWN2L/ChpuY5gPg9ZXXLps1acMCdwfdZ0Rycq3rxiieBReC7HMesGzjDkA+45kqhesdRxI78vxgEtpujxi/1esWxVLD3MWWBnoIxs1KLiCzroI4KHFAEYQkjCPhwI+tqwYSEmrosJcQkhruMMNyiQNqUsRhkOsSOah7RrZ2xIe+UptNqjU3z6E1Ka2CVNRHkJYXWaqeOpdlbsb94T7anuNqbVzbwCYiUuif1LN838r07VIjHAka6BU02xZlR1XuaS6ldRNze/VCXJISZO4T7FE8KeVk7f2dQaoWtXb+vo+JvOVKzae1luind1S2qw/bOds6BxULGtin1mlavHWavz2MYO9qifh6jiFDXUyfsaj3jCs9Eybo074/4z1chlmi18G8bDB86Ylkk=</latexit>

s _j

<latexit sha1_base64="VthyfEKxAnDw/AOVId4sqky7LpE=">AAAC1nicjVFLTsMwFJyGXym/FpZsIiokNlQJGxCrCjYsi0Q/UluqJHVLID/FDqiqYIfYcgG2cAEOwC0QN4Bb8OymElAhcBR7PO/N2M/PjjyXC8N4y2hT0zOzc9n53MLi0vJKvrBa42ESO6zqhF4YN2yLM88NWFW4wmONKGaWb3usbl8cynj9ksXcDYMTMYhY27f6gdtzHUsQ1ckXWnbodfnAp0XnnfPT7U6+aJQMNfRJYKagWC7tP78AqIT5V7TQRQgHCXwwBBCEPVjg9DVhwkBEXBtD4mJCroozXCNH2oSyGGVYxF7Q3KddM2UD2ktPrtQOneLRH5NSxyZpQsqLCcvTdBVPlLNkf/MeKk95twGtdurlEytwRuxfunHmf3WyFoEe9lQNLtUUKUZW56QuiXoVeXP9S1WCHCLiJO5SPCbsKOX4nXWl4ap2+baWir+rTMnKvZPmJviQt6QGmz/bOQlqOyXTKJnH1OkDjEYW69jAFvVzF2UcoYIqeV/hAY940hrajXar3Y1StUyqWcO3od1/An5nmGg=</latexit><latexit sha1_base64="zrMyHv8eQRx00azXVXKBySuFtY0=">AAAC1nicjVG7TsMwFD0N73eBkSUCITFAlbCAmBAsjCBRWolClbhuCXUeih1QhWBDrIwsrPADSHwK4g/gL7g2qQRUCBzFPj73nmNfXz8RgVSO81qw+voHBoeGR0bHxicmp4rTMwcyzlLGyywWcVr1PclFEPGyCpTg1STlXugLXvHb2zpeOeOpDOJoX3USfhR6rShoBsxTRNWL0zU/Fg3ZCWmxZf30eKVeXHBKjhl2L3BzsLBZ2nh6Xt67242LL6ihgRgMGUJwRFCEBTxI+g7hwkFC3BEuiEsJBSbOcYlR0maUxSnDI7ZNc4t2hzkb0V57SqNmdIqgPyWljUXSxJSXEtan2SaeGWfN/uZ9YTz13Tq0+rlXSKzCCbF/6bqZ/9XpWhSaWDc1BFRTYhhdHctdMvMq+ub2l6oUOSTEadygeEqYGWX3nW2jkaZ2/baeib+ZTM3qPctzM7zrW1KD3Z/t7AUHqyXXKbl71OktfI5hzGEeS9TPNWxiB7sok/c57vGAR6tqXVnX1s1nqlXINbP4NqzbD8pfmXQ=</latexit><latexit sha1_base64="zrMyHv8eQRx00azXVXKBySuFtY0=">AAAC1nicjVG7TsMwFD0N73eBkSUCITFAlbCAmBAsjCBRWolClbhuCXUeih1QhWBDrIwsrPADSHwK4g/gL7g2qQRUCBzFPj73nmNfXz8RgVSO81qw+voHBoeGR0bHxicmp4rTMwcyzlLGyywWcVr1PclFEPGyCpTg1STlXugLXvHb2zpeOeOpDOJoX3USfhR6rShoBsxTRNWL0zU/Fg3ZCWmxZf30eKVeXHBKjhl2L3BzsLBZ2nh6Xt67242LL6ihgRgMGUJwRFCEBTxI+g7hwkFC3BEuiEsJBSbOcYlR0maUxSnDI7ZNc4t2hzkb0V57SqNmdIqgPyWljUXSxJSXEtan2SaeGWfN/uZ9YTz13Tq0+rlXSKzCCbF/6bqZ/9XpWhSaWDc1BFRTYhhdHctdMvMq+ub2l6oUOSTEadygeEqYGWX3nW2jkaZ2/baeib+ZTM3qPctzM7zrW1KD3Z/t7AUHqyXXKbl71OktfI5hzGEeS9TPNWxiB7sok/c57vGAR6tqXVnX1s1nqlXINbP4NqzbD8pfmXQ=</latexit><latexit sha1_base64="H0u55wqzZ12eJXgI59f6CC8IssM=">AAAC1nicjVHLTsMwEJyGVymvFo5cIiokLlQJFzhWcOFYJPqQ2lIlqVtC85LtgKoKbogrP8AVPgnxB/AXrE2QgAqBo9jj2Z2x1+smgS+kZb3kjJnZufmF/GJhaXllda1YWm+IOOUeq3txEPOW6wgW+BGrS18GrJVw5oRuwJru6EjFm5eMCz+OTuU4Yd3QGUb+wPccSVSvWOq4cdAX45AWU/QuznZ7xbJVsfQwp4GdgTKyUYuLz+igjxgeUoRgiCAJB3Ag6GvDhoWEuC4mxHFCvo4zXKNA2pSyGGU4xI5oHtKunbER7ZWn0GqPTgno56Q0sU2amPI4YXWaqeOpdlbsb94T7anuNqbVzbxCYiXOif1L95n5X52qRWKAA12DTzUlmlHVeZlLql9F3dz8UpUkh4Q4hfsU54Q9rfx8Z1NrhK5dva2j4686U7Fq72W5Kd7ULanB9s92ToPGXsW2KvaJVa4eZq3OYxNb2KF+7qOKY9RQJ+8rPOART0bLuDFujbuPVCOXaTbwbRj379W6lkw=</latexit>

r n

<latexit sha1_base64="k8ur5PN8sSmCiAVLkX6B96MxrVU=">AAAC1HicjVG7TsMwFD0Nr1IeLSCxsEQgJKYqYYGxgoWxlehDaquSpKZEzUu2g1QVJsTKD7DCNyEWZvgLrt1UAioEjmIfn3vOta+vmwS+kJb1mjPm5hcWl/LLhZXVtfViaWOzIeKUe6zuxUHMW64jWOBHrC59GbBWwpkTugFrusNTFW9eMy78ODqXo4R1Q2cQ+Ze+50iieqVix42DvhiFtJi8R8yeVbb0MGeBnYG9ynbt7QJANS69oIM+YnhIEYIhgiQcwIGgrw0bFhLiuhgTxwn5Os5wiwJ5U1IxUjjEDmke0K6dsRHtVU6h3R6dEtDPyWlinzwx6ThhdZqp46nOrNjfco91TnW3Ea1uliskVuKK2L98U+V/faoWiUsc6xp8qinRjKrOy7Kk+lXUzc0vVUnKkBCncJ/inLCnndN3NrVH6NrV2zo6/q6VilV7L9Om+FC3pAbbP9s5CxqHZdsq2zXq9AkmI48d7OKA+nmECs5QRV33/BFPeDYaxo1xZ9xPpEYu82zh2zAePgFp95eV</latexit><latexit sha1_base64="hiPSZy7cEPsbNxYThu6JPuRy0Q4=">AAAC1HicjVG7SgQxFD07vtb1tSrY2AwugtUyY6OlaGO5gvuAdZGZ2ajBzIMkIyxqJbb+gK2W/oX/IDbW+hfexFnwgWiGSU7OPecmNzfMBFfa855Lzsjo2PhEebIyNT0zO1edX2ipNJcRa0apSGUnDBQTPGFNzbVgnUyyIA4Fa4enOybePmNS8TTZ14OM9eLgOOFHPAo0UYfVuYMwFX01iGlx5SExNa/u2eH+BH4BaltLey/8YfuxkVafcIA+UkTIEYMhgSYsEEDR14UPDxlxPZwTJwlxG2e4RIW8OakYKQJiT2k+pl23YBPam5zKuiM6RdAvyelilTwp6SRhc5pr47nNbNjfcp/bnOZuA1rDIldMrMYJsX/5hsr/+kwtGkfYtDVwqimzjKkuKrLk9lXMzd1PVWnKkBFncJ/iknBkncN3dq1H2drN2wY2/mqVhjX7qNDmeDO3pAb739v5E7TW675X9/eo09v4GGUsYwVr1M8NbGEXDTRtz29xh3un5Vw4V871h9QpFZ5FfBnOzTs45plR</latexit><latexit sha1_base64="hiPSZy7cEPsbNxYThu6JPuRy0Q4=">AAAC1HicjVG7SgQxFD07vtb1tSrY2AwugtUyY6OlaGO5gvuAdZGZ2ajBzIMkIyxqJbb+gK2W/oX/IDbW+hfexFnwgWiGSU7OPecmNzfMBFfa855Lzsjo2PhEebIyNT0zO1edX2ipNJcRa0apSGUnDBQTPGFNzbVgnUyyIA4Fa4enOybePmNS8TTZ14OM9eLgOOFHPAo0UYfVuYMwFX01iGlx5SExNa/u2eH+BH4BaltLey/8YfuxkVafcIA+UkTIEYMhgSYsEEDR14UPDxlxPZwTJwlxG2e4RIW8OakYKQJiT2k+pl23YBPam5zKuiM6RdAvyelilTwp6SRhc5pr47nNbNjfcp/bnOZuA1rDIldMrMYJsX/5hsr/+kwtGkfYtDVwqimzjKkuKrLk9lXMzd1PVWnKkBFncJ/iknBkncN3dq1H2drN2wY2/mqVhjX7qNDmeDO3pAb739v5E7TW675X9/eo09v4GGUsYwVr1M8NbGEXDTRtz29xh3un5Vw4V871h9QpFZ5FfBnOzTs45plR</latexit><latexit sha1_base64="4T1N+hGxFTGx5vDH/EfXqB0JC5E=">AAAC1HicjVHLSgMxFD0dX/XZqks3g0VwVWbc6LLoxmUF+4C2lJlpWodmHmQyQqmuxK0/4Fa/SfwD/QtvYgpqEc0wycm559zk5vopDzPpOK8Fa2FxaXmluLq2vrG5VSpv7zSzJBcBawQJT0Tb9zLGw5g1ZCg5a6eCeZHPWcsfn6l465qJLEziSzlJWS/yRnE4DANPEtUvl7p+wgfZJKLFFn1iKk7V0cOeB64BFZhRT8ov6GKABAFyRGCIIQlzeMjo68CFg5S4HqbECUKhjjPcYo28OakYKTxixzSPaNcxbEx7lTPT7oBO4fQLcto4IE9COkFYnWbreK4zK/a33FOdU91tQqtvckXESlwR+5dvpvyvT9UiMcSJriGkmlLNqOoCkyXXr6Jubn+pSlKGlDiFBxQXhAPtnL2zrT2Zrl29rafjb1qpWLUPjDbHu7olNdj92c550Dyquk7VvXAqtVPT6iL2sI9D6ucxajhHHQ3d80c84dlqWjfWnXX/KbUKxrOLb8N6+AA6dZWw</latexit>

quantization

000 001 010 011 100 101 110

000 111

001 010

011 100

101 111 110 semantics-preserving

learning

Part 5 convolutional neural network

··· ···

Part 1 ···

transform layer

v _n

<latexit sha1_base64="fOxRr1mr/p2eeAfuZ9IeHvxvNoM=">AAAC1HicjVG7TsMwFD0Nr1IeLSCxsEQgJKYqYYGxgoWxlehDKhUkqVuiOnGUOJWqwoRY+QFW+CbEwgx/wbWbSkCFwFHs43PPufb1dSPuJ9KyXnPG3PzC4lJ+ubCyurZeLG1sNhKRxh6re4KLuOU6CeN+yOrSl5y1opg5gctZ0x2cqnhzyOLEF+G5HEWsEzj90O/5niOJuiwVL1zBu8kooMUcXhKzZ5UtPcxZYGdgr7Jde7sCUBWlF1ygCwEPKQIwhJCEORwk9LVhw0JEXAdj4mJCvo4z3KJA3pRUjBQOsQOa+7RrZ2xIe5Uz0W6PTuH0x+Q0sU8eQbqYsDrN1PFUZ1bsb7nHOqe624hWN8sVECtxTexfvqnyvz5Vi0QPx7oGn2qKNKOq87IsqX4VdXPzS1WSMkTEKdyleEzY087pO5vak+ja1ds6Ov6ulYpVey/TpvhQt6QG2z/bOQsah2XbKts16vQJJiOPHezigPp5hArOUEVd9/wRT3g2GsaNcWfcT6RGLvNs4dswHj4Bc3+XmQ==</latexit><latexit sha1_base64="6/R+X+yyo2Bwfe+xRw+1iYnGjvQ=">AAAC1HicjVHLSsNAFD2Nr1ofrQpu3ASL4KokbnRZ6sZlC/YBtZQkndaheZFMCqW6Erf+gFtd+hf+g7hxrX/hnWkKahGdkJkz555zZ+5cO3R5LAzjNaMtLC4tr2RXc2vrG5v5wtZ2Iw6SyGF1J3CDqGVbMXO5z+qCC5e1wohZnu2ypj08lfHmiEUxD/xzMQ5Zx7MGPu9zxxJEdQv5Cztwe/HYo0UfdYkpGiVDDX0emCkolndrb/yp8lwNCi+4QA8BHCTwwOBDEHZhIaavDRMGQuI6mBAXEeIqznCNHHkTUjFSWMQOaR7Qrp2yPu1lzli5HTrFpT8ip44D8gSkiwjL03QVT1Rmyf6We6JyyruNabXTXB6xApfE/uWbKf/rk7UI9HGiauBUU6gYWZ2TZknUq8ib61+qEpQhJE7iHsUjwo5yzt5ZV55Y1S7f1lLxd6WUrNw7qTbBh7wlNdj82c550DgqmUbJrFGnK5iOLPawj0Pq5zHKOEMVddXzezzgUWtoV9qNdjuVapnUs4NvQ7v7BEJumVU=</latexit><latexit sha1_base64="6/R+X+yyo2Bwfe+xRw+1iYnGjvQ=">AAAC1HicjVHLSsNAFD2Nr1ofrQpu3ASL4KokbnRZ6sZlC/YBtZQkndaheZFMCqW6Erf+gFtd+hf+g7hxrX/hnWkKahGdkJkz555zZ+5cO3R5LAzjNaMtLC4tr2RXc2vrG5v5wtZ2Iw6SyGF1J3CDqGVbMXO5z+qCC5e1wohZnu2ypj08lfHmiEUxD/xzMQ5Zx7MGPu9zxxJEdQv5Cztwe/HYo0UfdYkpGiVDDX0emCkolndrb/yp8lwNCi+4QA8BHCTwwOBDEHZhIaavDRMGQuI6mBAXEeIqznCNHHkTUjFSWMQOaR7Qrp2yPu1lzli5HTrFpT8ip44D8gSkiwjL03QVT1Rmyf6We6JyyruNabXTXB6xApfE/uWbKf/rk7UI9HGiauBUU6gYWZ2TZknUq8ib61+qEpQhJE7iHsUjwo5yzt5ZV55Y1S7f1lLxd6WUrNw7qTbBh7wlNdj82c550DgqmUbJrFGnK5iOLPawj0Pq5zHKOEMVddXzezzgUWtoV9qNdjuVapnUs4NvQ7v7BEJumVU=</latexit><latexit sha1_base64="x5yvGmvScXLqTlpX7cgOeFIeLbY=">AAAC1HicjVHLTsMwEJyGVymPFjhyiaiQOFUJFzhWcOFYJPqQWlQlrgtRkziKnUpV4YS48gNc4ZsQfwB/wdqkElAhcBR7PDuz9nr9JAykcpzXgrWwuLS8Ulwtra1vbJYrW9stKbKU8SYToUg7vid5GMS8qQIV8k6Sci/yQ972R6c63h7zVAYivlCThF9G3lUcDAPmKaL6lXLPF+FATiJa7HGfmKpTc8yw54Gbgyry0RCVF/QwgABDhggcMRThEB4kfV24cJAQd4kpcSmhwMQ5blEib0YqTgqP2BHNV7Tr5mxMe51TGjejU0L6U3La2CePIF1KWJ9mm3hmMmv2t9xTk1PfbUKrn+eKiFW4JvYv30z5X5+uRWGIY1NDQDUlhtHVsTxLZl5F39z+UpWiDAlxGg8onhJmxjl7Z9t4pKldv61n4m9GqVm9Z7k2w7u+JTXY/dnOedA6rLlOzT13qvWTvNVF7GIPB9TPI9RxhgaapuePeMKz1bJurDvr/lNqFXLPDr4N6+EDQ/2VtA==</latexit>

r

<latexit sha1_base64="k8ur5PN8sSmCiAVLkX6B96MxrVU=">AAAC1HicjVG7TsMwFD0Nr1IeLSCxsEQgJKYqYYGxgoWxlehDaquSpKZEzUu2g1QVJsTKD7DCNyEWZvgLrt1UAioEjmIfn3vOta+vmwS+kJb1mjPm5hcWl/LLhZXVtfViaWOzIeKUe6zuxUHMW64jWOBHrC59GbBWwpkTugFrusNTFW9eMy78ODqXo4R1Q2cQ+Ze+50iieqVix42DvhiFtJi8R8yeVbb0MGeBnYG9ynbt7QJANS69oIM+YnhIEYIhgiQcwIGgrw0bFhLiuhgTxwn5Os5wiwJ5U1IxUjjEDmke0K6dsRHtVU6h3R6dEtDPyWlinzwx6ThhdZqp46nOrNjfco91TnW3Ea1uliskVuKK2L98U+V/faoWiUsc6xp8qinRjKrOy7Kk+lXUzc0vVUnKkBCncJ/inLCnndN3NrVH6NrV2zo6/q6VilV7L9Om+FC3pAbbP9s5CxqHZdsq2zXq9AkmI48d7OKA+nmECs5QRV33/BFPeDYaxo1xZ9xPpEYu82zh2zAePgFp95eV</latexit><latexit sha1_base64="hiPSZy7cEPsbNxYThu6JPuRy0Q4=">AAAC1HicjVG7SgQxFD07vtb1tSrY2AwugtUyY6OlaGO5gvuAdZGZ2ajBzIMkIyxqJbb+gK2W/oX/IDbW+hfexFnwgWiGSU7OPecmNzfMBFfa855Lzsjo2PhEebIyNT0zO1edX2ipNJcRa0apSGUnDBQTPGFNzbVgnUyyIA4Fa4enOybePmNS8TTZ14OM9eLgOOFHPAo0UYfVuYMwFX01iGlx5SExNa/u2eH+BH4BaltLey/8YfuxkVafcIA+UkTIEYMhgSYsEEDR14UPDxlxPZwTJwlxG2e4RIW8OakYKQJiT2k+pl23YBPam5zKuiM6RdAvyelilTwp6SRhc5pr47nNbNjfcp/bnOZuA1rDIldMrMYJsX/5hsr/+kwtGkfYtDVwqimzjKkuKrLk9lXMzd1PVWnKkBFncJ/iknBkncN3dq1H2drN2wY2/mqVhjX7qNDmeDO3pAb739v5E7TW675X9/eo09v4GGUsYwVr1M8NbGEXDTRtz29xh3un5Vw4V871h9QpFZ5FfBnOzTs45plR</latexit><latexit sha1_base64="hiPSZy7cEPsbNxYThu6JPuRy0Q4=">AAAC1HicjVG7SgQxFD07vtb1tSrY2AwugtUyY6OlaGO5gvuAdZGZ2ajBzIMkIyxqJbb+gK2W/oX/IDbW+hfexFnwgWiGSU7OPecmNzfMBFfa855Lzsjo2PhEebIyNT0zO1edX2ipNJcRa0apSGUnDBQTPGFNzbVgnUyyIA4Fa4enOybePmNS8TTZ14OM9eLgOOFHPAo0UYfVuYMwFX01iGlx5SExNa/u2eH+BH4BaltLey/8YfuxkVafcIA+UkTIEYMhgSYsEEDR14UPDxlxPZwTJwlxG2e4RIW8OakYKQJiT2k+pl23YBPam5zKuiM6RdAvyelilTwp6SRhc5pr47nNbNjfcp/bnOZuA1rDIldMrMYJsX/5hsr/+kwtGkfYtDVwqimzjKkuKrLk9lXMzd1PVWnKkBFncJ/iknBkncN3dq1H2drN2wY2/mqVhjX7qNDmeDO3pAb739v5E7TW675X9/eo09v4GGUsYwVr1M8NbGEXDTRtz29xh3un5Vw4V871h9QpFZ5FfBnOzTs45plR</latexit><latexit sha1_base64="4T1N+hGxFTGx5vDH/EfXqB0JC5E=">AAAC1HicjVHLSgMxFD0dX/XZqks3g0VwVWbc6LLoxmUF+4C2lJlpWodmHmQyQqmuxK0/4Fa/SfwD/QtvYgpqEc0wycm559zk5vopDzPpOK8Fa2FxaXmluLq2vrG5VSpv7zSzJBcBawQJT0Tb9zLGw5g1ZCg5a6eCeZHPWcsfn6l465qJLEziSzlJWS/yRnE4DANPEtUvl7p+wgfZJKLFFn1iKk7V0cOeB64BFZhRT8ov6GKABAFyRGCIIQlzeMjo68CFg5S4HqbECUKhjjPcYo28OakYKTxixzSPaNcxbEx7lTPT7oBO4fQLcto4IE9COkFYnWbreK4zK/a33FOdU91tQqtvckXESlwR+5dvpvyvT9UiMcSJriGkmlLNqOoCkyXXr6Jubn+pSlKGlDiFBxQXhAPtnL2zrT2Zrl29rafjb1qpWLUPjDbHu7olNdj92c550Dyquk7VvXAqtVPT6iL2sI9D6ucxajhHHQ3d80c84dlqWjfWnXX/KbUKxrOLb8N6+AA6dZWw</latexit>

n

Part 4 database of all tags

Y

<latexit sha1_base64="kkGNCEUPsAjW1c7Kyx+xPn/aHK8=">AAACznicjVHLSsNAFD2Nr/quunQTLIKrkoigy6IblxXsQ2qRyXTahubFZFIopbj1B9zqZ4l/oH/hnTEFtYhOSHLm3HPuzL3XSwI/VY7zWrAWFpeWV4qra+sbm1vbpZ3dRhpnkos6j4NYtjyWisCPRF35KhCtRAoWeoFoesMLHW+OhEz9OLpW40R0QtaP/J7PmSKqfRsyNeAsmNxM70plp+KYZc8DNwdl5KsWl15wiy5icGQIIRBBEQ7AkNLThgsHCXEdTIiThHwTF5hijbwZqQQpGLFD+vZp187ZiPY6Z2rcnE4J6JXktHFInph0krA+zTbxzGTW7G+5JyanvtuY/l6eKyRWYUDsX76Z8r8+XYtCD2emBp9qSgyjq+N5lsx0Rd/c/lKVogwJcRp3KS4Jc+Oc9dk2ntTUrnvLTPzNKDWr9zzXZnjXt6QBuz/HOQ8axxXXqbhXJ+XqeT7qIvZxgCOa5ymquEQNddPxRzzh2apZI2tq3X9KrULu2cO3ZT18AJ9Sk/A=</latexit><latexit sha1_base64="kkGNCEUPsAjW1c7Kyx+xPn/aHK8=">AAACznicjVHLSsNAFD2Nr/quunQTLIKrkoigy6IblxXsQ2qRyXTahubFZFIopbj1B9zqZ4l/oH/hnTEFtYhOSHLm3HPuzL3XSwI/VY7zWrAWFpeWV4qra+sbm1vbpZ3dRhpnkos6j4NYtjyWisCPRF35KhCtRAoWeoFoesMLHW+OhEz9OLpW40R0QtaP/J7PmSKqfRsyNeAsmNxM70plp+KYZc8DNwdl5KsWl15wiy5icGQIIRBBEQ7AkNLThgsHCXEdTIiThHwTF5hijbwZqQQpGLFD+vZp187ZiPY6Z2rcnE4J6JXktHFInph0krA+zTbxzGTW7G+5JyanvtuY/l6eKyRWYUDsX76Z8r8+XYtCD2emBp9qSgyjq+N5lsx0Rd/c/lKVogwJcRp3KS4Jc+Oc9dk2ntTUrnvLTPzNKDWr9zzXZnjXt6QBuz/HOQ8axxXXqbhXJ+XqeT7qIvZxgCOa5ymquEQNddPxRzzh2apZI2tq3X9KrULu2cO3ZT18AJ9Sk/A=</latexit><latexit sha1_base64="kkGNCEUPsAjW1c7Kyx+xPn/aHK8=">AAACznicjVHLSsNAFD2Nr/quunQTLIKrkoigy6IblxXsQ2qRyXTahubFZFIopbj1B9zqZ4l/oH/hnTEFtYhOSHLm3HPuzL3XSwI/VY7zWrAWFpeWV4qra+sbm1vbpZ3dRhpnkos6j4NYtjyWisCPRF35KhCtRAoWeoFoesMLHW+OhEz9OLpW40R0QtaP/J7PmSKqfRsyNeAsmNxM70plp+KYZc8DNwdl5KsWl15wiy5icGQIIRBBEQ7AkNLThgsHCXEdTIiThHwTF5hijbwZqQQpGLFD+vZp187ZiPY6Z2rcnE4J6JXktHFInph0krA+zTbxzGTW7G+5JyanvtuY/l6eKyRWYUDsX76Z8r8+XYtCD2emBp9qSgyjq+N5lsx0Rd/c/lKVogwJcRp3KS4Jc+Oc9dk2ntTUrnvLTPzNKDWr9zzXZnjXt6QBuz/HOQ8axxXXqbhXJ+XqeT7qIvZxgCOa5ymquEQNddPxRzzh2apZI2tq3X9KrULu2cO3ZT18AJ9Sk/A=</latexit><latexit sha1_base64="kkGNCEUPsAjW1c7Kyx+xPn/aHK8=">AAACznicjVHLSsNAFD2Nr/quunQTLIKrkoigy6IblxXsQ2qRyXTahubFZFIopbj1B9zqZ4l/oH/hnTEFtYhOSHLm3HPuzL3XSwI/VY7zWrAWFpeWV4qra+sbm1vbpZ3dRhpnkos6j4NYtjyWisCPRF35KhCtRAoWeoFoesMLHW+OhEz9OLpW40R0QtaP/J7PmSKqfRsyNeAsmNxM70plp+KYZc8DNwdl5KsWl15wiy5icGQIIRBBEQ7AkNLThgsHCXEdTIiThHwTF5hijbwZqQQpGLFD+vZp187ZiPY6Z2rcnE4J6JXktHFInph0krA+zTbxzGTW7G+5JyanvtuY/l6eKyRWYUDsX76Z8r8+XYtCD2emBp9qSgyjq+N5lsx0Rd/c/lKVogwJcRp3KS4Jc+Oc9dk2ntTUrnvLTPzNKDWr9zzXZnjXt6QBuz/HOQ8axxXXqbhXJ+XqeT7qIvZxgCOa5ymquEQNddPxRzzh2apZI2tq3X9KrULu2cO3ZT18AJ9Sk/A=</latexit>

correlation graph

···

word embedding model

{t

<latexit sha1_base64="3+rhXm2MWATWufkKH4RVckablTY=">AAAC8XicjVHLbtQwFD1NKX1AYSjLbiIKEotqlHQDm0oj2LAs0rykZiZyPJ6pVeeh2EEapfMT7NghtvwA2/YnEH8Ai/5Dr91U6kNVcRT7+Nx7jn19k0JJbYLgz5K3/Gjl8era+saTp5vPnrdebPV1XpVc9Hiu8nKYMC2UzETPSKPEsCgFSxMlBsnxRxsffBGllnnWNfNCjFI2y+RUcmaIilu7UR0luZroeUqLb2I5zqJFXMv9cDGuT6KUmSPOlN+Ns5NF3NoJ2oEb/l0QNmCn8zo97wA4yFu/EWGCHBwVUghkMIQVGDR9hwgRoCBuhJq4kpB0cYEFNkhbUZagDEbsMc0z2h02bEZ766mdmtMpiv6SlD7ekCanvJKwPc138co5W/Y+79p52rvNaU0ar5RYgyNiH9JdZf6vztZiMMV7V4OkmgrH2Op441K5V7E3969VZcihIM7iCcVLwtwpr97Zdxrtardvy1z8r8u0rN3zJrfCP3tLanB4u513QX+vHQbt8DN1+gMuxxq28QpvqZ/v0MEnHKBH3l/xC6c487T3zfvu/bhM9ZYazUvcGN7PC2HtpDw=</latexit><latexit sha1_base64="DD+zlKhwvqBD3p809f8tU84GRdI=">AAAC8XicjVHLTtwwFD0T+qDQlindlU1UWqmLapSwoZtKqAip6gokZgaJQOR4DFg4D8VOpVGYn2DHDrHtD3RbPgLEH8CCf+DaZKQWVIGj2Mfn3nPs65sUSmoTBBctb+LJ02fPJ19MTb989Xqm/Wa2p/Oq5KLLc5WXGwnTQslMdI00SmwUpWBpokQ/2V+28f5PUWqZZ+tmWIitlO1mckdyZoiK25+jOkpyNdDDlBbfxHI7i0ZxLb+Go+36IEqZ2eNM+etxdjCK2/NBJ3DDvw/CBswvfUivf7w7W1nN2+eIMEAOjgopBDIYwgoMmr5NhAhQELeFmriSkHRxgRGmSFtRlqAMRuw+zbu022zYjPbWUzs1p1MU/SUpfXwkTU55JWF7mu/ilXO27P+8a+dp7zakNWm8UmIN9oh9SDfOfKzO1mKwgy+uBkk1FY6x1fHGpXKvYm/u/1WVIYeCOIsHFC8Jc6ccv7PvNNrVbt+Wufily7Ss3fMmt8KVvSU1OLzbzvugt9AJg064Rp3+htsxiTm8xyfq5yKW8B2r6JL3IX7jD0497R15x97JbarXajRv8c/wft0AHlGlew==</latexit><latexit sha1_base64="DD+zlKhwvqBD3p809f8tU84GRdI=">AAAC8XicjVHLTtwwFD0T+qDQlindlU1UWqmLapSwoZtKqAip6gokZgaJQOR4DFg4D8VOpVGYn2DHDrHtD3RbPgLEH8CCf+DaZKQWVIGj2Mfn3nPs65sUSmoTBBctb+LJ02fPJ19MTb989Xqm/Wa2p/Oq5KLLc5WXGwnTQslMdI00SmwUpWBpokQ/2V+28f5PUWqZZ+tmWIitlO1mckdyZoiK25+jOkpyNdDDlBbfxHI7i0ZxLb+Go+36IEqZ2eNM+etxdjCK2/NBJ3DDvw/CBswvfUivf7w7W1nN2+eIMEAOjgopBDIYwgoMmr5NhAhQELeFmriSkHRxgRGmSFtRlqAMRuw+zbu022zYjPbWUzs1p1MU/SUpfXwkTU55JWF7mu/ilXO27P+8a+dp7zakNWm8UmIN9oh9SDfOfKzO1mKwgy+uBkk1FY6x1fHGpXKvYm/u/1WVIYeCOIsHFC8Jc6ccv7PvNNrVbt+Wufily7Ss3fMmt8KVvSU1OLzbzvugt9AJg064Rp3+htsxiTm8xyfq5yKW8B2r6JL3IX7jD0497R15x97JbarXajRv8c/wft0AHlGlew==</latexit><latexit sha1_base64="Lhp3sdZSkjrVDQsIOIvBR7IEYXg=">AAAC8XicjVHLTtwwFD2E8igFOtBlN1FHlbqoRgkb2FQawYblIM0AEoHI8RiwcB6KHaRRmJ9gxw6x5QfYwk+g/gH8Ra/dIEFRBY5iH597z7Gvb1IoqU0Q/J7wJj9MTc/Mfpz7NL+w+Lm1tLyt86rkYsBzlZe7CdNCyUwMjDRK7BalYGmixE5ysmHjO6ei1DLP+mZUiP2UHWXyUHJmiIpbP6M6SnI11KOUFt/E8iCLxnEtf4Xjg/osSpk55kz5/Tg7G8etdtAJ3PBfg7ABbTSjl7fuEWGIHBwVUghkMIQVGDR9ewgRoCBuHzVxJSHp4gJjzJG2oixBGYzYE5qPaLfXsBntrad2ak6nKPpLUvr4Tpqc8krC9jTfxSvnbNn/edfO095tRGvSeKXEGhwT+5buKfO9OluLwSHWXA2SaiocY6vjjUvlXsXe3H9WlSGHgjiLhxQvCXOnfHpn32m0q92+LXPxB5dpWbvnTW6FR3tLanD4bztfg+2VThh0wq2g3V1vWj2Lr/iGH9TPVXSxiR4G5H2OG9ziztPehXfpXf1N9SYazRe8GN71H/rXoj4=</latexit>

ⁿ i } ^|T i=1 ⁿ ^|

{t

<latexit sha1_base64="D044u0WpD7yODBTFZry80QdgskM=">AAAC7XicjVHLThsxFD1MC6SURyhLNqNSJFbRTDewQYpg02UqJQGJoZHHMcSKZzya8SBFk/wCu+6qbvsD3ZbfQPwBLPiHXjuD1BZVrUdjH597z7Gvb5wpWZgguFvwXrxcXFpuvFp5vbq2vtHcfNMvdJlz0eNa6fw0ZoVQMhU9I40Sp1kuWBIrcRKPj2385ErkhdRp10wycZ6wy1ReSM4MUYPmXlRFsVbDYpLQ4puBjGaDSh6Gs0/VNEqYGXGm/O50NmjuBK3ADf85CGuw036XPLYBdHTzFhGG0OAokUAghSGswFDQd4YQATLizlERlxOSLi4wwwppS8oSlMGIHdN8Sbuzmk1pbz0Lp+Z0iqI/J6WPXdJoyssJ29N8Fy+ds2X/5l05T3u3Ca1x7ZUQazAi9l+6p8z/1dlaDC5w4GqQVFPmGFsdr11K9yr25v4vVRlyyIizeEjxnDB3yqd39p2mcLXbt2Uufu8yLWv3vM4t8WBvSQ0O/2znc9B/3wqDVviROn2E+WhgG2+xR/3cRxsf0EGPvK/xHT9w42nvs/fF+zpP9RZqzRZ+G963n7wbons=</latexit><latexit sha1_base64="V3bJcujHI8vQR8spaSDsE1oHpm4=">AAAC7XicjVHLTtwwFD2ElvIqTemu3USdIrEaJd20m0qICqnqikoMIJHpyPF4BgsnjhIHaRTyC911h7rlB9i2n9GKP4BF/6HXJkgFhMBR7ONz7zn29U1yJUsThmdT3vSjxzNPZufmFxafLj3zny9vl7oquOhxrXSxm7BSKJmJnpFGid28ECxNlNhJDj7a+M6hKEqpsy0zyUU/ZeNMjiRnhqiBvxrXcaLVsJyktARmIONmUMsPUfO1PopTZvY5U8HWUTPwO2E3dCO4DaIWdNbepH8/v/y9san9P4gxhAZHhRQCGQxhBYaSvj1ECJET10dNXEFIurhAg3nSVpQlKIMRe0DzmHZ7LZvR3nqWTs3pFEV/QcoAK6TRlFcQtqcFLl45Z8ve5V07T3u3Ca1J65USa7BP7H26q8yH6mwtBiO8dzVIqil3jK2Oty6VexV78+C/qgw55MRZPKR4QZg75dU7B05Tutrt2zIXP3eZlrV73uZWuLC3pAZHN9t5G2y/7UZhN/pCnV7H5ZjFK7zGKvXzHdbwCZvokfc3nOInfnna++4dez8uU72pVvMC14Z38g94f6O6</latexit><latexit sha1_base64="V3bJcujHI8vQR8spaSDsE1oHpm4=">AAAC7XicjVHLTtwwFD2ElvIqTemu3USdIrEaJd20m0qICqnqikoMIJHpyPF4BgsnjhIHaRTyC911h7rlB9i2n9GKP4BF/6HXJkgFhMBR7ONz7zn29U1yJUsThmdT3vSjxzNPZufmFxafLj3zny9vl7oquOhxrXSxm7BSKJmJnpFGid28ECxNlNhJDj7a+M6hKEqpsy0zyUU/ZeNMjiRnhqiBvxrXcaLVsJyktARmIONmUMsPUfO1PopTZvY5U8HWUTPwO2E3dCO4DaIWdNbepH8/v/y9san9P4gxhAZHhRQCGQxhBYaSvj1ECJET10dNXEFIurhAg3nSVpQlKIMRe0DzmHZ7LZvR3nqWTs3pFEV/QcoAK6TRlFcQtqcFLl45Z8ve5V07T3u3Ca1J65USa7BP7H26q8yH6mwtBiO8dzVIqil3jK2Oty6VexV78+C/qgw55MRZPKR4QZg75dU7B05Tutrt2zIXP3eZlrV73uZWuLC3pAZHN9t5G2y/7UZhN/pCnV7H5ZjFK7zGKvXzHdbwCZvokfc3nOInfnna++4dez8uU72pVvMC14Z38g94f6O6</latexit><latexit sha1_base64="k7Jco+3WfjaQEWc6EndHxR/DvJI=">AAAC7XicjVHLbhMxFD0ZHg0phQDLbkZElbqKZrppN0gV3bAMUl5SpkQex0msesajGQ9SNM0vsGOH2PIDbOE3EH/Q/kWvzUQCoqr1aOzjc+859vWNMyULEwS/G96Dh48e7zSftHaf7j173n7xcljoMudiwLXS+ThmhVAyFQMjjRLjLBcsiZUYxRdnNj76KPJC6rRvVpk4T9gilXPJmSFq2j6MqijWalasElp8M5XRelrJN+H6Q3UZJcwsOVN+/3I9bXeCbuCGvw3CGnRQj55u/0KEGTQ4SiQQSGEIKzAU9E0QIkBG3Dkq4nJC0sUF1miRtqQsQRmM2AuaF7Sb1GxKe+tZODWnUxT9OSl9HJBGU15O2J7mu3jpnC17m3flPO3dVrTGtVdCrMGS2Lt0m8z76mwtBnOcuBok1ZQ5xlbHa5fSvYq9uf9XVYYcMuIsnlE8J8ydcvPOvtMUrnb7tszFr1ymZe2e17klru0tqcHh/+3cBsOjbhh0w/dB5/Rt3eom9vEah9TPY5ziHXoYkPcnfMcP/PS099n74n39k+o1as0r/DO8bzdVFKB9</latexit>

i } ^{|T |} i=1

Y ⁿ

<latexit sha1_base64="shDCyDxx/O98gfVsxX6NL38cXzI=">AAAC0HicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdCm6cVnFPqQtZTKdtqF5mUxEKUXc+gNu9avEP9C/8M6YglpEJyQ5c+49Z+be60Sem0jLes0ZM7Nz8wv5xcLS8srqWnF9o5aEacxFlYdeGDcclgjPDURVutITjSgWzHc8UXeGJypevxZx4obBhbyNRNtn/cDtuZxJototn8kBZ97octwJOsWSVbb0MqeBnYESslUJiy9ooYsQHCl8CASQhD0wJPQ0YcNCRFwbI+JiQq6OC4xRIG1KWYIyGLFD+vZp18zYgPbKM9FqTqd49MakNLFDmpDyYsLqNFPHU+2s2N+8R9pT3e2W/k7m5RMrMSD2L90k8786VYtED4e6BpdqijSjquOZS6q7om5ufqlKkkNEnMJdiseEuVZO+mxqTaJrV71lOv6mMxWr9jzLTfGubkkDtn+OcxrU9sq2VbbP9ktHx9mo89jCNnZpngc4wikqqJL3FR7xhGfj3Lgx7oz7z1Qjl2k28W0ZDx/bZ5TR</latexit><latexit sha1_base64="shDCyDxx/O98gfVsxX6NL38cXzI=">AAAC0HicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdCm6cVnFPqQtZTKdtqF5mUxEKUXc+gNu9avEP9C/8M6YglpEJyQ5c+49Z+be60Sem0jLes0ZM7Nz8wv5xcLS8srqWnF9o5aEacxFlYdeGDcclgjPDURVutITjSgWzHc8UXeGJypevxZx4obBhbyNRNtn/cDtuZxJototn8kBZ97octwJOsWSVbb0MqeBnYESslUJiy9ooYsQHCl8CASQhD0wJPQ0YcNCRFwbI+JiQq6OC4xRIG1KWYIyGLFD+vZp18zYgPbKM9FqTqd49MakNLFDmpDyYsLqNFPHU+2s2N+8R9pT3e2W/k7m5RMrMSD2L90k8786VYtED4e6BpdqijSjquOZS6q7om5ufqlKkkNEnMJdiseEuVZO+mxqTaJrV71lOv6mMxWr9jzLTfGubkkDtn+OcxrU9sq2VbbP9ktHx9mo89jCNnZpngc4wikqqJL3FR7xhGfj3Lgx7oz7z1Qjl2k28W0ZDx/bZ5TR</latexit><latexit sha1_base64="shDCyDxx/O98gfVsxX6NL38cXzI=">AAAC0HicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdCm6cVnFPqQtZTKdtqF5mUxEKUXc+gNu9avEP9C/8M6YglpEJyQ5c+49Z+be60Sem0jLes0ZM7Nz8wv5xcLS8srqWnF9o5aEacxFlYdeGDcclgjPDURVutITjSgWzHc8UXeGJypevxZx4obBhbyNRNtn/cDtuZxJototn8kBZ97octwJOsWSVbb0MqeBnYESslUJiy9ooYsQHCl8CASQhD0wJPQ0YcNCRFwbI+JiQq6OC4xRIG1KWYIyGLFD+vZp18zYgPbKM9FqTqd49MakNLFDmpDyYsLqNFPHU+2s2N+8R9pT3e2W/k7m5RMrMSD2L90k8786VYtED4e6BpdqijSjquOZS6q7om5ufqlKkkNEnMJdiseEuVZO+mxqTaJrV71lOv6mMxWr9jzLTfGubkkDtn+OcxrU9sq2VbbP9ktHx9mo89jCNnZpngc4wikqqJL3FR7xhGfj3Lgx7oz7z1Qjl2k28W0ZDx/bZ5TR</latexit><latexit sha1_base64="shDCyDxx/O98gfVsxX6NL38cXzI=">AAAC0HicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdCm6cVnFPqQtZTKdtqF5mUxEKUXc+gNu9avEP9C/8M6YglpEJyQ5c+49Z+be60Sem0jLes0ZM7Nz8wv5xcLS8srqWnF9o5aEacxFlYdeGDcclgjPDURVutITjSgWzHc8UXeGJypevxZx4obBhbyNRNtn/cDtuZxJototn8kBZ97octwJOsWSVbb0MqeBnYESslUJiy9ooYsQHCl8CASQhD0wJPQ0YcNCRFwbI+JiQq6OC4xRIG1KWYIyGLFD+vZp18zYgPbKM9FqTqd49MakNLFDmpDyYsLqNFPHU+2s2N+8R9pT3e2W/k7m5RMrMSD2L90k8786VYtED4e6BpdqijSjquOZS6q7om5ufqlKkkNEnMJdiseEuVZO+mxqTaJrV71lOv6mMxWr9jzLTfGubkkDtn+OcxrU9sq2VbbP9ktHx9mo89jCNnZpngc4wikqqJL3FR7xhGfj3Lgx7oz7z1Qjl2k28W0ZDx/bZ5TR</latexit>

Part 2 a training sample

x

<latexit sha1_base64="NQUv5etHyWo1mTBDoVYUs2OVU/8=">AAAC1HicjVG7TsMwFD0Nr1JeBSQWlogKialKWGBEsDC2En1IbQWJ65aoeclxEFVhQqz8ACt8E2Jhhr/g2k0loELgKPbxuedc+/q6se8l0rJec8bM7Nz8Qn6xsLS8srpWXN+oJ1EqGK+xyI9E03US7nshr0lP+rwZC+4Ers8b7uBExRtXXCReFJ7JYcw7gdMPvZ7HHEnUeXGt7UZ+NxkGtJjX58SUrLKlhzkN7AyUjraqbxcAKlHxBW10EYEhRQCOEJKwDwcJfS3YsBAT18GIOEHI03GOWxTIm5KKk8IhdkBzn3atjA1pr3Im2s3oFJ9+QU4Tu+SJSCcIq9NMHU91ZsX+lnukc6q7DWl1s1wBsRKXxP7lmyj/61O1SPRwqGvwqKZYM6o6lmVJ9auom5tfqpKUISZO4S7FBWGmnZN3NrUn0bWrt3V0/F0rFav2LNOm+FC3pAbbP9s5Der7Zdsq21Xq9DHGI49t7GCP+nmAI5yigpru+SOe8GzUjRvjzrgfS41c5tnEt2E8fAJ4Q5eb</latexit><latexit sha1_base64="8dbP5OCTNgqnG6BmWwoyUzk1/E8=">AAAC1HicjVG7TsMwFD0Nr1IeLSCxsERUSExVwgJjVRbGVqIPqa2qJHWLRV5KHERVmBArP8AKI3/BPyAWZvgLrt1UAioEjmIfn3vOta+vHbo8FobxmtHm5hcWl7LLuZXVtfV8YWOzEQdJ5LC6E7hB1LKtmLncZ3XBhctaYcQsz3ZZ0z4/lvHmBYtiHvinYhSyrmcNfT7gjiWI6hXyHTtw+/HIo0W/7BFTNEqGGvosMFNQLG/X3vhT5bkaFF7QQR8BHCTwwOBDEHZhIaavDRMGQuK6GBMXEeIqznCNHHkTUjFSWMSe0zykXTtlfdrLnLFyO3SKS39ETh175AlIFxGWp+kqnqjMkv0t91jllHcb0WqnuTxiBc6I/cs3Vf7XJ2sRGOBI1cCpplAxsjonzZKoV5E3179UJShDSJzEfYpHhB3lnL6zrjyxql2+raXi70opWbl3Um2CD3lLarD5s52zoHFQMo2SWaNOVzAZWexgF/vUz0OUcYIq6qrn93jAo9bQrrQb7XYi1TKpZwvfhnb3CUcymVc=</latexit><latexit sha1_base64="8dbP5OCTNgqnG6BmWwoyUzk1/E8=">AAAC1HicjVG7TsMwFD0Nr1IeLSCxsERUSExVwgJjVRbGVqIPqa2qJHWLRV5KHERVmBArP8AKI3/BPyAWZvgLrt1UAioEjmIfn3vOta+vHbo8FobxmtHm5hcWl7LLuZXVtfV8YWOzEQdJ5LC6E7hB1LKtmLncZ3XBhctaYcQsz3ZZ0z4/lvHmBYtiHvinYhSyrmcNfT7gjiWI6hXyHTtw+/HIo0W/7BFTNEqGGvosMFNQLG/X3vhT5bkaFF7QQR8BHCTwwOBDEHZhIaavDRMGQuK6GBMXEeIqznCNHHkTUjFSWMSe0zykXTtlfdrLnLFyO3SKS39ETh175AlIFxGWp+kqnqjMkv0t91jllHcb0WqnuTxiBc6I/cs3Vf7XJ2sRGOBI1cCpplAxsjonzZKoV5E3179UJShDSJzEfYpHhB3lnL6zrjyxql2+raXi70opWbl3Um2CD3lLarD5s52zoHFQMo2SWaNOVzAZWexgF/vUz0OUcYIq6qrn93jAo9bQrrQb7XYi1TKpZwvfhnb3CUcymVc=</latexit><latexit sha1_base64="zfhq+mCqwMFLj1lhBRemBHR1dVs=">AAAC1HicjVHLTsMwEJyGVymvAkcuERUSpyrhAkcEF45Fog+praokdUtU56HYQVSFE+LKD3CFb0L8AfwFa+NKQIXAUezx7Mza6/VTHgrpOK8Fa25+YXGpuFxaWV1b3yhvbjVEkmcBqwcJT7KW7wnGw5jVZSg5a6UZ8yKfs6Y/OlXx5hXLRJjEF3Kcsm7kDeNwEAaeJKpX3uj4Ce+LcUSLfd0jpuJUHT3sWeAaUIEZtaT8gg76SBAgRwSGGJIwhwdBXxsuHKTEdTEhLiMU6jjDLUrkzUnFSOERO6J5SLu2YWPaq5xCuwM6hdOfkdPGHnkS0mWE1Wm2juc6s2J/yz3ROdXdxrT6JldErMQlsX/5psr/+lQtEgMc6RpCqinVjKouMFly/Srq5vaXqiRlSIlTuE/xjHCgndN3trVH6NrV23o6/qaVilX7wGhzvKtbUoPdn+2cBY2DqutU3XOncnxiWl3EDnaxT/08xDHOUENd9/wRT3i2GtaNdWfdf0qtgvFs49uwHj4ASMGVtg==</latexit>

n

map the positive tag embeddings to hypersphere T e n normalization

! S n

<latexit sha1_base64="ASODTBeRsVYFyUgTw7tt8jhT32c=">AAADFXicjVFBT9RAGH0URUDFBRMvXhqIiadN6wWPRC4cl8gCCSXLdHbYnTDtNNOpCE1/B/+EGzfi1aMxMd4k8C/8ZugmAjE6TTtv3vfe63wzaaFkaaPo+1Qw/ejxzJPZufmnz54vvOgsLm2XujJc9LlW2uymrBRK5qJvpVVitzCCZakSO+nRuqvvfBKmlDrfsieF2M/YKJeHkjNL1KDTSzJmx5ypOjmWQ2GlGop6q2kGefLZyNHYMmP0ce1VJquTMmNKhbk2NMtTH9I0k4zw44AiV6Ju5Ef4EMQtWFl7tfnzAEBPd74hwRAaHBUyCOSwhBUYSnr2ECNCQdw+auIMIenrAg3myVuRSpCCEXtE3xGt9lo2p7XLLL2b018UvYacId6QR5POEHZ/C3298smO/Vt27TPd3k5oTtusjFiLMbH/8k2U/+tzvVgc4r3vQVJPhWdcd7xNqfypuJ2Hf3RlKaEgzuEh1Q1h7p2Tcw69p/S9u7Nlvn7tlY51a95qK9y4XdIFx/ev8yHYfteNo268STf9AbdjFq+xjLd0n6tYwwZ66FP2OX7gF66Cs+AiuAy+3EqDqdbzEndG8PU3FG61Dw==</latexit><latexit sha1_base64="YU6H0w7B74Sj1GjGgbyTJscrOYM=">AAADFXicjVFPT9RAHH0U+SMiLph48dJITDxtWi943ODF4xJZIKFkM50ddidMO810KkLTz8Gn8OqNG+Hq0ZAQbxr5Fvxm6CYqMTpNO2/e773X+c2khZKljaKrmWD2wdz8wuLDpUfLj1eedFbXdkpdGS4GXCtt9lJWCiVzMbDSKrFXGMGyVInd9Oitq+9+EKaUOt+2J4U4yNg4l4eSM0vUsNNPMmYnnKk6OZYjYaUaiXq7aYZ58tHI8cQyY/Rx7VUmq5MyY0qFuTY0y1Mf0jTTjPD9kCLXo27kR3gfxC1Y7z3b+iY/bV70decrEoygwVEhg0AOS1iBoaRnHzEiFMQdoCbOEJK+LtBgibwVqQQpGLFH9B3Tar9lc1q7zNK7Of1F0WvIGeIleTTpDGH3t9DXK5/s2L9l1z7T7e2E5rTNyoi1mBD7L99U+b8+14vFId74HiT1VHjGdcfblMqfitt5+EtXlhIK4hweUd0Q5t45PefQe0rfuztb5us/vdKxbs1bbYUbt0u64PjP67wPdl5346gbb9FNb+JuLOI5XuAV3ecGeniHPgaU/RnX+I4fwVlwHlwEl3fSYKb1PMVvI/hyC+NOtss=</latexit><latexit sha1_base64="YU6H0w7B74Sj1GjGgbyTJscrOYM=">AAADFXicjVFPT9RAHH0U+SMiLph48dJITDxtWi943ODF4xJZIKFkM50ddidMO810KkLTz8Gn8OqNG+Hq0ZAQbxr5Fvxm6CYqMTpNO2/e773X+c2khZKljaKrmWD2wdz8wuLDpUfLj1eedFbXdkpdGS4GXCtt9lJWCiVzMbDSKrFXGMGyVInd9Oitq+9+EKaUOt+2J4U4yNg4l4eSM0vUsNNPMmYnnKk6OZYjYaUaiXq7aYZ58tHI8cQyY/Rx7VUmq5MyY0qFuTY0y1Mf0jTTjPD9kCLXo27kR3gfxC1Y7z3b+iY/bV70decrEoygwVEhg0AOS1iBoaRnHzEiFMQdoCbOEJK+LtBgibwVqQQpGLFH9B3Tar9lc1q7zNK7Of1F0WvIGeIleTTpDGH3t9DXK5/s2L9l1z7T7e2E5rTNyoi1mBD7L99U+b8+14vFId74HiT1VHjGdcfblMqfitt5+EtXlhIK4hweUd0Q5t45PefQe0rfuztb5us/vdKxbs1bbYUbt0u64PjP67wPdl5346gbb9FNb+JuLOI5XuAV3ecGeniHPgaU/RnX+I4fwVlwHlwEl3fSYKb1PMVvI/hyC+NOtss=</latexit><latexit sha1_base64="N7k1U40fmwDtLnz0r/vdwHY7sIY=">AAADFXicjVHBTtwwFBzSllJaYCnHXqKuKvW0SrjAEdELx63KAhJBK8drdi2cOHIcKI3yHfwJt96qXntESFxB8Bc8m6zUFlXUUeLxvJmJn50WSpY2ii5ngmfPX8y+nHs1//rNwuJSZ/ntTqkrw8WAa6XNXspKoWQuBlZaJfYKI1iWKrGbHn1y9d1jYUqp8217WoiDjI1zeSg5s0QNO/0kY3bCmaqTEzkSVqqRqLebZpgnX40cTywzRp/UXmWyOikzplSYa0Oz/OZDmmaaEX4ZUmQ36kV+hI9B3IIu2tHXnQskGEGDo0IGgRyWsAJDSc8+YkQoiDtATZwhJH1doME8eStSCVIwYo/oO6bVfsvmtHaZpXdz+oui15AzxAfyaNIZwu5voa9XPtmx/8qufabb2ynNaZuVEWsxIfYp31T5vz7Xi8Uh1n0PknoqPOO6421K5U/F7Tz8rStLCQVxDo+obghz75yec+g9pe/dnS3z9VuvdKxb81Zb4c7tki44/vs6H4Od1V4c9eLPUXdjs73qObzDe3yk+1zDBrbQx4Cyz3GFa9wEZ8H34Efw80EazLSeFfwxgl/35N2zKg==</latexit>

Part 3 merge synonyms

semantic enhancement

⌧

<latexit sha1_base64="XQnkbPwwAj9Cq6T9aSsC/0gCHl4=">AAACx3icjVHLSsNAFD3GV62vqks3wSK4Kokb3YhFN7qrYB/QFkmm0zY0yYRkUixF0B9wq3/kJ4h/oH/hnWkKahGdkOTMueecmTvjRr6XSMt6mzPmFxaXlnMr+dW19Y3NwtZ2LRFpzHiVCV/EDddJuO+FvCo96fNGFHMncH1edwfnql4f8jjxRHgtRxFvB04v9Loec6SiWtJJbwpFq2TpYc4COwPF05f8yT2Aiii8ooUOBBhSBOAIIQn7cJDQ04QNCxFxbYyJiwl5us5xhzx5U1JxUjjEDujbo1kzY0Oaq8xEuxmt4tMbk9PEPnkE6WLCajVT11OdrNjfssc6U+1tRH83ywqIlegT+5dvqvyvT/Ui0cWx7sGjniLNqO5YlpLqU1E7N790JSkhIk7hDtVjwkw7p+dsak+ie1dn6+j6u1YqVs1Zpk3xoXZJF2z/vM5ZUDss2VbJvrKK5TNMRg672MMB3ecRyrhABVXK7uMRT3g2Lg1hDI3bidSYyzw7+DaMh0+mbpKS</latexit><latexit sha1_base64="xQbLKebmAYwoR9veiSthkXpwTIk=">AAACx3icjVHLSsNAFD2NrxpfVZdugkVwVRI3uhGLbnRXwT6gLZKk0zY0LyaTYiku/AG3uvdj/ITiH+hfeGeaglpEJyQ5c+45Z+bOOLHvJcI033LawuLS8kp+VV9b39jcKmzv1JIo5S6rupEf8YZjJ8z3QlYVnvBZI+bMDhyf1Z3BhazXh4wnXhTeiFHM2oHdC72u59pCUi1hp7eFolky1TDmgZWB4tmrfhq/TPRKVJighQ4iuEgRgCGEIOzDRkJPExZMxMS1MSaOE/JUneEeOnlTUjFS2MQO6NujWTNjQ5rLzES5XVrFp5eT08ABeSLSccJyNUPVU5Us2d+yxypT7m1EfyfLCogV6BP7l2+m/K9P9iLQxYnqwaOeYsXI7twsJVWnIndufOlKUEJMnMQdqnPCrnLOztlQnkT1Ls/WVvV3pZSsnLuZNsWH3CVdsPXzOudB7ahkmSXr2iyWzzEdeexhH4d0n8co4xIVVCm7j0c84Vm70iJtqN1NpVou8+zi29AePgHXvJQG</latexit><latexit sha1_base64="xQbLKebmAYwoR9veiSthkXpwTIk=">AAACx3icjVHLSsNAFD2NrxpfVZdugkVwVRI3uhGLbnRXwT6gLZKk0zY0LyaTYiku/AG3uvdj/ITiH+hfeGeaglpEJyQ5c+45Z+bOOLHvJcI033LawuLS8kp+VV9b39jcKmzv1JIo5S6rupEf8YZjJ8z3QlYVnvBZI+bMDhyf1Z3BhazXh4wnXhTeiFHM2oHdC72u59pCUi1hp7eFolky1TDmgZWB4tmrfhq/TPRKVJighQ4iuEgRgCGEIOzDRkJPExZMxMS1MSaOE/JUneEeOnlTUjFS2MQO6NujWTNjQ5rLzES5XVrFp5eT08ABeSLSccJyNUPVU5Us2d+yxypT7m1EfyfLCogV6BP7l2+m/K9P9iLQxYnqwaOeYsXI7twsJVWnIndufOlKUEJMnMQdqnPCrnLOztlQnkT1Ls/WVvV3pZSsnLuZNsWH3CVdsPXzOudB7ahkmSXr2iyWzzEdeexhH4d0n8co4xIVVCm7j0c84Vm70iJtqN1NpVou8+zi29AePgHXvJQG</latexit><latexit sha1_base64="97OiP1YNEhHop4BQw7gLhKcybAI=">AAACx3icjVHLSsNAFD2Nr1pfVZdugkVwVRI3uiy60V0F+4C2yGQ6bYfmRTIpluLCH3Crfyb+gf6Fd8YU1CI6IcmZc+85M/deL/ZlqhzntWAtLa+srhXXSxubW9s75d29ZhplCRcNHvlR0vZYKnwZioaSyhftOBEs8HzR8sYXOt6aiCSVUXijprHoBWwYyoHkTGmqq1h2W644VccsexG4OaggX/Wo/IIu+ojAkSGAQAhF2AdDSk8HLhzExPUwIy4hJE1c4B4l0maUJSiDETum75B2nZwNaa89U6PmdIpPb0JKG0ekiSgvIaxPs008M86a/c17Zjz13ab093KvgFiFEbF/6eaZ/9XpWhQGODM1SKopNoyujucumemKvrn9pSpFDjFxGvcpnhDmRjnvs200qald95aZ+JvJ1Kze8zw3w7u+JQ3Y/TnORdA8qbpO1b12KrXzfNRFHOAQxzTPU9RwiToa5D3CI57wbF1ZkTWx7j5TrUKu2ce3ZT18AKrskMU=</latexit>

T e

<latexit sha1_base64="2nws7hYNxHN2GhAznUVwPF4TRkg=">AAAC23icjVHLSsQwFD3W97squHFTHARXQyuCLgfduBxhHoIzSJqJGkwftKkyjK7ciVt/wK3+j/gH+hfexAo+EE1pe3LuPSe594apkrn2/echZ3hkdGx8YnJqemZ2bt5dWGzlSZFx0eSJSrKDkOVCyVg0tdRKHKSZYFGoRDs82zXx9rnIcpnEDd1PRTdiJ7E8lpxpoo7c5c6F7AktVU8MOhHTp5wpr3F15Fb8qm+X9xMEJaigXPXEfUIHPSTgKBBBIIYmrMCQ03OIAD5S4roYEJcRkjYucIUp0haUJSiDEXtG3xPaHZZsTHvjmVs1p1MUvRkpPayRJqG8jLA5zbPxwjob9jfvgfU0d+vTPyy9ImI1Ton9S/eR+V+dqUXjGNu2Bkk1pZYx1fHSpbBdMTf3PlWlySElzuAexTPC3Co/+uxZTW5rN71lNv5iMw1r9rzMLfBqbkkDDr6P8ydobVQDvxrsb1ZqO+WoJ7CCVazTPLdQwx7qaJL3Je7xgEen61w7N87te6ozVGqW8GU5d28SnpjB</latexit><latexit sha1_base64="2nws7hYNxHN2GhAznUVwPF4TRkg=">AAAC23icjVHLSsQwFD3W97squHFTHARXQyuCLgfduBxhHoIzSJqJGkwftKkyjK7ciVt/wK3+j/gH+hfexAo+EE1pe3LuPSe594apkrn2/echZ3hkdGx8YnJqemZ2bt5dWGzlSZFx0eSJSrKDkOVCyVg0tdRKHKSZYFGoRDs82zXx9rnIcpnEDd1PRTdiJ7E8lpxpoo7c5c6F7AktVU8MOhHTp5wpr3F15Fb8qm+X9xMEJaigXPXEfUIHPSTgKBBBIIYmrMCQ03OIAD5S4roYEJcRkjYucIUp0haUJSiDEXtG3xPaHZZsTHvjmVs1p1MUvRkpPayRJqG8jLA5zbPxwjob9jfvgfU0d+vTPyy9ImI1Ton9S/eR+V+dqUXjGNu2Bkk1pZYx1fHSpbBdMTf3PlWlySElzuAexTPC3Co/+uxZTW5rN71lNv5iMw1r9rzMLfBqbkkDDr6P8ydobVQDvxrsb1ZqO+WoJ7CCVazTPLdQwx7qaJL3Je7xgEen61w7N87te6ozVGqW8GU5d28SnpjB</latexit><latexit sha1_base64="2nws7hYNxHN2GhAznUVwPF4TRkg=">AAAC23icjVHLSsQwFD3W97squHFTHARXQyuCLgfduBxhHoIzSJqJGkwftKkyjK7ciVt/wK3+j/gH+hfexAo+EE1pe3LuPSe594apkrn2/echZ3hkdGx8YnJqemZ2bt5dWGzlSZFx0eSJSrKDkOVCyVg0tdRKHKSZYFGoRDs82zXx9rnIcpnEDd1PRTdiJ7E8lpxpoo7c5c6F7AktVU8MOhHTp5wpr3F15Fb8qm+X9xMEJaigXPXEfUIHPSTgKBBBIIYmrMCQ03OIAD5S4roYEJcRkjYucIUp0haUJSiDEXtG3xPaHZZsTHvjmVs1p1MUvRkpPayRJqG8jLA5zbPxwjob9jfvgfU0d+vTPyy9ImI1Ton9S/eR+V+dqUXjGNu2Bkk1pZYx1fHSpbBdMTf3PlWlySElzuAexTPC3Co/+uxZTW5rN71lNv5iMw1r9rzMLfBqbkkDDr6P8ydobVQDvxrsb1ZqO+WoJ7CCVazTPLdQwx7qaJL3Je7xgEen61w7N87te6ozVGqW8GU5d28SnpjB</latexit><latexit sha1_base64="2nws7hYNxHN2GhAznUVwPF4TRkg=">AAAC23icjVHLSsQwFD3W97squHFTHARXQyuCLgfduBxhHoIzSJqJGkwftKkyjK7ciVt/wK3+j/gH+hfexAo+EE1pe3LuPSe594apkrn2/echZ3hkdGx8YnJqemZ2bt5dWGzlSZFx0eSJSrKDkOVCyVg0tdRKHKSZYFGoRDs82zXx9rnIcpnEDd1PRTdiJ7E8lpxpoo7c5c6F7AktVU8MOhHTp5wpr3F15Fb8qm+X9xMEJaigXPXEfUIHPSTgKBBBIIYmrMCQ03OIAD5S4roYEJcRkjYucIUp0haUJSiDEXtG3xPaHZZsTHvjmVs1p1MUvRkpPayRJqG8jLA5zbPxwjob9jfvgfU0d+vTPyy9ImI1Ton9S/eR+V+dqUXjGNu2Bkk1pZYx1fHSpbBdMTf3PlWlySElzuAexTPC3Co/+uxZTW5rN71lNv5iMw1r9rzMLfBqbkkDDr6P8ydobVQDvxrsb1ZqO+WoJ7CCVazTPLdQwx7qaJL3Je7xgEen61w7N87te6ozVGqW8GU5d28SnpjB</latexit>

Figure 2: The proposed Weakly Supervised Deep Hyperspherical Quantization (WSDHQ) consists of five main parts: 1) a standard CNN, 2) a word embedding model, 3) a correlation graph, 4) a transform layer and 5) a semantic hypersphere.

lead to unacceptably high computational cost as |S| is usu- ally huge in real world. In practice, we only involve K n most tricky negative semantic embeddings as our negative set, i.e.,

N ⁿ = arg max

s ⁻ _i ∈S\S n , |N n |=K n

cos θ _(s − i ,r n ) .

To keep the image representations consistent with precise semantic information, we adopt an adaptive margin strategy

∆ _ij that depends on the discrepancy between positive and negative semantics. For example, a smaller margin ∆ (cat, dog)

is acceptable while ∆ (kiss, dog) should adapt to be larger, since the semantic representation of negative semantics “dog” is closer to that of positive semantics “cat” than positive se- mantics “kiss”. Specifically, we define

∆ _ij = 2 ^1−γ ·

1 − cos θ (s ⁺ _i ,s ⁻ _j )

γ

,

where γ is a hyper-parameter such that smaller γ leads to larger adaptive margin under same semantic similarity.

Supervised Cosine Quantization. We propose a super- vised quantizer on hypersphere to enable efficient image re- trieval. Specifically, each image embedding r n will be quan- tized with a set of M codebooks C = [C 1 , C 2 , · · · , C ^M ], each C m = {c ^m1 , c m2 , · · · , c ^mK } contains K codewords, and each codeword c mk is a D-dimensional centroid vec- tor obtained by k-means. The codeword assignment vec- tor b n is segmented into M 1-of-K indicator vectors b n = [b 1n ; b 2n ; · · · ; b ^{M n} ] w.r.t. M codebooks, and each one-hot indicator vector b mn indicates which one of K codewords in the m-th codebook is used to compose the approximation for r n . The proposed quantizer encodes each image embed- ding r n as the sum of M codewords, each of which comes from its codebook C m assigned by an indicator vector b n , i.e., r n ≈ ˆ r n ≡ P M

m=1 C m b mn . Then we integrate seman- tic supervision into quantization learning, with the cosine quantization loss formulated as

Q ⁿ = X

s i ∈S

cos θ (s i ,r n ) − cos θ (s i , ˆ r n )

2 .

Approximate Nearest Neighbor Search. Approximate nearest neighbor (ANN) search by maximum inner-product

similarity (MIPS) is a powerful tool for quantization meth- ods (Cao et al. 2017; Liu et al. 2018). Note that on the unit hypersphere, the cosine similarity between two points can be equivalently transformed into inner-product. Hence, with the image database of N quantized binary codes {b ⁿ } ^N n=1 , we adopt Asymmetric Quantizer Distance (AQD) (Jegou, Douze, and Schmid 2010) as the metric, which computes the cosine of the angle on hypersphere between a given query q and the reconstruction of a database point x n as

AQD(q, x n ) = cos θ (r q , ˆ r n ) = r ^> _q X M m=1

C m b mn

! ,

where r q is the hyperspherical embedding w.r.t. query q. We set up a query-specific lookup table of M × K items for q, which stores the pre-computed results of inner-product between q and all codewords in C. Hence, the AQD can be efficiently computed by summing chosen items from lookup table according to the quantization code b n .

Learning Algorithm

Overall Objective. WSDHQ enables efficient image re- trieval in an end-to-end architecture, which jointly learns semantics-preserving hyperspherical embedding and super- vised quantization in a total objective as

W,C,B min X N n=1

( L ⁿ + λ Q ⁿ ) , (1) where λ is a positive hyper-parameter to balance the adap- tive cosine margin loss L and the semantically supervised quantization loss Q, and W denotes the learnable network parameters. Through optimizing objective (1), WSDHQ pre- serves the semantic information of tags into hyperspherical embeddings while effectively reducing quantization error.

There are three sets of variables in objective (1): 1)

the network parameters W, 2) N binary codes B =

[b 1 , b 2 , · · · , b ^N ] = [B 1 ; B 2 ; · · · ; B ^M ], where B m =

[b m1 , b m2 , · · · , b ^mN ] is the subcode matrix of all points

w.r.t. the m-th codebook C m and 3) M codebooks C =

(5)

[C 1 , C 2 , · · · , C ^M ]. We adopt an commonly used alternating optimization paradigm (Liu et al. 2018), which iteratively optimizes one variable set while fixing the others.

Learning W. Many back-propagation algorithms can be adopted to optimize the network parameters W. With the ad- vance of automatic differentiation techniques in mainstream machine learning libraries (Abadi et al. 2016; Paszke et al.

2019), it is easy to optimize W within a few code lines.

Learning B. We update N quantization codes B by fixing W and C as known variables. Since each b ⁿ is independent with {b ⁿ ⁰ } ⁿ ⁰ 6=n , the optimization of B can be split into N subproblems, e.g., we optimize b n as

min b n

r n − X M m=1

C m b mn

! >

Σ _S r n − X M m=1

C m b mn

! , (2) s.t. kb ^mn k ⁰ = 1, b mn ∈ {0, 1} ^K ,

where Σ _S = P

s i ∈S s i s i > is the covariance matrix of se- mantic embeddings, which reflects the latent distribution of queries as all visual features will be eventually embedded to the hypersphere spanned by these embeddings. Objective (2) is generally an NP-hard problem of discrete combinato- rial optimization, while some stochastic local search algo- rithms can provide a promising solution. We take the widely used Iterated Conditional Modes (ICM) algorithm (Besag 1986; Zhang, Du, and Wang 2014) to solve it.Given fixed {b ^m ⁰ ⁿ } ^m ⁰ 6=m , we update b mn by exhaustively checking all codewords in C m and finding the codeword such that objec- tive (2) is minimized. In each encoding iteration, the M in- dicators {b ^mn } ^M m=1 are calculated alternatively in this way.

Moreover, we inject some stochastic relaxations (Zeger et al.

1992; Martinez et al. 2018) into ICM to avoid local optimum so as to improve the performance of quantization. Specif- ically, for the i-th encoding iteration, we add an iteration- decaying perturbation π(i) = (T (i)/M ) · to codebooks C and get perturbed codebooks ˜ C = C + π(i), where T (i) = p

1 − (i/I) is the temperature scheduled for the i-th iteration among all I encoding iterations, ∼ N (0, Σ) and Σ = diag(cov(R)) is the diagonal covariance proportional to R. Finally, we replace the codebook C m in objective (2) with ˜ C m before learning {b ^mn } ^N n=1 .

Learning C. We update M codebooks C by fixing W and B as known variables, and rewrite the objective (1) as

min C tr (R − CB) ^> Σ _S (R − CB)

, (3)

where R = [r 1 , r 2 , · · · , r ^N ] is the image embedding ma- trix. The optimal solution of objective (3) comes in a closed form as C = RB ^> ( BB ^> ) ⁻¹ . Note that the binary ma- trix B satisfies structurally regular conditions: 1) Diagonal B _i ^> B i can be computed as histograms of the codes in B i , and B ^> _i B j = ( B ^> _j B i ) ^> can be computed as bivariate his- tograms of the codes in B i and B j . 2) RB i can be com- puted by treating the columns of B i as binary vectors that se- lect the columns of R to sum together. With the help of these properties, the optimal solution of C can be solved with time complexity O(MND) and O(M ² N ), compared with

their na¨ıve computations of O(MKND) and O(M ² K ² N ) (Martinez et al. 2018).

Experiments

In this section, we conduct extensive experiments to evaluate our proposed WSDHQ model with several state-of-art shal- low and deep hashing methods on two web image datasets.

Setup

To our best knowledge, there are only two large-scale and commonly-used web image datasets (MIR-FLICKR25K, NUS-WIDE) that contain image-tag-label triplets. (E.g., ImageNet does not contain tag information.) Thus we con- duct our empirical evaluations on them.

MIR-FLICKR25K (Huiskes and Lew 2008) is a dataset of 25,000 Flickr images associated with 1,386 tags. The au- thors labelled the images with 38 semantic concepts in total, which are only used for evaluation in our experiments. 2,000 images are randomly sampled as test queries and the rest are used as retrieval database and training images.

NUS-WIDE (Chua et al. 2009) is a large-scale web image dataset also collected from Flickr, which contains 269,648 images with 5,018 tags provided by users. The authors have manually annotated each image with a pre-defined set of 81 ground-truth labels, which are only used for evaluation. We collect a subset of 193,752 images with the 21 most frequent labels for experiments. We follow (Cao et al. 2017; Liu et al.

2018) to randomly sample 5,000 images as queries and re- main the rest as the database, from which we further sample 10,000 images and their tag sets as training data.

Following standard evaluation protocols adopted in previ- ous works (Cao et al. 2017; Liu et al. 2018; Gattupalli, Zhuo, and Li 2019), we use three evaluation metrics: Mean Av- erage Precision (MAP), Precision-Recall curves (PR), and Precision curves w.r.t. the number of top returned results (P@N). To make fair comparisons, all methods use identical training and test sets, and we follow previous works to adopt MAP@5000 for both datasets. Given a query, the ground truth is defined as: if a result shares at least one common label with the query, it is relevant; otherwise it is irrelevant.

We compare the retrieval performance of the proposed WSDHQ model with several state-of-art hashing meth- ods, including: 1) Five shallow unsupervised methods, LSH (Gionis et al. 1999), SH (Weiss, Torralba, and Fer- gus 2009), SpH (Lee 2012), ITQ (Gong et al. 2013) and AQ (Babenko and Lempitsky 2014). 2) One deep unsuper- vised method, DeepBit (Lin et al. 2016). 3) Two shallow weakly-supervised methods, WMH (Tang and Li 2018) and WDH (Cui et al. 2020). 4) One deep weakly-supervised method, WDHT (Gattupalli, Zhuo, and Li 2019).

Weakly Supervised Deep Hyperspherical Quantization for Image Retrieval