Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing
2078
A Stack-Propagation Framework with Token-Level Intent Detection for
Spoken Language Understanding
Libo Qin, Wanxiang Che∗, Yangming Li, Haoyang Wen, Ting Liu
Research Center for Social Computing and Information Retrieval Harbin Institute of Technology, China
{lbqin,car,yangmingli,hywen,tliu}@ir.hit.edu.cn
Abstract
Intent detection and slot filling are two main tasks for building a spoken language
under-standing (SLU) system. The two tasks are
closely tied and the slots often highly
de-pend on the intent. In this paper, we
pro-pose a novel framework for SLU to better in-corporate the intent information, which
fur-ther guides the slot filling. In our
frame-work, we adopt a joint model with Stack-Propagation which can directly use the intent information as input for slot filling, thus to
capture the intent semantic knowledge. In
addition, to further alleviate the error propa-gation, we perform the token-level intent de-tection for the Stack-Propagation framework. Experiments on two publicly datasets show that our model achieves the state-of-the-art performance and outperforms other previous methods by a large margin. Finally, we use the Bidirectional Encoder Representation from Transformer (BERT) model in our framework, which further boost our performance in SLU task.
1 Introduction
Spoken language understanding (SLU) is a criti-cal component in task-oriented dialogue systems. It usually consists of intent detection to identify users’ intents and slot filling task to extract se-mantic constituents from the natural language
ut-terances (Tur and De Mori, 2011). As shown in
Table 1, given a movie-related utterance “watch
action movie”, there are different slot labels for each token and an intent for the whole utterance.
Usually, intent detection and slot filling are
implemented separately. But intuitively, these
two tasks are not independent and the slots of-ten highly depend on the inof-tent (Goo et al., 2018). For example, if the intent of a utterance is
∗
* Email corresponding.
Sentence watch action movie
Gold Slots O B-movie name I-movie name
Gold Intent WatchMovie
Table 1: An example with intent and slot annotation (BIO format), which indicates the slot of movie name
from an utterance with an intentWatchMovie.
WatchMovie, it is more likely to contain the slot
movie namerather than the slotmusic name.
Hence, it is promising to incorporate the intent in-formation to guide the slot filling.
Considering this strong correlation between the two tasks, some joint models are proposed based on the multi-task learning framework (Zhang and
Wang, 2016; Hakkani-T¨ur et al., 2016; Liu and
Lane,2016) and all these models outperform the
pipeline models via mutual enhancement between
two tasks. However, their work just modeled
the relationship between intent and slots by shar-ing parameters. Recently, some work begins to model the intent information for slot filling
ex-plicitly in joint model. Goo et al. (2018) and Li
et al.(2018) proposed the gate mechanism to
ex-plore incorporating the intent information for slot filling. Though achieving the promising perfor-mance, their models still suffer from two issues including: (1) They all adopt the gate vector to incorporate the intent information. In the paper, we argue that it is risky to simply rely on the gate function to summarize or memorize the intent in-formation. Besides, the interpretability of how the intent information guides slot filling procedure is still weak due to the interaction with hidden vector between the two tasks. (2) The utterance-level in-tent information they use for slot filling may mis-lead the prediction for all slots in an utterance if the predicted utterance-level intent is incorrect.
Encoder
Task B Task A
Encoder Task B
Task A differentiable link
[image:2.595.74.292.64.153.2](a) Multi-task framework (b) Stack-propagation
Figure 1: Multi-task framework vs. Stack-Propagation.
proposed by Zhang and Weiss(2016) to leverage
the POS tagging features for parsing and achieved good performance, we propose a joint model with Stack-Propagation for SLU tasks. Our framework directly use the output of the intent detection as the input for slot filling to better guide the slot
prediction process. In addition, the framework
make it easy to design oracle intent experiment to intuitively show how intent information enhances slot filling task. For the second issue, we perform a token-level intent prediction in our framework, which can provide the token-level intent informa-tion for slot filling. If some token-level intents in the utterance are predicted incorrectly, other cor-rect token-level intents will still be useful for the corresponding slot prediction. In practice, we use a self-attentive encoder for intent detection to cap-ture the contextual information at each token and hence predict an intent label at each token. The intent of an utterance is computed by voting from predictions at each token of the utterance. This token-level prediction, like ensemble neural
net-works (Lee et al., 2016), reduces the predicted
variance to improve the performance of intent de-tection. And it fits better in our Stack-Propagation framework, where intent detection can provide token-level intent features and retain more useful intent information for slot filling.
We conduct experiments on two benchmarks
SNIPS (Coucke et al.,2018) and ATIS (Goo et al.,
2018) datasets. The results of both experiments show the effectiveness of our framework by out-performing the current state-of-the-art methods by
a large margin. Finally, Bidirectional Encoder
Representation from Transformer (Devlin et al., 2018, BERT), as the pre-trained model, is used to further boost the performance of our model.
To summarize, the contributions of this work are as follows:
• We propose a Stack-Propagation framework
in SLU task, which can better incorporate the
intent semantic knowledge to guide the slot filling and make our joint model more inter-pretable.
• We perform the token-level intent detection
for Stack-Propagation framework, which im-proves the intent detection performance and further alleviate the error propagation.
• We present extensive experiments
demon-strating the benefit of our proposed frame-work. Our experiments on two publicly avail-able datasets show substantial improvement and our framework achieve the state-of-the-art performance.
• We explore and analyze the effect of
incorpo-rating BERT in SLU tasks.
For reproducibility, our code for this paper is
publicly available at https://github.com/
LeePleased/StackPropagation-SLU.
2 Background
In this section, we will describe the formulation definition for intent detection and slot filling, and then we give a brief description of the multi-task framework and the joint model with Stack-Propagation framework.
2.1 Intent Detection and Slot Filling
Intent detection can be seen as a classification
problem to decide the intent label oI of an
utter-ance. Slot filling is a sequence labeling task that
maps an input word sequencex=(x1, . . . , xT)to
slots sequenceoS =(oS
1, . . . , oST).
2.2 Multi-task Framework vs.
Stack-Propagation
For two correlative tasksTask AandTask B,
multi-task framework which is shown in Figure 1 (a)
can learn the correlations between these two tasks by the shared encoder. However, the basic multi-task framework cannot provide features from up-stream task to down-up-stream task explicitly. The joint model with Stack-Propagation framework
which is shown in Figure 1 (b) can mitigate the
shortcoming. In this figure, Task B can leverage
features ofTask A without breaking the
Token-level Intent Detection decoder !"#$%&' Self-Attentive Encoder Slot Filling decoder
watch action movie
ListenMusic WatchMovie WatchMovie
O B-movie_name I-movie_name
Slot Filling Input layer
e1
<latexit sha1_base64="UOJMj9uT+rMuVRiicAlHNm5A4CY=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmLZQS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO400ziTjHovDWLYCP+WhiLinhAp5K5HcHwUhbwbDcx1v3nGZiji6VuOEd0b+IBJ9wXxFlMe7E3faLVecqmOWPQ/cHFSQr3pcfsENeojBkGEEjgiKcAgfKT1tuHCQENfBhDhJSJg4xxQl0maUxSnDJ3ZI3wHt2jkb0V57pkbN6JSQXklKGwekiSlPEtan2SaeGWfN/uY9MZ76bmP6B7nXiFiFW2L/0s0y/6vTtSj0cWpqEFRTYhhdHctdMtMVfXP7S1WKHBLiNO5RXBJmRjnrs200qald99Y38TeTqVm9Z3luhnd9Sxqw+3Oc86BxVHWdqnt1XKmd5aMuYg/7OKR5nqCGC9ThkbfAI57wbF1aiXVvjT9TrUKu2cW3ZT18AHwckRo=</latexit><latexit sha1_base64="UOJMj9uT+rMuVRiicAlHNm5A4CY=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmLZQS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO400ziTjHovDWLYCP+WhiLinhAp5K5HcHwUhbwbDcx1v3nGZiji6VuOEd0b+IBJ9wXxFlMe7E3faLVecqmOWPQ/cHFSQr3pcfsENeojBkGEEjgiKcAgfKT1tuHCQENfBhDhJSJg4xxQl0maUxSnDJ3ZI3wHt2jkb0V57pkbN6JSQXklKGwekiSlPEtan2SaeGWfN/uY9MZ76bmP6B7nXiFiFW2L/0s0y/6vTtSj0cWpqEFRTYhhdHctdMtMVfXP7S1WKHBLiNO5RXBJmRjnrs200qald99Y38TeTqVm9Z3luhnd9Sxqw+3Oc86BxVHWdqnt1XKmd5aMuYg/7OKR5nqCGC9ThkbfAI57wbF1aiXVvjT9TrUKu2cW3ZT18AHwckRo=</latexit><latexit sha1_base64="UOJMj9uT+rMuVRiicAlHNm5A4CY=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmLZQS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO400ziTjHovDWLYCP+WhiLinhAp5K5HcHwUhbwbDcx1v3nGZiji6VuOEd0b+IBJ9wXxFlMe7E3faLVecqmOWPQ/cHFSQr3pcfsENeojBkGEEjgiKcAgfKT1tuHCQENfBhDhJSJg4xxQl0maUxSnDJ3ZI3wHt2jkb0V57pkbN6JSQXklKGwekiSlPEtan2SaeGWfN/uY9MZ76bmP6B7nXiFiFW2L/0s0y/6vTtSj0cWpqEFRTYhhdHctdMtMVfXP7S1WKHBLiNO5RXBJmRjnrs200qald99Y38TeTqVm9Z3luhnd9Sxqw+3Oc86BxVHWdqnt1XKmd5aMuYg/7OKR5nqCGC9ThkbfAI57wbF1aiXVvjT9TrUKu2cW3ZT18AHwckRo=</latexit><latexit sha1_base64="UOJMj9uT+rMuVRiicAlHNm5A4CY=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmLZQS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO400ziTjHovDWLYCP+WhiLinhAp5K5HcHwUhbwbDcx1v3nGZiji6VuOEd0b+IBJ9wXxFlMe7E3faLVecqmOWPQ/cHFSQr3pcfsENeojBkGEEjgiKcAgfKT1tuHCQENfBhDhJSJg4xxQl0maUxSnDJ3ZI3wHt2jkb0V57pkbN6JSQXklKGwekiSlPEtan2SaeGWfN/uY9MZ76bmP6B7nXiFiFW2L/0s0y/6vTtSj0cWpqEFRTYhhdHctdMtMVfXP7S1WKHBLiNO5RXBJmRjnrs200qald99Y38TeTqVm9Z3luhnd9Sxqw+3Oc86BxVHWdqnt1XKmd5aMuYg/7OKR5nqCGC9ThkbfAI57wbF1aiXVvjT9TrUKu2cW3ZT18AHwckRo=</latexit> e2<latexit sha1_base64="F3mSGmP2ncMwlvmE/DRl8ixSd+o=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRlxVMK1QS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO800ziTjHovDWF4HfspDEXFPCRXy60RyfxSEvBUMz3S8dcdlKuLoSo0T3hn5g0j0BfMVUR7vTmrTbrniVB2z7Hng5qCCfDXi8gtu0EMMhgwjcERQhEP4SOlpw4WDhLgOJsRJQsLEOaYokTajLE4ZPrFD+g5o187ZiPbaMzVqRqeE9EpS2jggTUx5krA+zTbxzDhr9jfvifHUdxvTP8i9RsQq3BL7l26W+V+drkWhjxNTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GP4pIwM8pZn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAfNWtV1qu7lUaV+mo+6iD3s45DmeYw6ztGAR94Cj3jCs3VhJda9Nf5MtQq5ZhfflvXwAX59kRs=</latexit><latexit sha1_base64="F3mSGmP2ncMwlvmE/DRl8ixSd+o=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRlxVMK1QS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO800ziTjHovDWF4HfspDEXFPCRXy60RyfxSEvBUMz3S8dcdlKuLoSo0T3hn5g0j0BfMVUR7vTmrTbrniVB2z7Hng5qCCfDXi8gtu0EMMhgwjcERQhEP4SOlpw4WDhLgOJsRJQsLEOaYokTajLE4ZPrFD+g5o187ZiPbaMzVqRqeE9EpS2jggTUx5krA+zTbxzDhr9jfvifHUdxvTP8i9RsQq3BL7l26W+V+drkWhjxNTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GP4pIwM8pZn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAfNWtV1qu7lUaV+mo+6iD3s45DmeYw6ztGAR94Cj3jCs3VhJda9Nf5MtQq5ZhfflvXwAX59kRs=</latexit><latexit sha1_base64="F3mSGmP2ncMwlvmE/DRl8ixSd+o=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRlxVMK1QS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO800ziTjHovDWF4HfspDEXFPCRXy60RyfxSEvBUMz3S8dcdlKuLoSo0T3hn5g0j0BfMVUR7vTmrTbrniVB2z7Hng5qCCfDXi8gtu0EMMhgwjcERQhEP4SOlpw4WDhLgOJsRJQsLEOaYokTajLE4ZPrFD+g5o187ZiPbaMzVqRqeE9EpS2jggTUx5krA+zTbxzDhr9jfvifHUdxvTP8i9RsQq3BL7l26W+V+drkWhjxNTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GP4pIwM8pZn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAfNWtV1qu7lUaV+mo+6iD3s45DmeYw6ztGAR94Cj3jCs3VhJda9Nf5MtQq5ZhfflvXwAX59kRs=</latexit><latexit sha1_base64="F3mSGmP2ncMwlvmE/DRl8ixSd+o=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRlxVMK1QS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO800ziTjHovDWF4HfspDEXFPCRXy60RyfxSEvBUMz3S8dcdlKuLoSo0T3hn5g0j0BfMVUR7vTmrTbrniVB2z7Hng5qCCfDXi8gtu0EMMhgwjcERQhEP4SOlpw4WDhLgOJsRJQsLEOaYokTajLE4ZPrFD+g5o187ZiPbaMzVqRqeE9EpS2jggTUx5krA+zTbxzDhr9jfvifHUdxvTP8i9RsQq3BL7l26W+V+drkWhjxNTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GP4pIwM8pZn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAfNWtV1qu7lUaV+mo+6iD3s45DmeYw6ztGAR94Cj3jCs3VhJda9Nf5MtQq5ZhfflvXwAX59kRs=</latexit> e3<latexit sha1_base64="eb4TJOGLUZm66tydoux6u5LjkOo=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0I64qmLZQS0mm0zqYJiGZKKV04w+41S8T/0D/wjvjFNQiOiHJmXPvOTP33iAJRSYd57Vgzc0vLC4Vl0srq2vrG+XNrUYW5ynjHovDOG0FfsZDEXFPChnyVpJyfxiEvBncnql4846nmYijKzlKeGfoDyLRF8yXRHm8Oz6cdMsVp+roZc8C14AKzKrH5Rdco4cYDDmG4IggCYfwkdHThgsHCXEdjIlLCQkd55igRNqcsjhl+MTe0ndAu7ZhI9orz0yrGZ0S0puS0sYeaWLKSwmr02wdz7WzYn/zHmtPdbcR/QPjNSRW4obYv3TTzP/qVC0SfZzoGgTVlGhGVceMS667om5uf6lKkkNCnMI9iqeEmVZO+2xrTaZrV731dfxNZypW7ZnJzfGubkkDdn+OcxY0DqquU3Uvjyq1UzPqInawi32a5zFqOEcdHnkLPOIJz9aFlVj31ugz1SoYzTa+LevhA4DekRw=</latexit><latexit sha1_base64="eb4TJOGLUZm66tydoux6u5LjkOo=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0I64qmLZQS0mm0zqYJiGZKKV04w+41S8T/0D/wjvjFNQiOiHJmXPvOTP33iAJRSYd57Vgzc0vLC4Vl0srq2vrG+XNrUYW5ynjHovDOG0FfsZDEXFPChnyVpJyfxiEvBncnql4846nmYijKzlKeGfoDyLRF8yXRHm8Oz6cdMsVp+roZc8C14AKzKrH5Rdco4cYDDmG4IggCYfwkdHThgsHCXEdjIlLCQkd55igRNqcsjhl+MTe0ndAu7ZhI9orz0yrGZ0S0puS0sYeaWLKSwmr02wdz7WzYn/zHmtPdbcR/QPjNSRW4obYv3TTzP/qVC0SfZzoGgTVlGhGVceMS667om5uf6lKkkNCnMI9iqeEmVZO+2xrTaZrV731dfxNZypW7ZnJzfGubkkDdn+OcxY0DqquU3Uvjyq1UzPqInawi32a5zFqOEcdHnkLPOIJz9aFlVj31ugz1SoYzTa+LevhA4DekRw=</latexit><latexit sha1_base64="eb4TJOGLUZm66tydoux6u5LjkOo=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0I64qmLZQS0mm0zqYJiGZKKV04w+41S8T/0D/wjvjFNQiOiHJmXPvOTP33iAJRSYd57Vgzc0vLC4Vl0srq2vrG+XNrUYW5ynjHovDOG0FfsZDEXFPChnyVpJyfxiEvBncnql4846nmYijKzlKeGfoDyLRF8yXRHm8Oz6cdMsVp+roZc8C14AKzKrH5Rdco4cYDDmG4IggCYfwkdHThgsHCXEdjIlLCQkd55igRNqcsjhl+MTe0ndAu7ZhI9orz0yrGZ0S0puS0sYeaWLKSwmr02wdz7WzYn/zHmtPdbcR/QPjNSRW4obYv3TTzP/qVC0SfZzoGgTVlGhGVceMS667om5uf6lKkkNCnMI9iqeEmVZO+2xrTaZrV731dfxNZypW7ZnJzfGubkkDdn+OcxY0DqquU3Uvjyq1UzPqInawi32a5zFqOEcdHnkLPOIJz9aFlVj31ugz1SoYzTa+LevhA4DekRw=</latexit><latexit sha1_base64="eb4TJOGLUZm66tydoux6u5LjkOo=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0I64qmLZQS0mm0zqYJiGZKKV04w+41S8T/0D/wjvjFNQiOiHJmXPvOTP33iAJRSYd57Vgzc0vLC4Vl0srq2vrG+XNrUYW5ynjHovDOG0FfsZDEXFPChnyVpJyfxiEvBncnql4846nmYijKzlKeGfoDyLRF8yXRHm8Oz6cdMsVp+roZc8C14AKzKrH5Rdco4cYDDmG4IggCYfwkdHThgsHCXEdjIlLCQkd55igRNqcsjhl+MTe0ndAu7ZhI9orz0yrGZ0S0puS0sYeaWLKSwmr02wdz7WzYn/zHmtPdbcR/QPjNSRW4obYv3TTzP/qVC0SfZzoGgTVlGhGVceMS667om5uf6lKkkNCnMI9iqeEmVZO+2xrTaZrV731dfxNZypW7ZnJzfGubkkDdn+OcxY0DqquU3Uvjyq1UzPqInawi32a5zFqOEcdHnkLPOIJz9aFlVj31ugz1SoYzTa+LevhA4DekRw=</latexit>
hI
1
<latexit sha1_base64="DT+3KsZZtd7aVK4fLuT/2gohxms=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0oxupYB+itSTTaRuaF8lEKKFbf8Ctfpf4B/oX3hmnoBbRCUnOnHvPmbn3urHvpcKyXgvG3PzC4lJxubSyura+Ud7caqZRljDeYJEfJW3XSbnvhbwhPOHzdpxwJ3B93nJHpzLeuudJ6kXhlRjHvBM4g9Dre8wRRF0Pu7k9ucvPJ91yxapaapmzwNagAr3qUfkFt+ghAkOGABwhBGEfDlJ6bmDDQkxcBzlxCSFPxTkmKJE2oyxOGQ6xI/oOaHej2ZD20jNVakan+PQmpDSxR5qI8hLC8jRTxTPlLNnfvHPlKe82pr+rvQJiBYbE/qWbZv5XJ2sR6ONY1eBRTbFiZHVMu2SqK/Lm5peqBDnExEnco3hCmCnltM+m0qSqdtlbR8XfVKZk5Z7p3Azv8pY0YPvnOGdB86BqW1X78rBSO9GjLmIHu9ineR6hhjPU0SDvAI94wrNxYQgjNyafqUZBa7bxbRkPH/5pkuQ=</latexit><latexit sha1_base64="DT+3KsZZtd7aVK4fLuT/2gohxms=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0oxupYB+itSTTaRuaF8lEKKFbf8Ctfpf4B/oX3hmnoBbRCUnOnHvPmbn3urHvpcKyXgvG3PzC4lJxubSyura+Ud7caqZRljDeYJEfJW3XSbnvhbwhPOHzdpxwJ3B93nJHpzLeuudJ6kXhlRjHvBM4g9Dre8wRRF0Pu7k9ucvPJ91yxapaapmzwNagAr3qUfkFt+ghAkOGABwhBGEfDlJ6bmDDQkxcBzlxCSFPxTkmKJE2oyxOGQ6xI/oOaHej2ZD20jNVakan+PQmpDSxR5qI8hLC8jRTxTPlLNnfvHPlKe82pr+rvQJiBYbE/qWbZv5XJ2sR6ONY1eBRTbFiZHVMu2SqK/Lm5peqBDnExEnco3hCmCnltM+m0qSqdtlbR8XfVKZk5Z7p3Azv8pY0YPvnOGdB86BqW1X78rBSO9GjLmIHu9ineR6hhjPU0SDvAI94wrNxYQgjNyafqUZBa7bxbRkPH/5pkuQ=</latexit><latexit sha1_base64="DT+3KsZZtd7aVK4fLuT/2gohxms=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0oxupYB+itSTTaRuaF8lEKKFbf8Ctfpf4B/oX3hmnoBbRCUnOnHvPmbn3urHvpcKyXgvG3PzC4lJxubSyura+Ud7caqZRljDeYJEfJW3XSbnvhbwhPOHzdpxwJ3B93nJHpzLeuudJ6kXhlRjHvBM4g9Dre8wRRF0Pu7k9ucvPJ91yxapaapmzwNagAr3qUfkFt+ghAkOGABwhBGEfDlJ6bmDDQkxcBzlxCSFPxTkmKJE2oyxOGQ6xI/oOaHej2ZD20jNVakan+PQmpDSxR5qI8hLC8jRTxTPlLNnfvHPlKe82pr+rvQJiBYbE/qWbZv5XJ2sR6ONY1eBRTbFiZHVMu2SqK/Lm5peqBDnExEnco3hCmCnltM+m0qSqdtlbR8XfVKZk5Z7p3Azv8pY0YPvnOGdB86BqW1X78rBSO9GjLmIHu9ineR6hhjPU0SDvAI94wrNxYQgjNyafqUZBa7bxbRkPH/5pkuQ=</latexit><latexit sha1_base64="DT+3KsZZtd7aVK4fLuT/2gohxms=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0oxupYB+itSTTaRuaF8lEKKFbf8Ctfpf4B/oX3hmnoBbRCUnOnHvPmbn3urHvpcKyXgvG3PzC4lJxubSyura+Ud7caqZRljDeYJEfJW3XSbnvhbwhPOHzdpxwJ3B93nJHpzLeuudJ6kXhlRjHvBM4g9Dre8wRRF0Pu7k9ucvPJ91yxapaapmzwNagAr3qUfkFt+ghAkOGABwhBGEfDlJ6bmDDQkxcBzlxCSFPxTkmKJE2oyxOGQ6xI/oOaHej2ZD20jNVakan+PQmpDSxR5qI8hLC8jRTxTPlLNnfvHPlKe82pr+rvQJiBYbE/qWbZv5XJ2sR6ONY1eBRTbFiZHVMu2SqK/Lm5peqBDnExEnco3hCmCnltM+m0qSqdtlbR8XfVKZk5Z7p3Azv8pY0YPvnOGdB86BqW1X78rBSO9GjLmIHu9ineR6hhjPU0SDvAI94wrNxYQgjNyafqUZBa7bxbRkPH/5pkuQ=</latexit> h I 2
<latexit sha1_base64="HeowE0H4h5VL5Eukv4XZhEZLcw4=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRjdSwT6k1pJMp21oXiQToYRs/QG3+l3iH+hfeGdMQS2iE5KcOfeeM3PvtUPXiYVhvBa0hcWl5ZXiamltfWNzq7y904qDJGK8yQI3iDq2FXPX8XlTOMLlnTDilme7vG1PzmS8fc+j2An8azENec+zRr4zdJgliLoZ99NadpdeZP1yxagaaunzwMxBBflqBOUX3GKAAAwJPHD4EIRdWIjp6cKEgZC4HlLiIkKOinNkKJE2oSxOGRaxE/qOaNfNWZ/20jNWakanuPRGpNRxQJqA8iLC8jRdxRPlLNnfvFPlKe82pb+de3nECoyJ/Us3y/yvTtYiMMSJqsGhmkLFyOpY7pKorsib61+qEuQQEifxgOIRYaaUsz7rShOr2mVvLRV/U5mSlXuW5yZ4l7ekAZs/xzkPWrWqaVTNq6NK/TQfdRF72MchzfMYdZyjgSZ5e3jEE561S01oqZZ9pmqFXLOLb0t7+AAA3ZLl</latexit><latexit sha1_base64="HeowE0H4h5VL5Eukv4XZhEZLcw4=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRjdSwT6k1pJMp21oXiQToYRs/QG3+l3iH+hfeGdMQS2iE5KcOfeeM3PvtUPXiYVhvBa0hcWl5ZXiamltfWNzq7y904qDJGK8yQI3iDq2FXPX8XlTOMLlnTDilme7vG1PzmS8fc+j2An8azENec+zRr4zdJgliLoZ99NadpdeZP1yxagaaunzwMxBBflqBOUX3GKAAAwJPHD4EIRdWIjp6cKEgZC4HlLiIkKOinNkKJE2oSxOGRaxE/qOaNfNWZ/20jNWakanuPRGpNRxQJqA8iLC8jRdxRPlLNnfvFPlKe82pb+de3nECoyJ/Us3y/yvTtYiMMSJqsGhmkLFyOpY7pKorsib61+qEuQQEifxgOIRYaaUsz7rShOr2mVvLRV/U5mSlXuW5yZ4l7ekAZs/xzkPWrWqaVTNq6NK/TQfdRF72MchzfMYdZyjgSZ5e3jEE561S01oqZZ9pmqFXLOLb0t7+AAA3ZLl</latexit><latexit sha1_base64="HeowE0H4h5VL5Eukv4XZhEZLcw4=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRjdSwT6k1pJMp21oXiQToYRs/QG3+l3iH+hfeGdMQS2iE5KcOfeeM3PvtUPXiYVhvBa0hcWl5ZXiamltfWNzq7y904qDJGK8yQI3iDq2FXPX8XlTOMLlnTDilme7vG1PzmS8fc+j2An8azENec+zRr4zdJgliLoZ99NadpdeZP1yxagaaunzwMxBBflqBOUX3GKAAAwJPHD4EIRdWIjp6cKEgZC4HlLiIkKOinNkKJE2oSxOGRaxE/qOaNfNWZ/20jNWakanuPRGpNRxQJqA8iLC8jRdxRPlLNnfvFPlKe82pb+de3nECoyJ/Us3y/yvTtYiMMSJqsGhmkLFyOpY7pKorsib61+qEuQQEifxgOIRYaaUsz7rShOr2mVvLRV/U5mSlXuW5yZ4l7ekAZs/xzkPWrWqaVTNq6NK/TQfdRF72MchzfMYdZyjgSZ5e3jEE561S01oqZZ9pmqFXLOLb0t7+AAA3ZLl</latexit><latexit sha1_base64="HeowE0H4h5VL5Eukv4XZhEZLcw4=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRjdSwT6k1pJMp21oXiQToYRs/QG3+l3iH+hfeGdMQS2iE5KcOfeeM3PvtUPXiYVhvBa0hcWl5ZXiamltfWNzq7y904qDJGK8yQI3iDq2FXPX8XlTOMLlnTDilme7vG1PzmS8fc+j2An8azENec+zRr4zdJgliLoZ99NadpdeZP1yxagaaunzwMxBBflqBOUX3GKAAAwJPHD4EIRdWIjp6cKEgZC4HlLiIkKOinNkKJE2oSxOGRaxE/qOaNfNWZ/20jNWakanuPRGpNRxQJqA8iLC8jRdxRPlLNnfvFPlKe82pb+de3nECoyJ/Us3y/yvTtYiMMSJqsGhmkLFyOpY7pKorsib61+qEuQQEifxgOIRYaaUsz7rShOr2mVvLRV/U5mSlXuW5yZ4l7ekAZs/xzkPWrWqaVTNq6NK/TQfdRF72MchzfMYdZyjgSZ5e3jEE561S01oqZZ9pmqFXLOLb0t7+AAA3ZLl</latexit>
hI
3
<latexit sha1_base64="Lhjjb5BqmKlvyQ2nXCZJSds+ox4=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0oxupYB9Sa0mm0zY0L5KJUEK2/oBb/S7xD/QvvDOmoBbRCUnOnHvPmbn32qHrxMIwXgva3PzC4lJxubSyura+Ud7casZBEjHeYIEbRG3birnr+LwhHOHydhhxy7Nd3rLHZzLeuudR7AT+tZiEvOtZQ98ZOMwSRN2MeulhdpdeZL1yxagaaumzwMxBBfmqB+UX3KKPAAwJPHD4EIRdWIjp6cCEgZC4LlLiIkKOinNkKJE2oSxOGRaxY/oOadfJWZ/20jNWakanuPRGpNSxR5qA8iLC8jRdxRPlLNnfvFPlKe82ob+de3nECoyI/Us3zfyvTtYiMMCJqsGhmkLFyOpY7pKorsib61+qEuQQEidxn+IRYaaU0z7rShOr2mVvLRV/U5mSlXuW5yZ4l7ekAZs/xzkLmgdV06iaV0eV2mk+6iJ2sIt9mucxajhHHQ3y9vCIJzxrl5rQUi37TNUKuWYb35b28AEDQpLm</latexit><latexit sha1_base64="Lhjjb5BqmKlvyQ2nXCZJSds+ox4=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0oxupYB9Sa0mm0zY0L5KJUEK2/oBb/S7xD/QvvDOmoBbRCUnOnHvPmbn32qHrxMIwXgva3PzC4lJxubSyura+Ud7casZBEjHeYIEbRG3birnr+LwhHOHydhhxy7Nd3rLHZzLeuudR7AT+tZiEvOtZQ98ZOMwSRN2MeulhdpdeZL1yxagaaumzwMxBBfmqB+UX3KKPAAwJPHD4EIRdWIjp6cCEgZC4LlLiIkKOinNkKJE2oSxOGRaxY/oOadfJWZ/20jNWakanuPRGpNSxR5qA8iLC8jRdxRPlLNnfvFPlKe82ob+de3nECoyI/Us3zfyvTtYiMMCJqsGhmkLFyOpY7pKorsib61+qEuQQEidxn+IRYaaU0z7rShOr2mVvLRV/U5mSlXuW5yZ4l7ekAZs/xzkLmgdV06iaV0eV2mk+6iJ2sIt9mucxajhHHQ3y9vCIJzxrl5rQUi37TNUKuWYb35b28AEDQpLm</latexit><latexit sha1_base64="Lhjjb5BqmKlvyQ2nXCZJSds+ox4=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0oxupYB9Sa0mm0zY0L5KJUEK2/oBb/S7xD/QvvDOmoBbRCUnOnHvPmbn32qHrxMIwXgva3PzC4lJxubSyura+Ud7casZBEjHeYIEbRG3birnr+LwhHOHydhhxy7Nd3rLHZzLeuudR7AT+tZiEvOtZQ98ZOMwSRN2MeulhdpdeZL1yxagaaumzwMxBBfmqB+UX3KKPAAwJPHD4EIRdWIjp6cCEgZC4LlLiIkKOinNkKJE2oSxOGRaxY/oOadfJWZ/20jNWakanuPRGpNSxR5qA8iLC8jRdxRPlLNnfvFPlKe82ob+de3nECoyI/Us3zfyvTtYiMMCJqsGhmkLFyOpY7pKorsib61+qEuQQEidxn+IRYaaU0z7rShOr2mVvLRV/U5mSlXuW5yZ4l7ekAZs/xzkLmgdV06iaV0eV2mk+6iJ2sIt9mucxajhHHQ3y9vCIJzxrl5rQUi37TNUKuWYb35b28AEDQpLm</latexit><latexit sha1_base64="Lhjjb5BqmKlvyQ2nXCZJSds+ox4=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0oxupYB9Sa0mm0zY0L5KJUEK2/oBb/S7xD/QvvDOmoBbRCUnOnHvPmbn32qHrxMIwXgva3PzC4lJxubSyura+Ud7casZBEjHeYIEbRG3birnr+LwhHOHydhhxy7Nd3rLHZzLeuudR7AT+tZiEvOtZQ98ZOMwSRN2MeulhdpdeZL1yxagaaumzwMxBBfmqB+UX3KKPAAwJPHD4EIRdWIjp6cCEgZC4LlLiIkKOinNkKJE2oSxOGRaxY/oOadfJWZ/20jNWakanuPRGpNSxR5qA8iLC8jRdxRPlLNnfvFPlKe82ob+de3nECoyI/Us3zfyvTtYiMMCJqsGhmkLFyOpY7pKorsib61+qEuQQEidxn+IRYaaU0z7rShOr2mVvLRV/U5mSlXuW5yZ4l7ekAZs/xzkLmgdV06iaV0eV2mk+6iJ2sIt9mucxajhHHQ3y9vCIJzxrl5rQUi37TNUKuWYb35b28AEDQpLm</latexit>
yI
1
<latexit sha1_base64="aXffR0+U7zD6WDCXHpfQVh2mHEs=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0oxupYB9Sa0nSaR2aF8lEKKFbf8Ctfpf4B/oX3hmnoBbRCUnOnHvPmbn3urHPU2FZrwVjbn5hcam4XFpZXVvfKG9uNdMoSzzW8CI/StqukzKfh6whuPBZO06YE7g+a7mjUxlv3bMk5VF4JcYx6wbOMOQD7jmCqOtxL7cnt/n5pFeuWFVLLXMW2BpUoFc9Kr/gBn1E8JAhAEMIQdiHg5SeDmxYiInrIicuIcRVnGGCEmkzymKU4RA7ou+Qdh3NhrSXnqlSe3SKT29CShN7pIkoLyEsTzNVPFPOkv3NO1ee8m5j+rvaKyBW4I7Yv3TTzP/qZC0CAxyrGjjVFCtGVudpl0x1Rd7c/FKVIIeYOIn7FE8Ie0o57bOpNKmqXfbWUfE3lSlZufd0boZ3eUsasP1znLOgeVC1rap9eVipnehRF7GDXezTPI9QwxnqaJB3gEc84dm4MISRG5PPVKOgNdv4toyHDydgkvU=</latexit><latexit sha1_base64="aXffR0+U7zD6WDCXHpfQVh2mHEs=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0oxupYB9Sa0nSaR2aF8lEKKFbf8Ctfpf4B/oX3hmnoBbRCUnOnHvPmbn3urHPU2FZrwVjbn5hcam4XFpZXVvfKG9uNdMoSzzW8CI/StqukzKfh6whuPBZO06YE7g+a7mjUxlv3bMk5VF4JcYx6wbOMOQD7jmCqOtxL7cnt/n5pFeuWFVLLXMW2BpUoFc9Kr/gBn1E8JAhAEMIQdiHg5SeDmxYiInrIicuIcRVnGGCEmkzymKU4RA7ou+Qdh3NhrSXnqlSe3SKT29CShN7pIkoLyEsTzNVPFPOkv3NO1ee8m5j+rvaKyBW4I7Yv3TTzP/qZC0CAxyrGjjVFCtGVudpl0x1Rd7c/FKVIIeYOIn7FE8Ie0o57bOpNKmqXfbWUfE3lSlZufd0boZ3eUsasP1znLOgeVC1rap9eVipnehRF7GDXezTPI9QwxnqaJB3gEc84dm4MISRG5PPVKOgNdv4toyHDydgkvU=</latexit><latexit sha1_base64="aXffR0+U7zD6WDCXHpfQVh2mHEs=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0oxupYB9Sa0nSaR2aF8lEKKFbf8Ctfpf4B/oX3hmnoBbRCUnOnHvPmbn3urHPU2FZrwVjbn5hcam4XFpZXVvfKG9uNdMoSzzW8CI/StqukzKfh6whuPBZO06YE7g+a7mjUxlv3bMk5VF4JcYx6wbOMOQD7jmCqOtxL7cnt/n5pFeuWFVLLXMW2BpUoFc9Kr/gBn1E8JAhAEMIQdiHg5SeDmxYiInrIicuIcRVnGGCEmkzymKU4RA7ou+Qdh3NhrSXnqlSe3SKT29CShN7pIkoLyEsTzNVPFPOkv3NO1ee8m5j+rvaKyBW4I7Yv3TTzP/qZC0CAxyrGjjVFCtGVudpl0x1Rd7c/FKVIIeYOIn7FE8Ie0o57bOpNKmqXfbWUfE3lSlZufd0boZ3eUsasP1znLOgeVC1rap9eVipnehRF7GDXezTPI9QwxnqaJB3gEc84dm4MISRG5PPVKOgNdv4toyHDydgkvU=</latexit><latexit sha1_base64="aXffR0+U7zD6WDCXHpfQVh2mHEs=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0oxupYB9Sa0nSaR2aF8lEKKFbf8Ctfpf4B/oX3hmnoBbRCUnOnHvPmbn3urHPU2FZrwVjbn5hcam4XFpZXVvfKG9uNdMoSzzW8CI/StqukzKfh6whuPBZO06YE7g+a7mjUxlv3bMk5VF4JcYx6wbOMOQD7jmCqOtxL7cnt/n5pFeuWFVLLXMW2BpUoFc9Kr/gBn1E8JAhAEMIQdiHg5SeDmxYiInrIicuIcRVnGGCEmkzymKU4RA7ou+Qdh3NhrSXnqlSe3SKT29CShN7pIkoLyEsTzNVPFPOkv3NO1ee8m5j+rvaKyBW4I7Yv3TTzP/qZC0CAxyrGjjVFCtGVudpl0x1Rd7c/FKVIIeYOIn7FE8Ie0o57bOpNKmqXfbWUfE3lSlZufd0boZ3eUsasP1znLOgeVC1rap9eVipnehRF7GDXezTPI9QwxnqaJB3gEc84dm4MISRG5PPVKOgNdv4toyHDydgkvU=</latexit> y I 2
<latexit sha1_base64="lCQF+ce0PZhNE9W3CXyq0bRUSS4=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRjdSwT5Ea0nSaR2aF8lEKKFbf8Ctfpf4B/oX3hmnoBbRCUnOnHvPmbn3urHPU2FZrwVjbn5hcam4XFpZXVvfKG9utdIoSzzW9CI/SjqukzKfh6wpuPBZJ06YE7g+a7ujExlv37Mk5VF4KcYx6wbOMOQD7jmCqKtxL69NbvOzSa9csaqWWuYssDWoQK9GVH7BDfqI4CFDAIYQgrAPByk917BhISaui5y4hBBXcYYJSqTNKItRhkPsiL5D2l1rNqS99EyV2qNTfHoTUprYI01EeQlheZqp4plyluxv3rnylHcb09/VXgGxAnfE/qWbZv5XJ2sRGOBI1cCpplgxsjpPu2SqK/Lm5peqBDnExEncp3hC2FPKaZ9NpUlV7bK3joq/qUzJyr2nczO8y1vSgO2f45wFrVrVtqr2xUGlfqxHXcQOdrFP8zxEHadooEneAR7xhGfj3BBGbkw+U42C1mzj2zIePgApxZL2</latexit><latexit sha1_base64="lCQF+ce0PZhNE9W3CXyq0bRUSS4=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRjdSwT5Ea0nSaR2aF8lEKKFbf8Ctfpf4B/oX3hmnoBbRCUnOnHvPmbn3urHPU2FZrwVjbn5hcam4XFpZXVvfKG9utdIoSzzW9CI/SjqukzKfh6wpuPBZJ06YE7g+a7ujExlv37Mk5VF4KcYx6wbOMOQD7jmCqKtxL69NbvOzSa9csaqWWuYssDWoQK9GVH7BDfqI4CFDAIYQgrAPByk917BhISaui5y4hBBXcYYJSqTNKItRhkPsiL5D2l1rNqS99EyV2qNTfHoTUprYI01EeQlheZqp4plyluxv3rnylHcb09/VXgGxAnfE/qWbZv5XJ2sRGOBI1cCpplgxsjpPu2SqK/Lm5peqBDnExEncp3hC2FPKaZ9NpUlV7bK3joq/qUzJyr2nczO8y1vSgO2f45wFrVrVtqr2xUGlfqxHXcQOdrFP8zxEHadooEneAR7xhGfj3BBGbkw+U42C1mzj2zIePgApxZL2</latexit><latexit sha1_base64="lCQF+ce0PZhNE9W3CXyq0bRUSS4=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRjdSwT5Ea0nSaR2aF8lEKKFbf8Ctfpf4B/oX3hmnoBbRCUnOnHvPmbn3urHPU2FZrwVjbn5hcam4XFpZXVvfKG9utdIoSzzW9CI/SjqukzKfh6wpuPBZJ06YE7g+a7ujExlv37Mk5VF4KcYx6wbOMOQD7jmCqKtxL69NbvOzSa9csaqWWuYssDWoQK9GVH7BDfqI4CFDAIYQgrAPByk917BhISaui5y4hBBXcYYJSqTNKItRhkPsiL5D2l1rNqS99EyV2qNTfHoTUprYI01EeQlheZqp4plyluxv3rnylHcb09/VXgGxAnfE/qWbZv5XJ2sRGOBI1cCpplgxsjpPu2SqK/Lm5peqBDnExEncp3hC2FPKaZ9NpUlV7bK3joq/qUzJyr2nczO8y1vSgO2f45wFrVrVtqr2xUGlfqxHXcQOdrFP8zxEHadooEneAR7xhGfj3BBGbkw+U42C1mzj2zIePgApxZL2</latexit><latexit sha1_base64="lCQF+ce0PZhNE9W3CXyq0bRUSS4=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRjdSwT5Ea0nSaR2aF8lEKKFbf8Ctfpf4B/oX3hmnoBbRCUnOnHvPmbn3urHPU2FZrwVjbn5hcam4XFpZXVvfKG9utdIoSzzW9CI/SjqukzKfh6wpuPBZJ06YE7g+a7ujExlv37Mk5VF4KcYx6wbOMOQD7jmCqKtxL69NbvOzSa9csaqWWuYssDWoQK9GVH7BDfqI4CFDAIYQgrAPByk917BhISaui5y4hBBXcYYJSqTNKItRhkPsiL5D2l1rNqS99EyV2qNTfHoTUprYI01EeQlheZqp4plyluxv3rnylHcb09/VXgGxAnfE/qWbZv5XJ2sRGOBI1cCpplgxsjpPu2SqK/Lm5peqBDnExEncp3hC2FPKaZ9NpUlV7bK3joq/qUzJyr2nczO8y1vSgO2f45wFrVrVtqr2xUGlfqxHXcQOdrFP8zxEHadooEneAR7xhGfj3BBGbkw+U42C1mzj2zIePgApxZL2</latexit> y I 3
<latexit sha1_base64="9uFL3tkFHzoasuUxqza6+iqxvSI=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0oxupYB9Sa0mm0zqYF8lEKKFbf8Ctfpf4B/oX3hlTUIvohCRnzr3nzNx73cgTibSs14IxMzs3v1BcLC0tr6yuldc3mkmYxow3WOiFcdt1Eu6JgDekkB5vRzF3fNfjLffuRMVb9zxORBhcylHEu74zDMRAMEcSdTXqZfvjm+xs3CtXrKqllzkN7BxUkK96WH7BNfoIwZDCB0cASdiDg4SeDmxYiIjrIiMuJiR0nGOMEmlTyuKU4RB7R98h7To5G9BeeSZazegUj96YlCZ2SBNSXkxYnWbqeKqdFfubd6Y91d1G9HdzL59YiVti/9JNMv+rU7VIDHCkaxBUU6QZVR3LXVLdFXVz80tVkhwi4hTuUzwmzLRy0mdTaxJdu+qto+NvOlOxas/y3BTv6pY0YPvnOKdBc69qW1X74qBSO85HXcQWtrFL8zxEDaeoo0HePh7xhGfj3JBGZow/U41CrtnEt2U8fAAsKpL3</latexit><latexit sha1_base64="9uFL3tkFHzoasuUxqza6+iqxvSI=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0oxupYB9Sa0mm0zqYF8lEKKFbf8Ctfpf4B/oX3hlTUIvohCRnzr3nzNx73cgTibSs14IxMzs3v1BcLC0tr6yuldc3mkmYxow3WOiFcdt1Eu6JgDekkB5vRzF3fNfjLffuRMVb9zxORBhcylHEu74zDMRAMEcSdTXqZfvjm+xs3CtXrKqllzkN7BxUkK96WH7BNfoIwZDCB0cASdiDg4SeDmxYiIjrIiMuJiR0nGOMEmlTyuKU4RB7R98h7To5G9BeeSZazegUj96YlCZ2SBNSXkxYnWbqeKqdFfubd6Y91d1G9HdzL59YiVti/9JNMv+rU7VIDHCkaxBUU6QZVR3LXVLdFXVz80tVkhwi4hTuUzwmzLRy0mdTaxJdu+qto+NvOlOxas/y3BTv6pY0YPvnOKdBc69qW1X74qBSO85HXcQWtrFL8zxEDaeoo0HePh7xhGfj3JBGZow/U41CrtnEt2U8fAAsKpL3</latexit><latexit sha1_base64="9uFL3tkFHzoasuUxqza6+iqxvSI=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0oxupYB9Sa0mm0zqYF8lEKKFbf8Ctfpf4B/oX3hlTUIvohCRnzr3nzNx73cgTibSs14IxMzs3v1BcLC0tr6yuldc3mkmYxow3WOiFcdt1Eu6JgDekkB5vRzF3fNfjLffuRMVb9zxORBhcylHEu74zDMRAMEcSdTXqZfvjm+xs3CtXrKqllzkN7BxUkK96WH7BNfoIwZDCB0cASdiDg4SeDmxYiIjrIiMuJiR0nGOMEmlTyuKU4RB7R98h7To5G9BeeSZazegUj96YlCZ2SBNSXkxYnWbqeKqdFfubd6Y91d1G9HdzL59YiVti/9JNMv+rU7VIDHCkaxBUU6QZVR3LXVLdFXVz80tVkhwi4hTuUzwmzLRy0mdTaxJdu+qto+NvOlOxas/y3BTv6pY0YPvnOKdBc69qW1X74qBSO85HXcQWtrFL8zxEDaeoo0HePh7xhGfj3JBGZow/U41CrtnEt2U8fAAsKpL3</latexit><latexit sha1_base64="9uFL3tkFHzoasuUxqza6+iqxvSI=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0oxupYB9Sa0mm0zqYF8lEKKFbf8Ctfpf4B/oX3hlTUIvohCRnzr3nzNx73cgTibSs14IxMzs3v1BcLC0tr6yuldc3mkmYxow3WOiFcdt1Eu6JgDekkB5vRzF3fNfjLffuRMVb9zxORBhcylHEu74zDMRAMEcSdTXqZfvjm+xs3CtXrKqllzkN7BxUkK96WH7BNfoIwZDCB0cASdiDg4SeDmxYiIjrIiMuJiR0nGOMEmlTyuKU4RB7R98h7To5G9BeeSZazegUj96YlCZ2SBNSXkxYnWbqeKqdFfubd6Y91d1G9HdzL59YiVti/9JNMv+rU7VIDHCkaxBUU6QZVR3LXVLdFXVz80tVkhwi4hTuUzwmzLRy0mdTaxJdu+qto+NvOlOxas/y3BTv6pY0YPvnOKdBc69qW1X74qBSO85HXcQWtrFL8zxEDaeoo0HePh7xhGfj3JBGZow/U41CrtnEt2U8fAAsKpL3</latexit>
e1
<latexit sha1_base64="UOJMj9uT+rMuVRiicAlHNm5A4CY=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmLZQS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO400ziTjHovDWLYCP+WhiLinhAp5K5HcHwUhbwbDcx1v3nGZiji6VuOEd0b+IBJ9wXxFlMe7E3faLVecqmOWPQ/cHFSQr3pcfsENeojBkGEEjgiKcAgfKT1tuHCQENfBhDhJSJg4xxQl0maUxSnDJ3ZI3wHt2jkb0V57pkbN6JSQXklKGwekiSlPEtan2SaeGWfN/uY9MZ76bmP6B7nXiFiFW2L/0s0y/6vTtSj0cWpqEFRTYhhdHctdMtMVfXP7S1WKHBLiNO5RXBJmRjnrs200qald99Y38TeTqVm9Z3luhnd9Sxqw+3Oc86BxVHWdqnt1XKmd5aMuYg/7OKR5nqCGC9ThkbfAI57wbF1aiXVvjT9TrUKu2cW3ZT18AHwckRo=</latexit><latexit sha1_base64="UOJMj9uT+rMuVRiicAlHNm5A4CY=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmLZQS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO400ziTjHovDWLYCP+WhiLinhAp5K5HcHwUhbwbDcx1v3nGZiji6VuOEd0b+IBJ9wXxFlMe7E3faLVecqmOWPQ/cHFSQr3pcfsENeojBkGEEjgiKcAgfKT1tuHCQENfBhDhJSJg4xxQl0maUxSnDJ3ZI3wHt2jkb0V57pkbN6JSQXklKGwekiSlPEtan2SaeGWfN/uY9MZ76bmP6B7nXiFiFW2L/0s0y/6vTtSj0cWpqEFRTYhhdHctdMtMVfXP7S1WKHBLiNO5RXBJmRjnrs200qald99Y38TeTqVm9Z3luhnd9Sxqw+3Oc86BxVHWdqnt1XKmd5aMuYg/7OKR5nqCGC9ThkbfAI57wbF1aiXVvjT9TrUKu2cW3ZT18AHwckRo=</latexit><latexit sha1_base64="UOJMj9uT+rMuVRiicAlHNm5A4CY=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmLZQS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO400ziTjHovDWLYCP+WhiLinhAp5K5HcHwUhbwbDcx1v3nGZiji6VuOEd0b+IBJ9wXxFlMe7E3faLVecqmOWPQ/cHFSQr3pcfsENeojBkGEEjgiKcAgfKT1tuHCQENfBhDhJSJg4xxQl0maUxSnDJ3ZI3wHt2jkb0V57pkbN6JSQXklKGwekiSlPEtan2SaeGWfN/uY9MZ76bmP6B7nXiFiFW2L/0s0y/6vTtSj0cWpqEFRTYhhdHctdMtMVfXP7S1WKHBLiNO5RXBJmRjnrs200qald99Y38TeTqVm9Z3luhnd9Sxqw+3Oc86BxVHWdqnt1XKmd5aMuYg/7OKR5nqCGC9ThkbfAI57wbF1aiXVvjT9TrUKu2cW3ZT18AHwckRo=</latexit><latexit sha1_base64="UOJMj9uT+rMuVRiicAlHNm5A4CY=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmLZQS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO400ziTjHovDWLYCP+WhiLinhAp5K5HcHwUhbwbDcx1v3nGZiji6VuOEd0b+IBJ9wXxFlMe7E3faLVecqmOWPQ/cHFSQr3pcfsENeojBkGEEjgiKcAgfKT1tuHCQENfBhDhJSJg4xxQl0maUxSnDJ3ZI3wHt2jkb0V57pkbN6JSQXklKGwekiSlPEtan2SaeGWfN/uY9MZ76bmP6B7nXiFiFW2L/0s0y/6vTtSj0cWpqEFRTYhhdHctdMtMVfXP7S1WKHBLiNO5RXBJmRjnrs200qald99Y38TeTqVm9Z3luhnd9Sxqw+3Oc86BxVHWdqnt1XKmd5aMuYg/7OKR5nqCGC9ThkbfAI57wbF1aiXVvjT9TrUKu2cW3ZT18AHwckRo=</latexit> e2<latexit sha1_base64="F3mSGmP2ncMwlvmE/DRl8ixSd+o=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRlxVMK1QS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO800ziTjHovDWF4HfspDEXFPCRXy60RyfxSEvBUMz3S8dcdlKuLoSo0T3hn5g0j0BfMVUR7vTmrTbrniVB2z7Hng5qCCfDXi8gtu0EMMhgwjcERQhEP4SOlpw4WDhLgOJsRJQsLEOaYokTajLE4ZPrFD+g5o187ZiPbaMzVqRqeE9EpS2jggTUx5krA+zTbxzDhr9jfvifHUdxvTP8i9RsQq3BL7l26W+V+drkWhjxNTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GP4pIwM8pZn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAfNWtV1qu7lUaV+mo+6iD3s45DmeYw6ztGAR94Cj3jCs3VhJda9Nf5MtQq5ZhfflvXwAX59kRs=</latexit><latexit sha1_base64="F3mSGmP2ncMwlvmE/DRl8ixSd+o=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRlxVMK1QS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO800ziTjHovDWF4HfspDEXFPCRXy60RyfxSEvBUMz3S8dcdlKuLoSo0T3hn5g0j0BfMVUR7vTmrTbrniVB2z7Hng5qCCfDXi8gtu0EMMhgwjcERQhEP4SOlpw4WDhLgOJsRJQsLEOaYokTajLE4ZPrFD+g5o187ZiPbaMzVqRqeE9EpS2jggTUx5krA+zTbxzDhr9jfvifHUdxvTP8i9RsQq3BL7l26W+V+drkWhjxNTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GP4pIwM8pZn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAfNWtV1qu7lUaV+mo+6iD3s45DmeYw6ztGAR94Cj3jCs3VhJda9Nf5MtQq5ZhfflvXwAX59kRs=</latexit><latexit sha1_base64="F3mSGmP2ncMwlvmE/DRl8ixSd+o=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRlxVMK1QS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO800ziTjHovDWF4HfspDEXFPCRXy60RyfxSEvBUMz3S8dcdlKuLoSo0T3hn5g0j0BfMVUR7vTmrTbrniVB2z7Hng5qCCfDXi8gtu0EMMhgwjcERQhEP4SOlpw4WDhLgOJsRJQsLEOaYokTajLE4ZPrFD+g5o187ZiPbaMzVqRqeE9EpS2jggTUx5krA+zTbxzDhr9jfvifHUdxvTP8i9RsQq3BL7l26W+V+drkWhjxNTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GP4pIwM8pZn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAfNWtV1qu7lUaV+mo+6iD3s45DmeYw6ztGAR94Cj3jCs3VhJda9Nf5MtQq5ZhfflvXwAX59kRs=</latexit><latexit sha1_base64="F3mSGmP2ncMwlvmE/DRl8ixSd+o=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoRlxVMK1QS0mm0zo0TcJkopTSjT/gVr9M/AP9C++MKahFdEKSM+fec2buvUESilQ5zmvBWlhcWl4prpbW1jc2t8rbO800ziTjHovDWF4HfspDEXFPCRXy60RyfxSEvBUMz3S8dcdlKuLoSo0T3hn5g0j0BfMVUR7vTmrTbrniVB2z7Hng5qCCfDXi8gtu0EMMhgwjcERQhEP4SOlpw4WDhLgOJsRJQsLEOaYokTajLE4ZPrFD+g5o187ZiPbaMzVqRqeE9EpS2jggTUx5krA+zTbxzDhr9jfvifHUdxvTP8i9RsQq3BL7l26W+V+drkWhjxNTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GP4pIwM8pZn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAfNWtV1qu7lUaV+mo+6iD3s45DmeYw6ztGAR94Cj3jCs3VhJda9Nf5MtQq5ZhfflvXwAX59kRs=</latexit> e3<latexit sha1_base64="eb4TJOGLUZm66tydoux6u5LjkOo=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0I64qmLZQS0mm0zqYJiGZKKV04w+41S8T/0D/wjvjFNQiOiHJmXPvOTP33iAJRSYd57Vgzc0vLC4Vl0srq2vrG+XNrUYW5ynjHovDOG0FfsZDEXFPChnyVpJyfxiEvBncnql4846nmYijKzlKeGfoDyLRF8yXRHm8Oz6cdMsVp+roZc8C14AKzKrH5Rdco4cYDDmG4IggCYfwkdHThgsHCXEdjIlLCQkd55igRNqcsjhl+MTe0ndAu7ZhI9orz0yrGZ0S0puS0sYeaWLKSwmr02wdz7WzYn/zHmtPdbcR/QPjNSRW4obYv3TTzP/qVC0SfZzoGgTVlGhGVceMS667om5uf6lKkkNCnMI9iqeEmVZO+2xrTaZrV731dfxNZypW7ZnJzfGubkkDdn+OcxY0DqquU3Uvjyq1UzPqInawi32a5zFqOEcdHnkLPOIJz9aFlVj31ugz1SoYzTa+LevhA4DekRw=</latexit><latexit sha1_base64="eb4TJOGLUZm66tydoux6u5LjkOo=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0I64qmLZQS0mm0zqYJiGZKKV04w+41S8T/0D/wjvjFNQiOiHJmXPvOTP33iAJRSYd57Vgzc0vLC4Vl0srq2vrG+XNrUYW5ynjHovDOG0FfsZDEXFPChnyVpJyfxiEvBncnql4846nmYijKzlKeGfoDyLRF8yXRHm8Oz6cdMsVp+roZc8C14AKzKrH5Rdco4cYDDmG4IggCYfwkdHThgsHCXEdjIlLCQkd55igRNqcsjhl+MTe0ndAu7ZhI9orz0yrGZ0S0puS0sYeaWLKSwmr02wdz7WzYn/zHmtPdbcR/QPjNSRW4obYv3TTzP/qVC0SfZzoGgTVlGhGVceMS667om5uf6lKkkNCnMI9iqeEmVZO+2xrTaZrV731dfxNZypW7ZnJzfGubkkDdn+OcxY0DqquU3Uvjyq1UzPqInawi32a5zFqOEcdHnkLPOIJz9aFlVj31ugz1SoYzTa+LevhA4DekRw=</latexit><latexit sha1_base64="eb4TJOGLUZm66tydoux6u5LjkOo=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0I64qmLZQS0mm0zqYJiGZKKV04w+41S8T/0D/wjvjFNQiOiHJmXPvOTP33iAJRSYd57Vgzc0vLC4Vl0srq2vrG+XNrUYW5ynjHovDOG0FfsZDEXFPChnyVpJyfxiEvBncnql4846nmYijKzlKeGfoDyLRF8yXRHm8Oz6cdMsVp+roZc8C14AKzKrH5Rdco4cYDDmG4IggCYfwkdHThgsHCXEdjIlLCQkd55igRNqcsjhl+MTe0ndAu7ZhI9orz0yrGZ0S0puS0sYeaWLKSwmr02wdz7WzYn/zHmtPdbcR/QPjNSRW4obYv3TTzP/qVC0SfZzoGgTVlGhGVceMS667om5uf6lKkkNCnMI9iqeEmVZO+2xrTaZrV731dfxNZypW7ZnJzfGubkkDdn+OcxY0DqquU3Uvjyq1UzPqInawi32a5zFqOEcdHnkLPOIJz9aFlVj31ugz1SoYzTa+LevhA4DekRw=</latexit><latexit sha1_base64="eb4TJOGLUZm66tydoux6u5LjkOo=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl0I64qmLZQS0mm0zqYJiGZKKV04w+41S8T/0D/wjvjFNQiOiHJmXPvOTP33iAJRSYd57Vgzc0vLC4Vl0srq2vrG+XNrUYW5ynjHovDOG0FfsZDEXFPChnyVpJyfxiEvBncnql4846nmYijKzlKeGfoDyLRF8yXRHm8Oz6cdMsVp+roZc8C14AKzKrH5Rdco4cYDDmG4IggCYfwkdHThgsHCXEdjIlLCQkd55igRNqcsjhl+MTe0ndAu7ZhI9orz0yrGZ0S0puS0sYeaWLKSwmr02wdz7WzYn/zHmtPdbcR/QPjNSRW4obYv3TTzP/qVC0SfZzoGgTVlGhGVceMS667om5uf6lKkkNCnMI9iqeEmVZO+2xrTaZrV731dfxNZypW7ZnJzfGubkkDdn+OcxY0DqquU3Uvjyq1UzPqInawi32a5zFqOEcdHnkLPOIJz9aFlVj31ugz1SoYzTa+LevhA4DekRw=</latexit>
yS
1
<latexit sha1_base64="7KQn+jiaa+WDI5N+WNipX54diVg=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl040oq2ofUWpJ0WofmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69buzzVFjWa8GYm19YXCoul1ZW19Y3yptbzTTKEo81vMiPkrbrpMznIWsILnzWjhPmBK7PWu7oVMZb9yxJeRReiXHMuoEzDPmAe44g6nrcy+3JbX456ZUrVtVSy5wFtgYV6FWPyi+4QR8RPGQIwBBCEPbhIKWnAxsWYuK6yIlLCHEVZ5igRNqMshhlOMSO6DukXUezIe2lZ6rUHp3i05uQ0sQeaSLKSwjL00wVz5SzZH/zzpWnvNuY/q72CogVuCP2L9008786WYvAAMeqBk41xYqR1XnaJVNdkTc3v1QlyCEmTuI+xRPCnlJO+2wqTapql711VPxNZUpW7j2dm+Fd3pIGbP8c5yxoHlRtq2pfHFZqJ3rURexgF/s0zyPUcIY6GuQd4BFPeDbODWHkxuQz1ShozTa+LePhAz8qkv8=</latexit><latexit sha1_base64="7KQn+jiaa+WDI5N+WNipX54diVg=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl040oq2ofUWpJ0WofmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69buzzVFjWa8GYm19YXCoul1ZW19Y3yptbzTTKEo81vMiPkrbrpMznIWsILnzWjhPmBK7PWu7oVMZb9yxJeRReiXHMuoEzDPmAe44g6nrcy+3JbX456ZUrVtVSy5wFtgYV6FWPyi+4QR8RPGQIwBBCEPbhIKWnAxsWYuK6yIlLCHEVZ5igRNqMshhlOMSO6DukXUezIe2lZ6rUHp3i05uQ0sQeaSLKSwjL00wVz5SzZH/zzpWnvNuY/q72CogVuCP2L9008786WYvAAMeqBk41xYqR1XnaJVNdkTc3v1QlyCEmTuI+xRPCnlJO+2wqTapql711VPxNZUpW7j2dm+Fd3pIGbP8c5yxoHlRtq2pfHFZqJ3rURexgF/s0zyPUcIY6GuQd4BFPeDbODWHkxuQz1ShozTa+LePhAz8qkv8=</latexit><latexit sha1_base64="7KQn+jiaa+WDI5N+WNipX54diVg=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl040oq2ofUWpJ0WofmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69buzzVFjWa8GYm19YXCoul1ZW19Y3yptbzTTKEo81vMiPkrbrpMznIWsILnzWjhPmBK7PWu7oVMZb9yxJeRReiXHMuoEzDPmAe44g6nrcy+3JbX456ZUrVtVSy5wFtgYV6FWPyi+4QR8RPGQIwBBCEPbhIKWnAxsWYuK6yIlLCHEVZ5igRNqMshhlOMSO6DukXUezIe2lZ6rUHp3i05uQ0sQeaSLKSwjL00wVz5SzZH/zzpWnvNuY/q72CogVuCP2L9008786WYvAAMeqBk41xYqR1XnaJVNdkTc3v1QlyCEmTuI+xRPCnlJO+2wqTapql711VPxNZUpW7j2dm+Fd3pIGbP8c5yxoHlRtq2pfHFZqJ3rURexgF/s0zyPUcIY6GuQd4BFPeDbODWHkxuQz1ShozTa+LePhAz8qkv8=</latexit>
<latexit sha1_base64="7KQn+jiaa+WDI5N+WNipX54diVg=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl040oq2ofUWpJ0WofmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69buzzVFjWa8GYm19YXCoul1ZW19Y3yptbzTTKEo81vMiPkrbrpMznIWsILnzWjhPmBK7PWu7oVMZb9yxJeRReiXHMuoEzDPmAe44g6nrcy+3JbX456ZUrVtVSy5wFtgYV6FWPyi+4QR8RPGQIwBBCEPbhIKWnAxsWYuK6yIlLCHEVZ5igRNqMshhlOMSO6DukXUezIe2lZ6rUHp3i05uQ0sQeaSLKSwjL00wVz5SzZH/zzpWnvNuY/q72CogVuCP2L9008786WYvAAMeqBk41xYqR1XnaJVNdkTc3v1QlyCEmTuI+xRPCnlJO+2wqTapql711VPxNZUpW7j2dm+Fd3pIGbP8c5yxoHlRtq2pfHFZqJ3rURexgF/s0zyPUcIY6GuQd4BFPeDbODWHkxuQz1ShozTa+LePhAz8qkv8=</latexit> y
S 2 <latexit sha1_base64="n1sH4BzpnCpJUU+cZTcSZN5qnqM=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoxpVUtA/RWpJ0WofmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69buzzVFjWa8GYm19YXCoul1ZW19Y3yptbrTTKEo81vciPko7rpMznIWsKLnzWiRPmBK7P2u7oRMbb9yxJeRReinHMuoEzDPmAe44g6mrcy2uT2/xi0itXrKqlljkLbA0q0KsRlV9wgz4ieMgQgCGEIOzDQUrPNWxYiInrIicuIcRVnGGCEmkzymKU4RA7ou+QdteaDWkvPVOl9ugUn96ElCb2SBNRXkJYnmaqeKacJfubd6485d3G9He1V0CswB2xf+mmmf/VyVoEBjhSNXCqKVaMrM7TLpnqiry5+aUqQQ4xcRL3KZ4Q9pRy2mdTaVJVu+yto+JvKlOycu/p3Azv8pY0YPvnOGdBq1a1rap9flCpH+tRF7GDXezTPA9RxykaaJJ3gEc84dk4M4SRG5PPVKOgNdv4toyHD0GPkwA=</latexit> <latexit sha1_base64="n1sH4BzpnCpJUU+cZTcSZN5qnqM=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoxpVUtA/RWpJ0WofmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69buzzVFjWa8GYm19YXCoul1ZW19Y3yptbrTTKEo81vciPko7rpMznIWsKLnzWiRPmBK7P2u7oRMbb9yxJeRReinHMuoEzDPmAe44g6mrcy2uT2/xi0itXrKqlljkLbA0q0KsRlV9wgz4ieMgQgCGEIOzDQUrPNWxYiInrIicuIcRVnGGCEmkzymKU4RA7ou+QdteaDWkvPVOl9ugUn96ElCb2SBNRXkJYnmaqeKacJfubd6485d3G9He1V0CswB2xf+mmmf/VyVoEBjhSNXCqKVaMrM7TLpnqiry5+aUqQQ4xcRL3KZ4Q9pRy2mdTaVJVu+yto+JvKlOycu/p3Azv8pY0YPvnOGdBq1a1rap9flCpH+tRF7GDXezTPA9RxykaaJJ3gEc84dk4M4SRG5PPVKOgNdv4toyHD0GPkwA=</latexit> <latexit sha1_base64="n1sH4BzpnCpJUU+cZTcSZN5qnqM=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoxpVUtA/RWpJ0WofmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69buzzVFjWa8GYm19YXCoul1ZW19Y3yptbrTTKEo81vciPko7rpMznIWsKLnzWiRPmBK7P2u7oRMbb9yxJeRReinHMuoEzDPmAe44g6mrcy2uT2/xi0itXrKqlljkLbA0q0KsRlV9wgz4ieMgQgCGEIOzDQUrPNWxYiInrIicuIcRVnGGCEmkzymKU4RA7ou+QdteaDWkvPVOl9ugUn96ElCb2SBNRXkJYnmaqeKacJfubd6485d3G9He1V0CswB2xf+mmmf/VyVoEBjhSNXCqKVaMrM7TLpnqiry5+aUqQQ4xcRL3KZ4Q9pRy2mdTaVJVu+yto+JvKlOycu/p3Azv8pY0YPvnOGdBq1a1rap9flCpH+tRF7GDXezTPA9RxykaaJJ3gEc84dk4M4SRG5PPVKOgNdv4toyHD0GPkwA=</latexit> <latexit sha1_base64="n1sH4BzpnCpJUU+cZTcSZN5qnqM=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoxpVUtA/RWpJ0WofmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69buzzVFjWa8GYm19YXCoul1ZW19Y3yptbrTTKEo81vciPko7rpMznIWsKLnzWiRPmBK7P2u7oRMbb9yxJeRReinHMuoEzDPmAe44g6mrcy2uT2/xi0itXrKqlljkLbA0q0KsRlV9wgz4ieMgQgCGEIOzDQUrPNWxYiInrIicuIcRVnGGCEmkzymKU4RA7ou+QdteaDWkvPVOl9ugUn96ElCb2SBNRXkJYnmaqeKacJfubd6485d3G9He1V0CswB2xf+mmmf/VyVoEBjhSNXCqKVaMrM7TLpnqiry5+aUqQQ4xcRL3KZ4Q9pRy2mdTaVJVu+yto+JvKlOycu/p3Azv8pY0YPvnOGdBq1a1rap9flCpH+tRF7GDXezTPA9RxykaaJJ3gEc84dk4M4SRG5PPVKOgNdv4toyHD0GPkwA=</latexit> yS 3 <latexit sha1_base64="YBzyFPDCtdngi/81pQ8hwLv1T6k=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl040oq2ofUWpLptA7mRTIRSujWH3Cr3yX+gf6Fd8YU1CI6IcmZc+85M/deN/JEIi3rtWDMzM7NLxQXS0vLK6tr5fWNZhKmMeMNFnph3HadhHsi4A0ppMfbUcwd3/V4y707UfHWPY8TEQaXchTxru8MAzEQzJFEXY162f74JrsY98oVq2rpZU4DOwcV5Ksell9wjT5CMKTwwRFAEvbgIKGnAxsWIuK6yIiLCQkd5xijRNqUsjhlOMTe0XdIu07OBrRXnolWMzrFozcmpYkd0oSUFxNWp5k6nmpnxf7mnWlPdbcR/d3cyydW4pbYv3STzP/qVC0SAxzpGgTVFGlGVcdyl1R3Rd3c/FKVJIeIOIX7FI8JM62c9NnUmkTXrnrr6PibzlSs2rM8N8W7uiUN2P45zmnQ3KvaVtU+P6jUjvNRF7GFbezSPA9RwynqaJC3j0c84dk4M6SRGePPVKOQazbxbRkPH0P0kwE=</latexit> <latexit sha1_base64="YBzyFPDCtdngi/81pQ8hwLv1T6k=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl040oq2ofUWpLptA7mRTIRSujWH3Cr3yX+gf6Fd8YU1CI6IcmZc+85M/deN/JEIi3rtWDMzM7NLxQXS0vLK6tr5fWNZhKmMeMNFnph3HadhHsi4A0ppMfbUcwd3/V4y707UfHWPY8TEQaXchTxru8MAzEQzJFEXY162f74JrsY98oVq2rpZU4DOwcV5Ksell9wjT5CMKTwwRFAEvbgIKGnAxsWIuK6yIiLCQkd5xijRNqUsjhlOMTe0XdIu07OBrRXnolWMzrFozcmpYkd0oSUFxNWp5k6nmpnxf7mnWlPdbcR/d3cyydW4pbYv3STzP/qVC0SAxzpGgTVFGlGVcdyl1R3Rd3c/FKVJIeIOIX7FI8JM62c9NnUmkTXrnrr6PibzlSs2rM8N8W7uiUN2P45zmnQ3KvaVtU+P6jUjvNRF7GFbezSPA9RwynqaJC3j0c84dk4M6SRGePPVKOQazbxbRkPH0P0kwE=</latexit> <latexit sha1_base64="YBzyFPDCtdngi/81pQ8hwLv1T6k=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl040oq2ofUWpLptA7mRTIRSujWH3Cr3yX+gf6Fd8YU1CI6IcmZc+85M/deN/JEIi3rtWDMzM7NLxQXS0vLK6tr5fWNZhKmMeMNFnph3HadhHsi4A0ppMfbUcwd3/V4y707UfHWPY8TEQaXchTxru8MAzEQzJFEXY162f74JrsY98oVq2rpZU4DOwcV5Ksell9wjT5CMKTwwRFAEvbgIKGnAxsWIuK6yIiLCQkd5xijRNqUsjhlOMTe0XdIu07OBrRXnolWMzrFozcmpYkd0oSUFxNWp5k6nmpnxf7mnWlPdbcR/d3cyydW4pbYv3STzP/qVC0SAxzpGgTVFGlGVcdyl1R3Rd3c/FKVJIeIOIX7FI8JM62c9NnUmkTXrnrr6PibzlSs2rM8N8W7uiUN2P45zmnQ3KvaVtU+P6jUjvNRF7GFbezSPA9RwynqaJC3j0c84dk4M6SRGePPVKOQazbxbRkPH0P0kwE=</latexit> <latexit sha1_base64="YBzyFPDCtdngi/81pQ8hwLv1T6k=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl040oq2ofUWpLptA7mRTIRSujWH3Cr3yX+gf6Fd8YU1CI6IcmZc+85M/deN/JEIi3rtWDMzM7NLxQXS0vLK6tr5fWNZhKmMeMNFnph3HadhHsi4A0ppMfbUcwd3/V4y707UfHWPY8TEQaXchTxru8MAzEQzJFEXY162f74JrsY98oVq2rpZU4DOwcV5Ksell9wjT5CMKTwwRFAEvbgIKGnAxsWIuK6yIiLCQkd5xijRNqUsjhlOMTe0XdIu07OBrRXnolWMzrFozcmpYkd0oSUFxNWp5k6nmpnxf7mnWlPdbcR/d3cyydW4pbYv3STzP/qVC0SAxzpGgTVFGlGVcdyl1R3Rd3c/FKVJIeIOIX7FI8JM62c9NnUmkTXrnrr6PibzlSs2rM8N8W7uiUN2P45zmnQ3KvaVtU+P6jUjvNRF7GFbezSPA9RwynqaJC3j0c84dk4M6SRGePPVKOQazbxbRkPH0P0kwE=</latexit> hS 1 <latexit sha1_base64="koGtsOnLJ5gNktNbiWo158+rn34=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl040oq2odoLUk6bYfmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69buzzVFjWa8GYm19YXCoul1ZW19Y3yptbzTTKEo81vMiPkrbrpMznIWsILnzWjhPmBK7PWu7oVMZb9yxJeRReiXHMOoEzCHmfe44g6nrYze3JXX456ZYrVtVSy5wFtgYV6FWPyi+4RQ8RPGQIwBBCEPbhIKXnBjYsxMR1kBOXEOIqzjBBibQZZTHKcIgd0XdAuxvNhrSXnqlSe3SKT29CShN7pIkoLyEsTzNVPFPOkv3NO1ee8m5j+rvaKyBWYEjsX7pp5n91shaBPo5VDZxqihUjq/O0S6a6Im9ufqlKkENMnMQ9iieEPaWc9tlUmlTVLnvrqPibypSs3Hs6N8O7vCUN2P45zlnQPKjaVtW+OKzUTvSoi9jBLvZpnkeo4Qx1NMg7wCOe8GycG8LIjclnqlHQmm18W8bDBxZCku4=</latexit> <latexit sha1_base64="koGtsOnLJ5gNktNbiWo158+rn34=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl040oq2odoLUk6bYfmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69buzzVFjWa8GYm19YXCoul1ZW19Y3yptbzTTKEo81vMiPkrbrpMznIWsILnzWjhPmBK7PWu7oVMZb9yxJeRReiXHMOoEzCHmfe44g6nrYze3JXX456ZYrVtVSy5wFtgYV6FWPyi+4RQ8RPGQIwBBCEPbhIKXnBjYsxMR1kBOXEOIqzjBBibQZZTHKcIgd0XdAuxvNhrSXnqlSe3SKT29CShN7pIkoLyEsTzNVPFPOkv3NO1ee8m5j+rvaKyBWYEjsX7pp5n91shaBPo5VDZxqihUjq/O0S6a6Im9ufqlKkENMnMQ9iieEPaWc9tlUmlTVLnvrqPibypSs3Hs6N8O7vCUN2P45zlnQPKjaVtW+OKzUTvSoi9jBLvZpnkeo4Qx1NMg7wCOe8GycG8LIjclnqlHQmm18W8bDBxZCku4=</latexit> <latexit sha1_base64="koGtsOnLJ5gNktNbiWo158+rn34=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl040oq2odoLUk6bYfmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69buzzVFjWa8GYm19YXCoul1ZW19Y3yptbzTTKEo81vMiPkrbrpMznIWsILnzWjhPmBK7PWu7oVMZb9yxJeRReiXHMOoEzCHmfe44g6nrYze3JXX456ZYrVtVSy5wFtgYV6FWPyi+4RQ8RPGQIwBBCEPbhIKXnBjYsxMR1kBOXEOIqzjBBibQZZTHKcIgd0XdAuxvNhrSXnqlSe3SKT29CShN7pIkoLyEsTzNVPFPOkv3NO1ee8m5j+rvaKyBWYEjsX7pp5n91shaBPo5VDZxqihUjq/O0S6a6Im9ufqlKkENMnMQ9iieEPaWc9tlUmlTVLnvrqPibypSs3Hs6N8O7vCUN2P45zlnQPKjaVtW+OKzUTvSoi9jBLvZpnkeo4Qx1NMg7wCOe8GycG8LIjclnqlHQmm18W8bDBxZCku4=</latexit> <latexit sha1_base64="koGtsOnLJ5gNktNbiWo158+rn34=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl040oq2odoLUk6bYfmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69buzzVFjWa8GYm19YXCoul1ZW19Y3yptbzTTKEo81vMiPkrbrpMznIWsILnzWjhPmBK7PWu7oVMZb9yxJeRReiXHMOoEzCHmfe44g6nrYze3JXX456ZYrVtVSy5wFtgYV6FWPyi+4RQ8RPGQIwBBCEPbhIKXnBjYsxMR1kBOXEOIqzjBBibQZZTHKcIgd0XdAuxvNhrSXnqlSe3SKT29CShN7pIkoLyEsTzNVPFPOkv3NO1ee8m5j+rvaKyBWYEjsX7pp5n91shaBPo5VDZxqihUjq/O0S6a6Im9ufqlKkENMnMQ9iieEPaWc9tlUmlTVLnvrqPibypSs3Hs6N8O7vCUN2P45zlnQPKjaVtW+OKzUTvSoi9jBLvZpnkeo4Qx1NMg7wCOe8GycG8LIjclnqlHQmm18W8bDBxZCku4=</latexit> hS 2 <latexit sha1_base64="3ALZUvutjwDg3w01ckmR7hv6LO0=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoxpVUtA+ptSTTaRuaF8lEKCFbf8Ctfpf4B/oX3hlTUIvohCRnzr3nzNx77dB1YmEYrwVtYXFpeaW4Wlpb39jcKm/vtOIgiRhvssANoo5txdx1fN4UjnB5J4y45dkub9uTMxlv3/ModgL/WkxD3vOske8MHWYJom7G/bSW3aVXWb9cMaqGWvo8MHNQQb4aQfkFtxggAEMCDxw+BGEXFmJ6ujBhICSuh5S4iJCj4hwZSqRNKItThkXshL4j2nVz1qe99IyVmtEpLr0RKXUckCagvIiwPE1X8UQ5S/Y371R5yrtN6W/nXh6xAmNi/9LNMv+rk7UIDHGianCoplAxsjqWuySqK/Lm+peqBDmExEk8oHhEmCnlrM+60sSqdtlbS8XfVKZk5Z7luQne5S1pwObPcc6DVq1qGlXz8qhSP81HXcQe9nFI8zxGHedooEneHh7xhGftQhNaqmWfqVoh1+zi29IePgAYp5Lv</latexit> <latexit sha1_base64="3ALZUvutjwDg3w01ckmR7hv6LO0=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoxpVUtA+ptSTTaRuaF8lEKCFbf8Ctfpf4B/oX3hlTUIvohCRnzr3nzNx77dB1YmEYrwVtYXFpeaW4Wlpb39jcKm/vtOIgiRhvssANoo5txdx1fN4UjnB5J4y45dkub9uTMxlv3/ModgL/WkxD3vOske8MHWYJom7G/bSW3aVXWb9cMaqGWvo8MHNQQb4aQfkFtxggAEMCDxw+BGEXFmJ6ujBhICSuh5S4iJCj4hwZSqRNKItThkXshL4j2nVz1qe99IyVmtEpLr0RKXUckCagvIiwPE1X8UQ5S/Y371R5yrtN6W/nXh6xAmNi/9LNMv+rk7UIDHGianCoplAxsjqWuySqK/Lm+peqBDmExEk8oHhEmCnlrM+60sSqdtlbS8XfVKZk5Z7luQne5S1pwObPcc6DVq1qGlXz8qhSP81HXcQe9nFI8zxGHedooEneHh7xhGftQhNaqmWfqVoh1+zi29IePgAYp5Lv</latexit> <latexit sha1_base64="3ALZUvutjwDg3w01ckmR7hv6LO0=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoxpVUtA+ptSTTaRuaF8lEKCFbf8Ctfpf4B/oX3hlTUIvohCRnzr3nzNx77dB1YmEYrwVtYXFpeaW4Wlpb39jcKm/vtOIgiRhvssANoo5txdx1fN4UjnB5J4y45dkub9uTMxlv3/ModgL/WkxD3vOske8MHWYJom7G/bSW3aVXWb9cMaqGWvo8MHNQQb4aQfkFtxggAEMCDxw+BGEXFmJ6ujBhICSuh5S4iJCj4hwZSqRNKItThkXshL4j2nVz1qe99IyVmtEpLr0RKXUckCagvIiwPE1X8UQ5S/Y371R5yrtN6W/nXh6xAmNi/9LNMv+rk7UIDHGianCoplAxsjqWuySqK/Lm+peqBDmExEk8oHhEmCnlrM+60sSqdtlbS8XfVKZk5Z7luQne5S1pwObPcc6DVq1qGlXz8qhSP81HXcQe9nFI8zxGHedooEneHh7xhGftQhNaqmWfqVoh1+zi29IePgAYp5Lv</latexit> <latexit sha1_base64="3ALZUvutjwDg3w01ckmR7hv6LO0=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoxpVUtA+ptSTTaRuaF8lEKCFbf8Ctfpf4B/oX3hlTUIvohCRnzr3nzNx77dB1YmEYrwVtYXFpeaW4Wlpb39jcKm/vtOIgiRhvssANoo5txdx1fN4UjnB5J4y45dkub9uTMxlv3/ModgL/WkxD3vOske8MHWYJom7G/bSW3aVXWb9cMaqGWvo8MHNQQb4aQfkFtxggAEMCDxw+BGEXFmJ6ujBhICSuh5S4iJCj4hwZSqRNKItThkXshL4j2nVz1qe99IyVmtEpLr0RKXUckCagvIiwPE1X8UQ5S/Y371R5yrtN6W/nXh6xAmNi/9LNMv+rk7UIDHGianCoplAxsjqWuySqK/Lm+peqBDmExEk8oHhEmCnlrM+60sSqdtlbS8XfVKZk5Z7luQne5S1pwObPcc6DVq1qGlXz8qhSP81HXcQe9nFI8zxGHedooEneHh7xhGftQhNaqmWfqVoh1+zi29IePgAYp5Lv</latexit> hS 3 <latexit sha1_base64="WDHlYMnkm+sSstpAtExanW9AleI=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl040oq2ofUWpJ02g7Ni2QilJCtP+BWv0v8A/0L74wpqEV0QpIz595zZu69dujyWBjGa0Gbm19YXCoul1ZW19Y3yptbzThIIoc1nMANorZtxczlPmsILlzWDiNmebbLWvb4TMZb9yyKeeBfi0nIup419PmAO5Yg6mbUSw+zu/Qq65UrRtVQS58FZg4qyFc9KL/gFn0EcJDAA4MPQdiFhZieDkwYCInrIiUuIsRVnCFDibQJZTHKsIgd03dIu07O+rSXnrFSO3SKS29ESh17pAkoLyIsT9NVPFHOkv3NO1We8m4T+tu5l0eswIjYv3TTzP/qZC0CA5yoGjjVFCpGVufkLonqiry5/qUqQQ4hcRL3KR4RdpRy2mddaWJVu+ytpeJvKlOycu/kuQne5S1pwObPcc6C5kHVNKrm5VGldpqPuogd7GKf5nmMGs5RR4O8PTziCc/ahSa0VMs+U7VCrtnGt6U9fAAbDJLw</latexit> <latexit sha1_base64="WDHlYMnkm+sSstpAtExanW9AleI=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl040oq2ofUWpJ02g7Ni2QilJCtP+BWv0v8A/0L74wpqEV0QpIz595zZu69dujyWBjGa0Gbm19YXCoul1ZW19Y3yptbzThIIoc1nMANorZtxczlPmsILlzWDiNmebbLWvb4TMZb9yyKeeBfi0nIup419PmAO5Yg6mbUSw+zu/Qq65UrRtVQS58FZg4qyFc9KL/gFn0EcJDAA4MPQdiFhZieDkwYCInrIiUuIsRVnCFDibQJZTHKsIgd03dIu07O+rSXnrFSO3SKS29ESh17pAkoLyIsT9NVPFHOkv3NO1We8m4T+tu5l0eswIjYv3TTzP/qZC0CA5yoGjjVFCpGVufkLonqiry5/qUqQQ4hcRL3KR4RdpRy2mddaWJVu+ytpeJvKlOycu/kuQne5S1pwObPcc6C5kHVNKrm5VGldpqPuogd7GKf5nmMGs5RR4O8PTziCc/ahSa0VMs+U7VCrtnGt6U9fAAbDJLw</latexit> <latexit sha1_base64="WDHlYMnkm+sSstpAtExanW9AleI=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl040oq2ofUWpJ02g7Ni2QilJCtP+BWv0v8A/0L74wpqEV0QpIz595zZu69dujyWBjGa0Gbm19YXCoul1ZW19Y3yptbzThIIoc1nMANorZtxczlPmsILlzWDiNmebbLWvb4TMZb9yyKeeBfi0nIup419PmAO5Yg6mbUSw+zu/Qq65UrRtVQS58FZg4qyFc9KL/gFn0EcJDAA4MPQdiFhZieDkwYCInrIiUuIsRVnCFDibQJZTHKsIgd03dIu07O+rSXnrFSO3SKS29ESh17pAkoLyIsT9NVPFHOkv3NO1We8m4T+tu5l0eswIjYv3TTzP/qZC0CA5yoGjjVFCpGVufkLonqiry5/qUqQQ4hcRL3KR4RdpRy2mddaWJVu+ytpeJvKlOycu/kuQne5S1pwObPcc6C5kHVNKrm5VGldpqPuogd7GKf5nmMGs5RR4O8PTziCc/ahSa0VMs+U7VCrtnGt6U9fAAbDJLw</latexit> <latexit sha1_base64="WDHlYMnkm+sSstpAtExanW9AleI=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl040oq2ofUWpJ02g7Ni2QilJCtP+BWv0v8A/0L74wpqEV0QpIz595zZu69dujyWBjGa0Gbm19YXCoul1ZW19Y3yptbzThIIoc1nMANorZtxczlPmsILlzWDiNmebbLWvb4TMZb9yyKeeBfi0nIup419PmAO5Yg6mbUSw+zu/Qq65UrRtVQS58FZg4qyFc9KL/gFn0EcJDAA4MPQdiFhZieDkwYCInrIiUuIsRVnCFDibQJZTHKsIgd03dIu07O+rSXnrFSO3SKS29ESh17pAkoLyIsT9NVPFHOkv3NO1We8m4T+tu5l0eswIjYv3TTzP/qZC0CA5yoGjjVFCpGVufkLonqiry5/qUqQQ4hcRL3KR4RdpRy2mddaWJVu+ytpeJvKlOycu/kuQne5S1pwObPcc6C5kHVNKrm5VGldpqPuogd7GKf5nmMGs5RR4O8PTziCc/ahSa0VMs+U7VCrtnGt6U9fAAbDJLw</latexit> oS 1 <latexit sha1_base64="8bMqy/XTwZZ49li6ATaPll+NUlo=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl040oq2ofUWpJ0WkOTTEgmQgnd+gNu9bvEP9C/8M44BbWITkhy5tx7zsy9140DPxWW9Vow5uYXFpeKy6WV1bX1jfLmVjPlWeKxhscDnrRdJ2WBH7GG8EXA2nHCnNANWMsdncp4654lqc+jKzGOWTd0hpE/8D1HEHXNe7k9uc0vJ71yxapaapmzwNagAr3qvPyCG/TB4SFDCIYIgnAAByk9HdiwEBPXRU5cQshXcYYJSqTNKItRhkPsiL5D2nU0G9FeeqZK7dEpAb0JKU3skYZTXkJYnmaqeKacJfubd6485d3G9He1V0iswB2xf+mmmf/VyVoEBjhWNfhUU6wYWZ2nXTLVFXlz80tVghxi4iTuUzwh7CnltM+m0qSqdtlbR8XfVKZk5d7TuRne5S1pwPbPcc6C5kHVtqr2xWGldqJHXcQOdrFP8zxCDWeoo0HeIR7xhGfj3BBGbkw+U42C1mzj2zIePgAnGpL1</latexit> <latexit sha1_base64="8bMqy/XTwZZ49li6ATaPll+NUlo=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl040oq2ofUWpJ0WkOTTEgmQgnd+gNu9bvEP9C/8M44BbWITkhy5tx7zsy9140DPxWW9Vow5uYXFpeKy6WV1bX1jfLmVjPlWeKxhscDnrRdJ2WBH7GG8EXA2nHCnNANWMsdncp4654lqc+jKzGOWTd0hpE/8D1HEHXNe7k9uc0vJ71yxapaapmzwNagAr3qvPyCG/TB4SFDCIYIgnAAByk9HdiwEBPXRU5cQshXcYYJSqTNKItRhkPsiL5D2nU0G9FeeqZK7dEpAb0JKU3skYZTXkJYnmaqeKacJfubd6485d3G9He1V0iswB2xf+mmmf/VyVoEBjhWNfhUU6wYWZ2nXTLVFXlz80tVghxi4iTuUzwh7CnltM+m0qSqdtlbR8XfVKZk5d7TuRne5S1pwPbPcc6C5kHVtqr2xWGldqJHXcQOdrFP8zxCDWeoo0HeIR7xhGfj3BBGbkw+U42C1mzj2zIePgAnGpL1</latexit> <latexit sha1_base64="8bMqy/XTwZZ49li6ATaPll+NUlo=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl040oq2ofUWpJ0WkOTTEgmQgnd+gNu9bvEP9C/8M44BbWITkhy5tx7zsy9140DPxWW9Vow5uYXFpeKy6WV1bX1jfLmVjPlWeKxhscDnrRdJ2WBH7GG8EXA2nHCnNANWMsdncp4654lqc+jKzGOWTd0hpE/8D1HEHXNe7k9uc0vJ71yxapaapmzwNagAr3qvPyCG/TB4SFDCIYIgnAAByk9HdiwEBPXRU5cQshXcYYJSqTNKItRhkPsiL5D2nU0G9FeeqZK7dEpAb0JKU3skYZTXkJYnmaqeKacJfubd6485d3G9He1V0iswB2xf+mmmf/VyVoEBjhWNfhUU6wYWZ2nXTLVFXlz80tVghxi4iTuUzwh7CnltM+m0qSqdtlbR8XfVKZk5d7TuRne5S1pwPbPcc6C5kHVtqr2xWGldqJHXcQOdrFP8zxCDWeoo0HeIR7xhGfj3BBGbkw+U42C1mzj2zIePgAnGpL1</latexit>
<latexit sha1_base64="8bMqy/XTwZZ49li6ATaPll+NUlo=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl040oq2ofUWpJ0WkOTTEgmQgnd+gNu9bvEP9C/8M44BbWITkhy5tx7zsy9140DPxWW9Vow5uYXFpeKy6WV1bX1jfLmVjPlWeKxhscDnrRdJ2WBH7GG8EXA2nHCnNANWMsdncp4654lqc+jKzGOWTd0hpE/8D1HEHXNe7k9uc0vJ71yxapaapmzwNagAr3qvPyCG/TB4SFDCIYIgnAAByk9HdiwEBPXRU5cQshXcYYJSqTNKItRhkPsiL5D2nU0G9FeeqZK7dEpAb0JKU3skYZTXkJYnmaqeKacJfubd6485d3G9He1V0iswB2xf+mmmf/VyVoEBjhWNfhUU6wYWZ2nXTLVFXlz80tVghxi4iTuUzwh7CnltM+m0qSqdtlbR8XfVKZk5d7TuRne5S1pwPbPcc6C5kHVtqr2xWGldqJHXcQOdrFP8zxCDWeoo0HeIR7xhGfj3BBGbkw+U42C1mzj2zIePgAnGpL1</latexit> o
S 2
<latexit sha1_base64="yza2LYCy4pl1iLmGZ9rXHT3DCMQ=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoxpVUtA/RWpLptIbmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69bux7qbCs14IxN7+wuFRcLq2srq1vlDe3WmmUJYw3WeRHScd1Uu57IW8KT/i8EyfcCVyft93RiYy373mSelF4KcYx7wbOMPQGHnMEUVdRL69NbvOLSa9csaqWWuYssDWoQK9GVH7BDfqIwJAhAEcIQdiHg5Sea9iwEBPXRU5cQshTcY4JSqTNKItThkPsiL5D2l1rNqS99EyVmtEpPr0JKU3skSaivISwPM1U8Uw5S/Y371x5yruN6e9qr4BYgTti/9JNM/+rk7UIDHCkavCoplgxsjqmXTLVFXlz80tVghxi4iTuUzwhzJRy2mdTaVJVu+yto+JvKlOycs90boZ3eUsasP1znLOgVavaVtU+P6jUj/Woi9jBLvZpnoeo4xQNNMk7wCOe8GycGcLIjclnqlHQmm18W8bDByl/kvY=</latexit><latexit sha1_base64="yza2LYCy4pl1iLmGZ9rXHT3DCMQ=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoxpVUtA/RWpLptIbmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69bux7qbCs14IxN7+wuFRcLq2srq1vlDe3WmmUJYw3WeRHScd1Uu57IW8KT/i8EyfcCVyft93RiYy373mSelF4KcYx7wbOMPQGHnMEUVdRL69NbvOLSa9csaqWWuYssDWoQK9GVH7BDfqIwJAhAEcIQdiHg5Sea9iwEBPXRU5cQshTcY4JSqTNKItThkPsiL5D2l1rNqS99EyVmtEpPr0JKU3skSaivISwPM1U8Uw5S/Y371x5yruN6e9qr4BYgTti/9JNM/+rk7UIDHCkavCoplgxsjqmXTLVFXlz80tVghxi4iTuUzwhzJRy2mdTaVJVu+yto+JvKlOycs90boZ3eUsasP1znLOgVavaVtU+P6jUj/Woi9jBLvZpnoeo4xQNNMk7wCOe8GycGcLIjclnqlHQmm18W8bDByl/kvY=</latexit>
<latexit sha1_base64="yza2LYCy4pl1iLmGZ9rXHT3DCMQ=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoxpVUtA/RWpLptIbmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69bux7qbCs14IxN7+wuFRcLq2srq1vlDe3WmmUJYw3WeRHScd1Uu57IW8KT/i8EyfcCVyft93RiYy373mSelF4KcYx7wbOMPQGHnMEUVdRL69NbvOLSa9csaqWWuYssDWoQK9GVH7BDfqIwJAhAEcIQdiHg5Sea9iwEBPXRU5cQshTcY4JSqTNKItThkPsiL5D2l1rNqS99EyVmtEpPr0JKU3skSaivISwPM1U8Uw5S/Y371x5yruN6e9qr4BYgTti/9JNM/+rk7UIDHCkavCoplgxsjqmXTLVFXlz80tVghxi4iTuUzwhzJRy2mdTaVJVu+yto+JvKlOycs90boZ3eUsasP1znLOgVavaVtU+P6jUj/Woi9jBLvZpnoeo4xQNNMk7wCOe8GycGcLIjclnqlHQmm18W8bDByl/kvY=</latexit>
<latexit sha1_base64="yza2LYCy4pl1iLmGZ9rXHT3DCMQ=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLoxpVUtA/RWpLptIbmRTIRSujWH3Cr3yX+gf6Fd8YpqEV0QpIz595zZu69bux7qbCs14IxN7+wuFRcLq2srq1vlDe3WmmUJYw3WeRHScd1Uu57IW8KT/i8EyfcCVyft93RiYy373mSelF4KcYx7wbOMPQGHnMEUVdRL69NbvOLSa9csaqWWuYssDWoQK9GVH7BDfqIwJAhAEcIQdiHg5Sea9iwEBPXRU5cQshTcY4JSqTNKItThkPsiL5D2l1rNqS99EyVmtEpPr0JKU3skSaivISwPM1U8Uw5S/Y371x5yruN6e9qr4BYgTti/9JNM/+rk7UIDHCkavCoplgxsjqmXTLVFXlz80tVghxi4iTuUzwhzJRy2mdTaVJVu+yto+JvKlOycs90boZ3eUsasP1znLOgVavaVtU+P6jUj/Woi9jBLvZpnoeo4xQNNMk7wCOe8GycGcLIjclnqlHQmm18W8bDByl/kvY=</latexit> o
[image:3.595.74.281.60.264.2]S 3 <latexit sha1_base64="05BLDbvFUmjoMdr8QHxP2FEZZGc=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl040oq2ofUWpLptA7mRTIRSujWH3Cr3yX+gf6Fd8YU1CI6IcmZc+85M/deN/JEIi3rtWDMzM7NLxQXS0vLK6tr5fWNZhKmMeMNFnph3HadhHsi4A0ppMfbUcwd3/V4y707UfHWPY8TEQaXchTxru8MAzEQzJFEXYW9bH98k12Me+WKVbX0MqeBnYMK8lUPyy+4Rh8hGFL44AggCXtwkNDTgQ0LEXFdZMTFhISOc4xRIm1KWZwyHGLv6DukXSdnA9orz0SrGZ3i0RuT0sQOaULKiwmr00wdT7WzYn/zzrSnutuI/m7u5RMrcUvsX7pJ5n91qhaJAY50DYJqijSjqmO5S6q7om5ufqlKkkNEnMJ9iseEmVZO+mxqTaJrV711dPxNZypW7Vmem+Jd3ZIGbP8c5zRo7lVtq2qfH1Rqx/moi9jCNnZpnoeo4RR1NMjbxyOe8GycGdLIjPFnqlHINZv4toyHDyvkkvc=</latexit> <latexit sha1_base64="05BLDbvFUmjoMdr8QHxP2FEZZGc=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl040oq2ofUWpLptA7mRTIRSujWH3Cr3yX+gf6Fd8YU1CI6IcmZc+85M/deN/JEIi3rtWDMzM7NLxQXS0vLK6tr5fWNZhKmMeMNFnph3HadhHsi4A0ppMfbUcwd3/V4y707UfHWPY8TEQaXchTxru8MAzEQzJFEXYW9bH98k12Me+WKVbX0MqeBnYMK8lUPyy+4Rh8hGFL44AggCXtwkNDTgQ0LEXFdZMTFhISOc4xRIm1KWZwyHGLv6DukXSdnA9orz0SrGZ3i0RuT0sQOaULKiwmr00wdT7WzYn/zzrSnutuI/m7u5RMrcUvsX7pJ5n91qhaJAY50DYJqijSjqmO5S6q7om5ufqlKkkNEnMJ9iseEmVZO+mxqTaJrV711dPxNZypW7Vmem+Jd3ZIGbP8c5zRo7lVtq2qfH1Rqx/moi9jCNnZpnoeo4RR1NMjbxyOe8GycGdLIjPFnqlHINZv4toyHDyvkkvc=</latexit> <latexit sha1_base64="05BLDbvFUmjoMdr8QHxP2FEZZGc=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl040oq2ofUWpLptA7mRTIRSujWH3Cr3yX+gf6Fd8YU1CI6IcmZc+85M/deN/JEIi3rtWDMzM7NLxQXS0vLK6tr5fWNZhKmMeMNFnph3HadhHsi4A0ppMfbUcwd3/V4y707UfHWPY8TEQaXchTxru8MAzEQzJFEXYW9bH98k12Me+WKVbX0MqeBnYMK8lUPyy+4Rh8hGFL44AggCXtwkNDTgQ0LEXFdZMTFhISOc4xRIm1KWZwyHGLv6DukXSdnA9orz0SrGZ3i0RuT0sQOaULKiwmr00wdT7WzYn/zzrSnutuI/m7u5RMrcUvsX7pJ5n91qhaJAY50DYJqijSjqmO5S6q7om5ufqlKkkNEnMJ9iseEmVZO+mxqTaJrV711dPxNZypW7Vmem+Jd3ZIGbP8c5zRo7lVtq2qfH1Rqx/moi9jCNnZpnoeo4RR1NMjbxyOe8GycGdLIjPFnqlHINZv4toyHDyvkkvc=</latexit> <latexit sha1_base64="05BLDbvFUmjoMdr8QHxP2FEZZGc=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkVwVRIVdFl040oq2ofUWpLptA7mRTIRSujWH3Cr3yX+gf6Fd8YU1CI6IcmZc+85M/deN/JEIi3rtWDMzM7NLxQXS0vLK6tr5fWNZhKmMeMNFnph3HadhHsi4A0ppMfbUcwd3/V4y707UfHWPY8TEQaXchTxru8MAzEQzJFEXYW9bH98k12Me+WKVbX0MqeBnYMK8lUPyy+4Rh8hGFL44AggCXtwkNDTgQ0LEXFdZMTFhISOc4xRIm1KWZwyHGLv6DukXSdnA9orz0SrGZ3i0RuT0sQOaULKiwmr00wdT7WzYn/zzrSnutuI/m7u5RMrcUvsX7pJ5n91qhaJAY50DYJqijSjqmO5S6q7om5ufqlKkkNEnMJ9iseEmVZO+mxqTaJrV711dPxNZypW7Vmem+Jd3ZIGbP8c5zRo7lVtq2qfH1Rqx/moi9jCNnZpnoeo4RR1NMjbxyOe8GycGdLIjPFnqlHINZv4toyHDyvkkvc=</latexit>
Figure 2: Illustration of our Stack-Propagation frame-work for joint intent detection and slot filling. It con-sists of one shared self-attentive encoder and two de-coders. The output distribution of intent detection net-work and the representations from encoder are concate-nated as the input for slot filling.
3 Approach
In this section, we will describe our
Stack-Propagation framework for SLU task. The
ar-chitecture of our framework is demonstrated in
Figure 2, which consists of an encoder and two
decoders. First, the encoder module uses one
shared self-attentive encoder to represent an utter-ance, which can grasp the shared knowledge be-tween two tasks. Then, the intent-detection de-coder performs a token-level intent detection. Fi-nally, our Stack-Propagation framework leverages the explicit token-level intent information for slot filling by concatenating the output of intent detec-tion decoder and the representadetec-tions from encoder as the input for slot filling decoder. Both intent detection and slot filling are optimized simultane-ously via a joint learning scheme.
3.1 Self-Attentive Encoder
In our Stack-Propagation framework, intent detec-tion task and slot filling task share one encoder, In the self-attentive encoder, we use BiLSTM with self-attention mechanism to leverage both advan-tages of temporal features and contextual informa-tion, which are useful for sequence labeling tasks
(Zhong et al.,2018;Yin et al.,2018).
The BiLSTM (Hochreiter and Schmidhuber,
1997) reads input utterance X = (x1,x2, ..,xT)
(T is the number of tokens in the input utterance)
forwardly and backwardly to produce
context-sensitive hidden states H = (h1,h2, ...,hT)
by repeatedly applying the recurrence hi =
BiLSTM φemb(x
i),hi−1
.
Self-attention is a very effective method of leveraging context-aware features over variable-length sequences for natural language processing
tasks (Tan et al.,2018;Zhong et al.,2018). In our
case, we use self-attention mechanism to capture the contextual information for each token. In this
paper, we adoptVaswani et al. (2017), where we
first map the matrix of input vectorsX∈RT×d(d
represents the mapped dimension) to queries (Q), keys (K) and values (V) matrices by using differ-ent linear projections and the self-attdiffer-ention output
C∈RT×dis a weighted sum of values:
C= softmax
QK> √ dk V. (1)
After obtaining the output of self-attention and BiLSTM. We concatenate these two representa-tions as the final encoding representation:
E=H⊕C, (2)
whereE∈RT×2d and⊕is concatenation
opera-tion.
3.2 Token-Level Intent Detection Decoder
In our framework, we perform a token-level tent detection, which can provide token-level in-tent features for slot filling, different from regard-ing the intent detection task as the sentence-level
classification problem (Liu and Lane,2016). The
token-level intent detection method can be formal-ized as a sequence labeling problem that maps a
input word sequencex=(x1, ..., xT)to sequence
of intent labeloI = (oI
1, ..., oIT). In training time,
we set the sentence’s intent label as every token’s gold intent label. The final intent of an utterance is computed by voting from predictions at each token of the utterance.
The self-attentive encoder generates a sequence
of contextual representationsE= (e1, ...,eT)and
each token can grasp the whole contextual infor-mation by self-attention mechanism. We use a uni-directional LSTM as the intent detection network.
At each decoding step i, the decoder state hI
i is
calculated by previous decoder statehIi−1, the
pre-vious emitted intent label distributionyIi−1and the
aligned encoder hidden stateei:
hIi =f hIi−1,yiI−1,ei
Then the decoder statehIi is utilized for intent de-tection:
yIi = softmax WIhhIi
, (4)
oIi = argmax(yIi), (5)
where yiI is the intent output distribution of the
ith token in the utterance;oI
i represents the intent
lable ofith token andWIhare trainable parameters
of the model.
The final utterance resultoIis generated by
vot-ing from all token intent results:
oI = argmax
m
X
i=1
nI
X
j=1
αj1[oIi =j], (6)
where m is the length of utterance andnI is the
number of intent labels;αj denotes a 0-1 vectorα
∈RnI of which thejth unit is one and the others
are zero;argmaxindicates the operation returning
the indices of the maximum values inα.
By performing the token-level intent detection, there are mainly two advantages:
1. Performing the token-level intent detection can provide features at each token to slot filling in our Stack-Propagation framework, which can ease the error propagation and retain more useful infor-mation for slot filling. Compared with
sentence-level intent detection (Li et al., 2018), if the
in-tent of the whole sentence is predicted wrongly, the wrong intent would possibly apply a negative impact on all slots. However, in token-level intent detection, if some tokens in the utterance predicted wrongly, other correct token-level intent informa-tion will still be useful for the corresponding slot filling.
2. Since each token can grasp the whole ut-terance contextual information by using the self-attentive encoder, we can consider predictions at each token in an utterance as individual predic-tion to the intent of this utterance. And there-fore, like ensemble neural networks, this approach will reduce the predicted variance and improve the performance of intent detection. The experiments section empirically demonstrate the effectiveness of the token-level intent detection.
3.3 Stack-propagation for Slot Filling
In this paper, one of the advantages of our Stack-Propagation framework is directly leveraging the explicit intent information to constrain the slots into a specific intent and alleviate the burden of
slot filling decoder. In our framework, we com-pose the input units for slot filling decoder by
con-catenating the intent output distributionyIi and the
aligned encoder hidden stateei.
For the slot-filling decoder, we similarly use an-other unidirectional LSTM as the slot-filling
de-coder. At the decoding stepi, the decoder statehSi
can be formalized as:
hSi =f hSi−1,ySi−1,yIi ⊕ei
, (7)
wherehS
i−1 is the previous decoder state;ySi−1 is
the previous emitted slot label distribution.
Similarily, the decoder state hIi is utilized for
slot filling:
yiS = softmax WhShSi
, (8)
oSi = argmax(ySi), (9)
where oS
i is the slot label of the ith word in the
utterance.
3.4 Joint Training
Another major difference between existing joint
work (Zhang and Wang, 2016;Goo et al., 2018)
and our framework is the training method for in-tent detection, where we convert the sentence-level classification task into token-sentence-level prediction to directly leverage token-level intent information for slot filling. And the intent detection objection is formulated as:
L1,−
m
X
j=1
nI
X
i=1 ˆ
yi,Ij logyji,I. (10)
Similarly, the slot filling task objection is defined as:
L2 ,−
m
X
j=1
nS
X
i=1 ˆ
yi,Sj logyi,Sj , (11)
whereyˆji,I andyˆi,Sj are the gold intent label and
gold slot label separately;nSis the number of slot
labels.
To obtain both slot filling and intent detection jointly, the final joint objective is formulated as
Lθ =L1+L2. (12)
Through the joint loss function, the shared rep-resentations learned by the shared self-attentive encoder can consider two tasks jointly and further ease the error propagation compared with pipeline
4 Experiments
4.1 Experimental Settings
To evaluate the efficiency of our proposed model, we conduct experiments on two bench-mark datasets. One is the publicly ATIS dataset
(Hemphill et al., 1990) containing audio
record-ings of flight reservations, and the other is the custom-intent-engines collected by Snips (SNIPS
dataset) (Coucke et al., 2018). 1 Both datasets
used in our paper follows the same format and
par-tition as inGoo et al.(2018). The dimensionalities
of the word embedding is 256 for ATIS dataset and 512 for SNIPS dataset. The self-attentive encoder hidden units are set as 256. L2 regularization is
used on our model is 1 ×10−6 and dropout
ra-tio is adopted is 0.4 for reducing overfit. We use
Adam (Kingma and Ba,2014) to optimize the
pa-rameters in our model and adopted the suggested hyper-parameters for optimization. For all the ex-periments, we select the model which works the best on the dev set, and then evaluate it on the test set.
4.2 Baselines
We compare our model with the existing baselines including:
• Joint Seq. Hakkani-T¨ur et al. (2016)
pro-posed a multi-task modeling approach for jointly modeling domain detection, intent de-tection, and slot filling in a single recurrent neural network (RNN) architecture.
• Attention BiRNN. Liu and Lane (2016)
leveraged the attention mechanism to allow the network to learn the relationship between slot and intent.
• Slot-Gated Atten. Goo et al. (2018)
pro-posed the slot-gated joint model to explore the correlation of slot filling and intent detec-tion better.
• Self-Attentive Model. Li et al. (2018)
pro-posed a novel self-attentive model with the intent augmented gate mechanism to utilize the semantic correlation between slot and in-tent.
1https://github.com/snipsco/
nlu-benchmark/tree/master/ 2017-06-custom-intent-engines
• Bi-Model. Wang et al. (2018) proposed the
Bi-model to consider the intent and slot fill-ing cross-impact to each other.
• CAPSULE-NLU. Zhang et al. (2019)
proposed a capsule-based neural network model with a dynamic routing-by-agreement schema to accomplish slot filling and intent detection.
• SF-ID Network. (E et al.,2019) introduced
an SF-ID network to establish direct connec-tions for the slot filling and intent detection to help them promote each other mutually.
For the Joint Seq, Attention BiRNN, Slot-gated
Atten, CAPSULE-NLU and SF-ID Network, we
adopt the reported results fromGoo et al.(2018);
Zhang et al.(2019);E et al.(2019). For the
Self-Attentive Model, Bi-Model, we re-implemented
the models and obtained the results on the same
datasets.2
4.3 Overall Results
FollowingGoo et al.(2018), we evaluate the SLU
performance of slot filling using F1 score and the performance of intent prediction using accuracy, and sentence-level semantic frame parsing using
overall accuracy. Table2shows the experiment
re-sults of the proposed models on SNIPS and ATIS datasets.
From the table, we can see that our model sig-nificantly outperforms all the baselines by a large margin and achieves the state-of-the-art perfor-mance. In the SNIPS dataset, compared with the
best prior joint workBi-Model, we achieve 0.7%
improvement on Slot (F1) score, 0.8% improve-ment on Intent (Acc) and 3.1% improveimprove-ment on Overall (Acc). In the ATIS dataset, we achieve 0.4% improvement on Slot (F1) score, 0.5% im-provement on Intent (Acc) and 0.8% imim-provement
on Overall (Acc). This indicates the
effective-ness of our Stack-Propagation framework. Espe-cially, our framework gains the largest improve-ments on sentence-level semantic frame accuracy,
2
Model SNIPS ATIS
Slot (F1) Intent (Acc) Overall (Acc) Slot (F1) Intent (Acc) Overall (Acc) Joint Seq (Hakkani-T¨ur et al.,2016) 87.3 96.9 73.2 94.3 92.6 80.7 Attention BiRNN (Liu and Lane,2016) 87.8 96.7 74.1 94.2 91.1 78.9 Slot-Gated Full Atten (Goo et al.,2018) 88.8 97.0 75.5 94.8 93.6 82.2 Slot-Gated Intent Atten (Goo et al.,2018) 88.3 96.8 74.6 95.2 94.1 82.6 Self-Attentive Model (Li et al.,2018) 90.0 97.5 81.0 95.1 96.8 82.2 Bi-Model (Wang et al.,2018) 93.5 97.2 83.8 95.5 96.4 85.7 CAPSULE-NLU (Zhang et al.,2019) 91.8 97.3 80.9 95.2 95.0 83.4 SF-ID Network (E et al.,2019) 90.5 97.0 78.4 95.6 96.6 86.0
Our model 94.2* 98.0* 86.9* 95.9* 96.9* 86.5*
Oracle (Intent) 96.1 - - 96.0 -
-Table 2: Slot filling and intent detection results on two datasets. The numbers with * indicate that the improvement
of our model over all baselines is statistically significant withp <0.05under t-test.
we attribute this to the fact that our framework di-rectly take the explicit intent information into con-sideration can better help grasp the relationship between the intent and slots and improve the SLU performance.
To see the role of intent information for SLU tasks intuitively, we also present the result when
using the gold intent information.3 The result is
shown in theoraclerow of Table2. From the
re-sult, we can see that further leveraging better in-tent information will lead to better slot filling per-formance. The result also verifies our assumptions that intent information can be used for guiding the slots prediction.
4.4 Analysis
In Section 4.3, significant improvements among all three metrics have been witnessed on both two publicly datasets. However, we would like
to know the reason for the improvement. In
this section, we first explore the effect of Stack-Propagation framework. Next, we study the ef-fect of our proposed token-level intent detection mechanism. Finally, we study the effect of self-attention mechanism in our framework.
4.4.1 Effect of Stack-Propagation
Framework
To verify the effectiveness of the
Stack-Propagation framework. we conduct experiments with the following ablations:
3
During the inference time, we concatenate the gold in-tent label distribution (one-hot vector) and the aligned en-coder hidden stateei as the composed input for slot filling
decoder. To keep the train and test procedure as the same, we replace our intent distribution as one-hot intent information for slot filling when train our model in our oracle experiments setting.
1) We conduct experiments to incorporate in-tent information by using gate-mechanism which
is similar to Goo et al.(2018), providing the
in-tent information by interacting with the slot filling
decoder by gate function.4 We refer it as
gate-mechanism.
2) We conduct experiments on the pipelined model where the intent detection and slot fill-ing has their own self-attentive encoder separately. The other model components keep the same as our
framework. We name it aspipelined model.
Table 3 gives the result of the comparison
ex-periment. From the result ofgate-mechanismrow,
we can observe that without the Stack-Propagation learning and simply using the gate-mechanism to incorporate the intent information, the slot fill-ing (F1) performance drops significantly, which demonstrates that directly leverage the intent in-formation with Stack-Propagation can improve the slot filling performance effectively than using the gate mechanism. Besides, we can see that the in-tent detection (Acc) and overall accuracy (Acc) decrease a lot. We attribute it to the fact that the bad slot filling performance harms the intent de-tection and the whole sentence semantic perfor-mance due to the joint learning scheme.
Besides, from the pipeline model row of
Ta-ble 3, we can see that without shared encoder,
the performance on all metrics declines signifi-cantly. This shows that Stack-Propagation model can learn the correlation knowledge which may promote each other and ease the error propagation effectively.
[image:6.595.72.522.61.193.2]
