Distributed Representations

Another approach to obtain distributional representations in VSMs is the distributed representation (Hinton et al., 1986). In a distributed representation, each word in the vocabulary set is associated with a low dimensional vector f : Σ → Rd where the

“meaning” of the word is distributed over many dimensions and captured by different dimensions of the vector. This means that a combination of several dimensions may capture a given aspect of meaning and that each dimension may contribute to capturing

2.1 Representation in Computational Linguistics 19 several aspects of meaning (Goldberg, 2017, p. 122). Thus, each dimension is not interpretable separately as opposed to the above-mentioned representations, such as the word–context matrix in which each dimension corresponds to a specific context the word occurs in.

Distributed representation models are based on the distributional hypothesis, as they try to capture the similarity between the words occurring in similar contexts (Goldberg, 2017). Levy and Goldberg (2014) show that distributed representations are deeply connected to distributional models and current distributed representation models use distributional signals to learn their representations.

Distributed representations are obtained by machine learning techniques, referred to as embeddings. More specifically, utilizing neural networks, low-dimensional vector representations for linguistic units are produced. These representations can be sub- divided intotask-specific representations andtask-agnostic representations. Task-specific representations are trained on a specific NLP task (e.g., sentiment analysis) to obtain specific information. The definitions are made clear throughout the following section.

In machine learning techniques, a model learns from training samples with respect to performance measureP and some taskT, if its performance as measured byP improves

with more training samples. We can reduce the problem of improving performance P at

task T to the problem of learning a particular target (objective) function f∗. Therefore, the type of knowledge that the model will learn is formulated as an objective function. Adjustable model parameters θare defined to achieve the best approximation f of the target function f∗ in the given taskT (Mitchell, 1997, p. 2).

In supervised learning tasks in machine learning, labeled training datasets are used to train the model (i.e., a set of ninput–output pairs{(x₁, y₁), . . . ,(xn, yn)}). The outputs

(or target values) are predetermined and usually obtained by human annotations. The objective is to train the model parameters θ, i.e., trainable weights, using the labeled dataset that result in the best approximation of the target values f(xi;θ) ≈ yi for

1≤i≤n.

A widely popular method to obtain distributed word representations is word2vec introduced by Mikolov et al. (2013a,b). In the following, we first introduce neural networks and then word2vec.

Neural Networks

Neural Networks (NNs), originally inspired by the biological neural networks, have received much attention in artificial intelligence, machine learning, and computational linguistics. Neural networks provide a class of machine learning methods for different problems “to approximate real-valued, discrete-valued and vector-valued target func- tions” (Mitchell, 1997, p. 81). A neural network connects several layers of neurons of which the first is the input, the last is the output, and in the middle there is a number of user-definable hidden layers. Each neuron is a computational unit that has inputs and outputs. Neurons of different layers are connected with connecting weights. These weights are adjustable parameters that are tuned to closely approximate the desired target function, given input–output data to the network.

There are several types of neural network architectures from Recurrent Neural Net- works (RNNs) to Convolutional Neural Networks (CNNs), the simplest of which being

feedforward NNs (Rosenblatt, 1957; Rumelhart et al., 1986).

Feedforward Neural Networks:

The reason these networks are called feedforward is that the information from the input layer is forward-propagated through the hidden layers to the output layer, and there is no recurrence during the feed-forward process. That is, there are no loops between neurons in different layers. The main goal in a feedforward NN is to approximate the target function f∗ which maps an input xi to a target value yi, given a training set of n input–output pairs {(x₁, y₁), . . . ,(xn, yn)}. A feedforward NN defines a mapping functiony =f(x;θ) and learns the value of the adjustable parameters θ, i.e., the weight matricesW connecting the layers and the bias vectors bfor each layer, that result in the

best function approximation (Goodfellow et al., 2016). Given a large set of input–output data as the training set, the goal is to generalize the function f to best approximate the output of previously unseen input data.

Each neuron in network layers takes input from neurons of previous layers and computes its activation value. Therefore, each layer’s output is the subsequent layer’s input. Fig. 2.4 illustrates an example of a two-layer fully connected feedforward NN with two input neurons, two neurons on the single hidden layer and one output neuron. W1 is a 2×2 weight matrix andW2 is a 2×1 weight matrix connecting input to hidden

and hidden to output layer, respectively.

b1 g g g x1 x2 o Input

Layer HiddenLayer OutputLayer

W₂1×2 W22×1

Figure 2.4: Example of a fully connected feedforward neural network with input x=

(x₁, x₂), outputo, bias b1, and one hidden layer. An activation function g is applied to its input argument at each layer.

Given a set of training input–output data, the objective is to approximate the tar- get function f∗ producing the output value by training the weight matrices and bias vectors (parameters θ) and optimizing the performance of the neural network in the approximation.

The output valueo of a given input x∈_Rdin is predicted by a network of_L layers

using the following computation:

o=gL−1(· · ·g2(g1(xW1+b1)W2+b2)· · ·WL−1+bL−1)WL,

whereWl ∈_Rdl−1×dl shows the weight matrix connecting layer_l−1 with dimension_d

l−₁

to layer l with dimension dl, and each gl is a differentiable activation function, such as the sigmoid function that is applied element-wise to its argument. bl ∈ Rdl is a bias

2.1 Representation in Computational Linguistics 21 vector at each layerl. In our example, given two inputsx₁ andx₂ shown asx= (x₁, x₂), the output is computed as follows:

h=g(xW1+b1), o=hW2.

To best approximate the target values, an optimizer and an objective functionE (also called loss function (Goldberg, 2017)) are defined to train the parameters of the network. The objective functionE at any point of training is a function of the difference between approximationomade by the network and the target value yit is trying to reach. There are different objective functions. A common one is the Mean Squared Error (MSE) defined as follows: E = 1 n n X i=1 (yi−oi)2,

whereoi is the output value that is compared with the target value yi, andnis the size of training set.

To train the weight matrices, different algorithms, such as backpropagation, are applied. Backpropagation is one of the most popular methods in NNs that minimizes the objective function by optimizing the weights using an optimizer, such as gradient descent (Cauchy, 1847; Robbins and Monro, 1951; Berstsekas, 1999). Gradient descent is an iterative optimization algorithm, which is applied to machine learning problems. In gradient descent, the goal is to find the local minimum/maximum (i.e., local optimum) of an objective function by taking steps proportional to the negative/positive gradient of the function at the current point toward the local optimum. In a NN, it computes the gradient of the objective function E with respect to each element in the weight matrix Wl of layerl for a set of ninput–output data:

∂E ∂Wl_{(j, k)} = n X i=1 ∂E ∂oi × ∂oi ∂Wl_{(j, k)}.

In our example, the gradient of the objective function E with respect to each weight element is computed using partial derivatives of the functions computed in the forward propagation: ∂E ∂W2(j, k) = n X i=1 ∂E ∂oi × ∂oi ∂W2(j, k), ∂E ∂W1(j, k) = n X i=1 ∂E ∂oi × ∂oi ∂W1(j, k), wherej andkrange from 1 to 2, and ∂oi

∂W2_(j,k) and

∂oi

∂W1_(j,k) are computed using the chain

rule as follows: ∂oi ∂W2(j, k) = ∂oi ∂g × ∂g ∂W2(j, k), ∂oi ∂W1(j, k) = ∂oi ∂g × ∂g ∂h × ∂h ∂W1(j, k).

function as follows:

Wl(j, k) =Wl(j, k)−η ∂E ∂Wl(_{j, k}),

whereηis a constant calledlearning ratethat controls how much the step size is adapted towards the local minimum of the objective function. This way, the error propagates backward in the network and all weights of the layers are updated. The forward and backward propagations are repeated many times (i.e., in many training steps), until there is no significant decrease in the error computation (i.e., the amount of the decrease is less than a threshold value ϵ). Thus, the weights are optimized and the network reached its best approximation of input–output pairs from training set. The network can then be used to predict the output of previously unseen input data.

Recurrent Neural Networks (RNNs):

RNNs (Rumelhart et al., 1986; Elman, 1990) allow to operate on inputs and outputs as sequences. The input data x to RNNs is an arbitrary length sequence x₁,x₂, . . . ,xT with xt ∈ Rdin (fixed length vector) where at each time step t the input xt is fed to the network. RNNs have a memory that remembers information from all previous time steps in a hidden state (also called hidden layer). At each time step, the previous hidden layerht−₁ is fed to the network at the current time step and is combined with input xt to compute the current hidden layer as follows:

ht=g(xtU+ht−₁W +bt),

whereU andW are shared weight matrices for input-to-hidden and hidden-to-hidden layers, respectively, andbtis the bias of the network. gis a nonlinear activation function, such as the tanhfunction. This way, historical information about the input sequence is kept in the hidden layers. The output y_t at each time step is a vector y_t∈_Rdout, which

is computed as follows:

y_t=g(htV +c),

whereV is a shared weight matrix for the hidden-to-output layer. c is a bias and g is either a linear or a nonlinear activation function. In general, the output at the final time step T (i.e., y_T), is used to approximate the prediction of the network for a given

sequence of inputs.

To train the network, training data is fed to the network and the objective function is computed. Then, the error is back-propagated through the network to update all weight matrices. We are not concerned with the detail of RNNs in this thesis and the interested reader is referred to the Deep Learning book by Goodfellow et al. (2016) for details.