Matrix Factorization

In this section, we describe the matrix factorization relation extraction model by Riedel et al. [2013], which is an instance of a simple neural link prediction model. We discuss this model in detail, as it the basis on which we develop rule injection methods in Chapter 3 and 4.

Assume a set of observed entity pair symbols _{C and a set of predicate sym-} bols _{P, which can either represent structured binary relations from Freebase, a} large collaborative knowledge base [Bollacker et al., 2008], or unstructured Open Information Extraction (OpenIE) [Etzioni et al., 2008] textual surface patterns col- lected from news articles. Examples for structured and unstructured relations are company/founders and #2-co-founder-of-#1, respectively. Here, #2-co-founder-of-#1 is a textual pattern where #1 and #2 are placehold- ers for entities. For instance, the relationship between Elon Musk and Tesla in the sentence “Elon Musk, the co-founder of Tesla and the CEO of SpaceX, cites The Foundation Trilogy by Isaac Asimov as a major influence on his think- ing”9could be expressed by the ground atom #2-co-founder-of-#1(TESLA,

ELON MUSK). In this example, ELON MUSK appeared first in the textual pattern, but #2 indicates that this constant will be used as the second argument in the predicate corresponding to the pattern. That way we can later introduce a rule

∀X, Y : company/founders(X, Y) :– #2-co-founder-of-#1(X, Y) with- out changing the order of the variables in the body of the rule.

Let_{O = {r}s(ei, ej))} be the set of observed ground atoms. Model F by Riedel

et al. [2013] maps all symbols in a knowledge base to subsymbolic representations, i.e., it learns a dense k-dimensional vector representation for every relation and entity pair. Thus, a training factrs(ei, ej) is represented by two vectors, vs ∈ Rk

and vij ∈ Rk, respectively. We will refer to these as embeddings, subsymbolic

representations, vector representations, neural representations, or simply (symbol) representations when it is clear from the context. The truth estimate of a fact is modeled via the sigmoid of the dot product of the two symbol representations:

psij = σ(vs>vij). (2.6)

In fact, this expression corresponds to the computation graph shown in Fig. 2.1a withx = vsandy = vij. The scorepsij is measuring the compatibility between the

relation and entity pair representation and can be interpreted as the probability of the fact being true conditioned on parameters of the model. We would like to train symbol representations such that true ground atoms get a score close to one and false ground atoms a score close to zero. This results in a low-rank matrix factorization corresponding to Generalized Principal Component Analysis (GPCA) [Collins et al., 2001]

K _{≈ σ(P C}>) _{∈ R}|P|×|C| (2.7)

where P _{∈ R}|P|×k _{and C ∈ R}|C|×k _{are matrices of all relation and entity pair}

representations, respectively. This factorization is depicted in Fig. 2.6 where known facts are shown in green and the task is to complete this matrix for cells with a question mark. Equation 2.7 leads to generalization with respect to unseen facts as every relation and entity pair is represented in a low-dimensional space, and this information bottleneck will lead to similar entity pairs being represented close in distance in the vector space (likewise for similar predicates).

The distributional hypothesis states that “one shall know a word by the company it keeps” [Firth, 1957]. It has been used for learning word meaning from large collections of text [Lowe and McDonald, 2000, Pad´o and Lapata, 2007]. Applied to automated KB completion, one could say that the meaning of a relation or entity pair can be estimated from the entity pairs and relations that respectively appear together

#1-is-historian-at-# 2 #1-is-professor-at-# 2 #1-museum-at-#2#1-teaches-history-a t-#2 employeeAt petrie, ucl ferguson, harvard andrew, cambridge trevelyan, cambridge T T T T T T T T ? ? ? ? ? ? ? ? ? ? ? ? vp1 vp2 vp3 vp4 vp5 vc1 vc2 vc3 vc4 ∈ Rk

Textual Patterns Freebase

Figure 2.6: Knowledge base inference as matrix completion with true training facts (green), unobserved facts (question mark), relation representations (red), and entity pair representations (orange).

in facts in a KB. That is, if for many entity pairs we observe that both fatherOf and dadOf is true, we can assume that both relations are similar.

2.3.1.1

Bayesian Personalized Ranking

A common problem in automated KB completion is that we do not observe any negative facts. In the recommendation literature, this problem is called implicit feedback [Rendle et al., 2009]. Applied to KB completion, we would like to infer (or recommend) for a target relation some (unobserved) facts that we do not know from facts that we do know. Facts that we do not know can be unknown either because they are not true or because they are true but missing in the KB.

One method to address this issue is to formulate the problem in terms of a ranking loss, and sample unobserved facts as negative facts during training. Given a known factrs(ei, ej)∈ O, Bayesian Personalized Ranking (BPR) [Rendle et al.,

2009] samples another entity pair(em, en)∈ C such that rs(em, en) 6∈ O and adds

the soft-constraint

v_s>vij ≥ vs>vmn. (2.8)

This follows a relaxation of the local Closed World Assumption (CWA) [Gal´arraga et al., 2013, Dong et al., 2014]. In the CWA, the knowledge about true facts

1s 1ij 1mn

vs vij vmn

matmul matmul matmul

C P u1 u2 dot dot u6 − u8 sigm u10 −log u5 u3 u4 k•k2₂ k•k2₂ k•k2₂ u7 + u9 u11 ∗ ∗ λp λc loss sum

Figure 2.7: A complete computation graph of a single training example for Bayesian per- sonalized ranking with`2-regularization.

for relation rs is assumed to be complete and any sampled unobserved fact can

consequently be assumed to be negative. In BPR, the assumption is that unseen facts are not necessarily false but their probability of being true should be less than for known facts. Thus, sampled unobserved facts should have a lower score than known facts. Instead of working with a fixed set of samples, we resample negative facts for every known fact in every epoch, where an epoch is a full iteration through all known facts in a KB. We denote a sample from the set of constant pairs as(em, en)∼ C.10

This leads to the overall approximate loss

L= X rs(ei,ej) ∈ O, (em,en) ∼ C, rs(em,en) 6∈ O −wslog σ(vs>vij − v>svmn) + λpkvsk22 + λc(kvijk 2 2+kvmnk 2 2) (2.9) 10_{Note that}

whereλpandλcare the strength of the`2regularization of the relation and entity pair

representations, respectively. Furthermore,wsis a relation-dependent implicit weight

accounting for how often(em, en) is sampled during training time. As we resample

an unobserved fact every time we visit an observed fact during training, unobserved facts for relations with many observed facts are more likely to be sampled as negative facts than for relations with few observed facts. A complete computation graph of a single training example for matrix factorization using BPR with`2regularization is

shown in Fig. 2.7.