Threshold Learning

Our final goal is to find a mapping of the OIE relations to a correct DBpedia prop- erty. In the previous section we have formulated a score for each association tuple as ⌧i

p. It is clear from the examples in Table11that there can be a wide range of

values for ⌧i

p, for varying i and p. In this section we propose a technique to learn

a correct threshold value for ⌧i

p for a given p. We cannot have a single threshold

value across all the relations since each relation is different with varying mapping degrees and confidence scores.

Intuitively, for each instance set of p (i.e. Tp), there can be three broad observ-

able association patterns: • a single high confi

passociation, among many others.

• multiple closely spaced possible KB properties with almost same confi p

100 rule based approach Ap p Kp i dp dom(p) ran(p) ⌧p ˆ⌧ Aare in are in

1 isPartOf Village AdministrativeRegion 29.0 3.07 2 country City Country 19.67 3.07 15% 3 isPartOf Settlement Settlement 20.3 3.07 4 family Plant Plant 29.0 3.07

5 . . . . Ag r e w up in g r e w up in

1 hometown Person Settlement 0.42 0.52 2 residence Person PopulatedPlace 0.33 0.52 79% 3 deathPlace Boxer Settlement 19.17 0.52 4 hometown Band Town 115.0 0.52 5 citizenship Person Country 3.02 0.52

6 . . . . A airpo rtincity airpo rtincity 1 location MilitaryStructure - 150 1.21 2 location Airport - 0.86 1.21 55% 3 isPartOf Settlement Settlement 150 1.21 4 isPartOf Settlement - 37.5 1.21 5 city Airport - 0.86 1.21 6 . . . . Aag entcr ea ted ag entcr ea t e

d 1 notableWork Writer Play 496.6 3.12 2 notableWork Writer TelevisionShow 3973 3.12 9% 3 occupation Person Book 12.7 3.12 4 occupation Settlement - 37.5 3.12 5 knownFor Scientist Book 3973 3.12

. . . .

Table 11: A snippet of the actual associations presenting a positive example with 4 exam- ple OIE relations: airportincity and a negative example with agentcreated (from Nell); grew up in and a negative example with are in (from Reverb). A blank value (’-’) is attributed to a missing instance type in DBpedia or often scenar- ios where there are no DBpedia relation to capture the semantic relationship expressed by p.

• multiple candidates with low confi p

We aim at modeling these different scenarios which would select the first two cases but not the third one. The rational is, any association rule with a low con- fidence is not an appropriate predictor for p. Now, when we include Kp into the

equation, we get the confidence values translated as ⌧ values. From the expression of ⌧ in Section9.1.1, it is clear that the best score is attained when a particular rule confidence for a relation p attains maximum. We denote this minimum tau value simply as ⌧min

p . This is of particular interest since, this is the best possible score

a particular relation can have. For different p this value changes. We capture this variation by presenting a distribution pattern for ⌧pover Kpwith detailed figures

9.1 methodology 101

It must be observed that, with relations with very low mapping factor Kp, there

is a drawback in using them for the purpose of relation mapping task. A low Kpindicates low mapping degree which is caused due to dearth of adequate ev-

idences supporting a mapping p ! dp. Hence, the observed evidences are too

weak to make a strong decision on the mapping. If we chose such relations, we are then trying to deduce a relation mapping based on low evidence, which in our opinion is not justified. However, relations with higher Kpare devoid of this prob-

lem. Hence, we select a threshold value of Kp which is appropriate for a given

data set. The choice of this threshold is different across Nell and Reverb. In Sec- tion11.1.2, we report this behavior with some experiments and provide rationale for our choices. The set of experiments allow us to make an empirical decision in choosing a Kp which is high enough to make conclusive deductions. We define a

set D consisting of pairs { . . . , (Kp, ⌧minp ), . . . }, for all p 2 Tpsuch that Kpis higher

than the empirically determined threshold value. We fit a linear regression model on set D, and compute the squared loss for each. The threshold which gives the minimum loss is finally chosen. In Section9.2we briefly discuss linear regression as relevant for the error calculation in our approach.

With such a linear predictive analysis method, we can have an estimate of ⌧, defined as ˆ⌧ for every Kp. Note that, we trained our model using the data points

attained using the maximum confidence (analogously ⌧min

p ), hence, the linear

model is an automatically learnt threshold on ⌧p. We use ˆ⌧ to compare with

every ⌧i

p, 8 i 2 Ap. Some scores fall below the prediction (the likely associations)

and some are way above it (less likely ones) (refer to Table11, correct association values are marked in bold). The likely associations allow us to select the rule with acceptable ⌧ values. These are the rules which are considered to have higher credibility than the rest. We would re-visit the three goals we set in the beginning of this section and show that the computation scheme actually addresses them well.

• Multiple associations but a single one with a high confi

p. This makes ⌧ip'

ˆ⌧, since the high confidence association will lead to the best ⌧p score.

• Multiple closely placed associations with almost same confi

p, making ⌧ip'

⌧jp' ˆ⌧; i 6= j. (refer Table11, ⌧2airportincity and ⌧5airportincity)

• No clear winner, but multiple candidates with low confi

p making ⌧ip o ˆ⌧

(refer Table11, ⌧1

102 rule based approach

Recollect from the association tuple (Expression11.1.2) and the association rule (Equation13), that we can have a direct mapping for the corresponding DBpedia relation. In our entire analysis, we never used the DBpedia relation for any direct computation but let its domain/range restrictions allow us to find the appropriate mapping.