In INEX [G¨overt et al., 2006] the scales for exhaustivity and specificity are mapped onto ratio scales. To this end, INEX uses quantisation functions over the two parameters of exhaustivity (e) and specificity (s): f (e, s). A single relevance scale [0,1] is the result, as presented in Table6.1for INEX 2004 and Table 6.2for INEX 2005. The first two columns in both tables are taken from [Ogilvie and Lalmas,2006].
The quantisation functions order the combinations of exhaustivity and specificity val- ues. In Tables 6.1and 6.2, the strict functions (Strict4 and Strict5) are used to evaluate
XML retrieval methods with respect to their capability of retrieving highly exhaustive and highly specific components. The generalised functions (Gen4 and gen5) also reward only
fairly relevant elements. Other quantisation functions have more specific aims. AnyRel in Table6.1evaluates whether INEX approaches can return any relevant element, regard- less of their exhaustivity and specificity value. In Table 6.2, F ullySpec and BinExh are functions that reward elements independently of exhaustivity. The ? stands for elements that are too small to allow an aboutness conclusion. This value of f (e, s) is new to INEX 2005 reflecting the specific problem with XML document components that are too small to bear information. Further discussion of the quantisations will follow below.
In order to deliver agent representations for the INEX quantisations, we need to express these first in a Situation Theory framework. To do so, we divide the document component and query situations into subsituations, with D ≡ D1⊗ ... ⊗ Dn and Q ≡ Q1⊗ ... ⊗ Qm.
Please recall that according to Section 3.3.3, a subsituation is a situation Si that is part
of another situation S, where we count the situation as a part of itself, i.e. a situation is a subsituation of itself. Thus, a situation S is about a situation T if and only if T contains a subsituation Ti such that situation S is about situation Ti. Also, we distinguish strict
subsituations, i.e. those Si that are not S.
Using the subsituation-based aboutness criterion from Section 3.3.3, we assume that if D is an exhaustive answer to Q, then it is due to one of the situations Di that D is
composed of. With our subsituation-based aboutness criterion, we are able to represent agent reasoning according to INEX in Section 6.4and the INEX assessment methodology in Section6.5within a single theoretical framework. We speak of rational agent reasoning to include both system and user reasoning.
Quantisations in INEX reflect the importance attached to exhaustivity and specificity. As such they can be used to describe user agent reasoning about results that system agents should return. E.g., Strict4 in Table 6.1 as much as Strict5 in Table 6.2 only
credit highly exhaustive and highly specific elements and thus express very demanding user requirements. Within our Situation Theory framework, we have the advantage of being able to express a user’s need and a system’s attempt to satisfy it within the same framework. Both are reasoning processes that follow rules. This can be considered to be one of the major advantages of a logical theoretical evaluation approach. User assessments are as much as system assessments results of reasoning processes [Huibers, 1996]. In this section, we demonstrate the reasoning of user agents, as we are concerned with the
Table 6.2: Quantisations in INEX 2005
Function f(e, s) User model
Strict5 f(e, s) = ( 1 if e=2 and s=1 0 otherwise UU F ullySpec f(e, s) = ( 1 if s = 1 0 otherwise SDRU Gen5 f(e, s) = ( e∗ s if e ∈ {1,2} 0 otherwise EU
GenLif ted f(e, s) = (e + 1) ∗ s if e ∈ {1,2} s if e = ? 0 otherwise EU BinExh f(e, s) = ( s if e ∈ {?,1,2} 0 otherwise SDRU
representation of the INEX evaluation methodology.
In the following formalisations Dj and Qj denote one of n unique subsituations of an
XML situation such as an XML element or a query. Dex marks the subsituation that
determines a component to be an exhaustive answer, while Qsp states that the component
is a specific answer.
The quantisation of Strict4 as much as its INEX 2005 equivalent Strict5 simulates
those user agents only interested in highly exhaustive and highly specific answers. These
unanimous users will only be satisfied if aboutness systems return the highest exhaustivity
and specificity values [Huibers,1996]. The Unanimous User (UU) will only be happy if she can find nothing else, but the two subsituations Dex and Qsp. She wants them to be
equivalent to the situations D and Q, respectively, in order to conclude either D Q or Q D.
Unanimous User (UU)
Dex Q, Dex≡ D, Qsp D, Qsp ≡ Q
D Q, Q D
A user looking for specific answers but at the same time not wanting to entirely lose out on exhaustivity can be called a Specificity-oriented User (SU) represented by SOG in INEX 2004, but without a real equivalent in INEX 2005. SOG only gives preferences to specificity by assigning higher quantisation values to higher specificity values.
Specificity-oriented User (SU)
D1 ⊠ / Q, ..., Dn⊠ / Q, Qsp D, Qsp≡ Q
D ⊠ / Q, Q D
The complement to SOG with a tendency to favouring exhaustivity is Gen4. It values
higher exhaustivity and represents the Exhaustivity-oriented User (EU). As long as most aspects of the query are discussed, the focus is secondary. The Exhaustivity-oriented User (EU) does not neglect specificity fully. The focus, however, is to have D Q. For INEX 2005, Gen and GenLif ted both place an emphasis on exhaustivity and their Situation
Table 6.3: INEX 2005 exhaustivity and specificity situations
Scale Exhaustivity Specificity
D Q Q D 0 D1 / Q, ..., Dn / Q Q1 / D, ..., Qm / D D / Q Q / D 1 D1 ⊠ / Q , ... , Di Q, ... , Dn⊠ / Q Q1⊠ / D , ... , Qi D , ... , Qm⊠ / D D ⊠ / Q , ... , D Q , ... , D ⊠ / Q Q ⊠ / D , ... , Q D , ... , Q ⊠ / D 2 D1⊠ / Q, ..., Di Q, ..., Dn⊠ / Q Q1⊠ / D, ..., Qi D, ..., Qm⊠ / D D Q Q D 3 D1 Q, ..., Dn Q Q1 D, ..., Qn D D Q Q D
Theory representation reflects this by demanding D Q as an overall conclusion and rewarding those XML elements that include exhaustivity subsituations.
Exhaustivity-oriented User (EU)
Dex Q, Dex≡ D, Q1 ⊠ / D, ..., Qn⊠ / D
D Q, Q ⊠ / D
In INEX 2004, the AnyRel-function captures the typical user of mass information systems, happy with any relevant component. There is no equivalent in INEX 2005. The Typical User (TU) would like to see any kind of subsituations, allowing to conclude either exhaustivity or specificity. She is not interested in an overall conclusion of D Q or Q D, but in partial conclusions indicating either an exhaustive or a specific answer.
Typical User (TU)
D1 Q D Q , ... , Dn Q D Q , Q1 D Q D , ... , Qn D Q D
Instead of a direct equivalent to SU , INEX 2005 comes up with two new user types BinExh and F ullySpec. Both only look for specificity, as long as exhaustivity is not impossible. BinExh is not as strict with respect to the exhaustivity value. In this sense, it corresponds to Chiaramella’s earlier suggestions that describe the focus of the answer as the specific interest of XML retrieval. [Chiaramella, 2001] has demonstrated within a theoretical experiment that Structured Document Retrieval Users (SDRU) are interested in specificity as long as the answer remains exhaustive enough. This is why we call this model SDRU:
Structured Document Retrieval User (SDRU)
D1 ⊠ / Q, ..., Dn⊠ / Q, Qsp D, Qsp≡ Q
Q D
SDRU’s differ from SU’s only in that their overall conclusion is only influenced by specificity. SDRU’s are looking to find a Qsp D in order to conclude Q D. Not
Table 6.4: INEX 2004 exhaustivity and specificity situations
Scale Exhaustivity Specificity
D Q Q D 0 D1 / Q, ..., Dn / Q Q1 / D, ..., Qn / D D / Q Q / D 1 D1⊠ / Q, ..., Dn⊠ / Q Q1⊠ / D, ..., Qn⊠ / D D Q Q D 2 D1⊠ / Q, ..., Dn⊠ / Q, Dex Q Q1⊠ / D, ..., Qn⊠ / D, Qsp D D Q Q D 3 Dex Q, Dex ≡ D Qsp D, Qsp≡ Q D Q Q D
retrieval compared to flat document retrieval is better represented by SDRU’s than by other agent models, as the overall conclusion is focussed on specificity only.
To better see the overall use of these agent reasoning models, let us briefly investigate what is possible if we can express system and user reasoning in the same framework. We can combine, for instance, Left Monotonic Union (LMU) and Unanimous User (UU) model in: UU: Dex Q, Dex≡ D D Q LMU: D Q D ⊗ D1 Q ⇒ D ⊗ D1 Q D ⊗ D1 6≡ Dex
The conclusion that D ⊗ D1 Q clearly contradicts the assumption of the UU that Dex≡ D, which means UU’s will not be served well by aboutness reasoning systems that
include LMU.
In this section, we have presented agent reasoning models, as expressed in the INEX quantisations for XML retrieval, based on Chiaramella’s differentiation of D Q and Q D. We have added a third column to Tables6.1and6.2to summarise these results. We have shown the new focus in INEX 2005 on specificity and would like to investigate this issue further by looking at the transition in terms of the system agents’ rewards from INEX 2004 to INEX 2005.
The next section places the INEX exhaustivity and specificity assessment scales into the context of Situation Theory. We will show that system agents are rewarded if they reflect the user agent reasonings. For example, in order to reach the highest values for exhaustivity and specificity, they must support the reasoning of unanimous users.