Insights from other Areas

To get a better understanding of how resource modeling can be done, we have looked towards other areas to find representations of domain-specific search and resource structure. The user study we have described earlier has informed us that domain-specific user needs center around both content (i.e., type of infor- mation) as well as format (i.e., how the information is organized). Therefore, we believe that there is a need to label segments of a resource according to the type of information presented and the resource itself according to its format. Corre- spondingly, the resource representation should further model resource segments in addition to itself, as is confirmed by [Price et al., 2007;Price et al., 2009].

Relevant work in examining and categorizing fragments of webpages exists in the area of information extraction. We note that several of these works (e.g., [Schapke and Scherer, 2004;Wong et al., 2008]) use a layered probabilistic network that models the generation of a webpage fragment starting from the conceptual entity. Both insights have inspired us to come up with our proposed layered graph to combine their strengths to make a generic yet well-structured representation to handle the indexing and searching of domain-specific resources.

Despite the fact that the research on IR with graphical representations has started two decades ago, the document representation in these works did not go beyond word- (concept-) level and was constrained by the generic document structure. We believe this is a major limitation, as the findings from our user study indicate that certain user needs require segments more fine-grained than the document as a whole, but more coarse-grained than just the word-level. By looking at works from other areas, we have confirmed our belief that the resource representation should further model segments in addition to itself and noted that relevant works in information extraction use a layered probabilistic framework to model the generation of segments starting from conceptual entities. To draw on the successes of these works, we have proposed to also use a layered graph in modeling domain-specific resources as described in Section 1.1.

3

Resource Categorization on Nominal

Facets – A Case Study in Key

Information Extraction for

Evidence-based Practice

As pointed out in Chapter 2, domain-specific search engines should be able to categorize domain-specific resources automatically so that specific user needs can be satisfied by specific types of resources without distracting irrelevant results.

This problem of Resource Categorization is a broad topic in the sense that it can be done at many different granularities and on many different facets. For example, at the top level, resources can be categorized by resource type (i.e., the genre of a resource defined based on the types of information it contains and how such information is organized) and readability (i.e., how difficult it is to understand a resource). At the middle level, segments or sentences that compose the resources can be categorized based on the types of information they contain (e.g., definitions, examples and proofs). At the bottom level, words and domain- specific constructs can be categorized according to the types of information they represent (e.g., person names, locations and patient demographics) and their forms (e.g., math variables/operators, chemical elements/compounds and DNA codes/sequences) respectively.

To make our investigation into this problem more manageable, we have di- vided it into two sub-problems: one for nominal facets and the other for ordi- nal facets. The values of nominal facets are categories which are distinct from

separate classes and apply supervised learning to perform the desired catego- rization [Sebastiani, 2002]. At more fine-grained levels (i.e., sentence-level and below), rule-based extraction is also popular [Sarawagi, 2008]. In contrast, the values of ordinal facets are meant to establish an ordering. Therefore, tradi- tional approaches for such facets simply compute some scores heuristically (e.g., the Flesch-Kincaid Reading Ease formula [Flesch, 1948]) to derive the ordering. Although it is possible to treat them as nominal by using an ordered set of cat- egories as values, the fact that they are relative and inexact in nature calls for a different way of handling. Therefore, we focus on nominal facets in this chapter and save the discussion on ordinal facets for the next chapter.

Resource categorization on nominal facets has been studied in various con- texts but often only at one specific granularity level.

For example, genre classification [Lee and Myaeng, 2002] is performed at resource-level, while named entity recognition [Nadeau and Sekine, 2007] and bio-informatics information extraction [Tanabe and Wilbur, 2002] are at word- level. Although classification at sentence-level is often employed in the context of question answering [Demner-Fushman and Lin, 2007] and information ex- traction [Sitter and Daelemans, 2003] to identify the sentences that contain the information of interest, it is treated as a preprocessing step instead of part of the main task in such contexts.

We believe this is a limitation in domain-specific IR, due to two reasons. First, the categorization of domain-specific resources needs to be done at multiple granularities. Only in this way would users be able to filter out unsuitable results by coarse-grained facets and then select the most appropriate ones based on fine-grained facets. In addition, more coarse-grained categorizations may serve as a fallback when more fine-grained ones are unreliable or unable to capture the desired information well. Second, without considering categorizations at different granularities simultaneously, their correlations, which are often useful in improving categorization performance, will be left unexploited.

Therefore, we choose to focus on investigating how to improve categorizations of different granularities by exploiting the correlations among them. To this 36

end, we use key information extraction for evidence-based practice in healthcare as a case in point. The problem of key information extraction is to extract information pertinent to evidence-based practice, such as patient demographics, interventions, results and study design, from research articles in the form of sentences and words. As such, there are two correlated categorizations to be performed: one at sentence-level and the other at word-level.

Figure 3.1: Correlation graph fragment showing nodes and edges relevant to segment and sub-segment type. The edges (i.e., correlations) bounded by the dashed line box are examined in this chapter.

In our correlation graph (Figure 3.1), to categorize at these two levels is to find the values for two nodes: segment (i.e., sentence) type and sub-segment (i.e., word) type in the segment layer. As represented by the edges in the graph, these two nodes are correlated with many other nodes including those above in the resource layer or below the sub-segment level. Since our primary interest is to examine how the categorizations of two different levels interact, we ignore the correlations beyond these two levels. This leaves us six correlations as bounded by the dashed line box shown in Figure3.1. Without considering the correlation between the two categorizations (i.e., treating the two categorization as inde- pendent), the remaining correlations simply mean that the categorization of a segment can be done based on the information from itself and its context as

established by the types of its neighbours. This is suboptimal, as knowing the segment type helps to determine the sub-segment type and vice versa.

For example, in key information extraction, knowing that a sentence de- scribes patients in a medical study will increase the likelihood that the words in this sentence represent patient demographics (e.g., age and sex) and vice versa. Therefore, for this part of our research, we have treated these categorizations as supervised classification problems and examined how to exploit their correlation through propagating information between the classifications.

We believe the findings from our research can be applied to improve resource categorization on other pairs of correlated nominal facets, such as resource type and segment type. We will elaborate on this towards the end of the chapter.

The rest of the chapter is organized as follows. We start with a detailed de- scription of the problem of key information extraction for evidence-based practice in the domain of healthcare in Section3.1, followed by a literature review in Sec- tion 3.2 on entity extraction and key information extraction. Then we present our models for exploiting the correlation between categorizations for key infor- mation extraction in Section 3.3. We evaluate the performance of the models with different settings and explore the effects of data filtering and feature selec- tion in Section3.4. We present directions for future research in Section3.5 and end with a discussion on Resource Categorization on nominal facets based on our findings in Section 3.6.