Full-text Search

Storing data in a document data store as described in Section 5.3 enables variants of full- text search. Three variants of full-text search have been observed:

 Searching attached documents. This search relies solely on document attachments in the CouchDB data store. Because original files are attached to the CouchDB document in their original form, this search is somewhat similar to using an indexing and search engine, Lucene in this case, directly on the original files. This strategy does not take advantage of any data processing performed during knowledge acquisition and is the baseline for comparison with other strategies.

 Searching extracted text. This strategy includes only the contents of docText field. Because text extracted from images is in docImageText fields, this strategy ignores text contained in images as well as text in images embedded in other documents. Note that ontology-based simulation models are stored in docText fields and therefore are included in this strategy.

 Searching extracted text, including text from images. This approach takes full advantage of text extraction service described in Section 7.2.1, including Tika text extraction and OCR text extraction, by engaging both fields, docText and docImageText, in the search strategy. This strategy also takes full advantage of the data processing performed in the knowledge acquisition stage.

A full-text search screen from the implemented Web application is displayed in Figure 7.1. This application enables users to choose among the three described search strategies; on the screen in Figure 7.1, the extracted text strategy is selected. The result of searching for the term ―power house‖ are displayed in the table with two columns: document and last modified. The document column displays the file name, and it can be noted that the search result is made up of various file types, including pdf and text files, MS Word, PowerPoint, and simulation model files. Some of the files appear several times with different last modified date. This is caused by files residing in different folders, but having the same name. Disaster-CDM does not check whether files with the same name have identical content, but rather creates a new document in the data store for each loaded file.

Table 7.2 provides an overview of different full-text search strategies with respect to the main file categories addressed in this study. For the three file categories, PDF, text and I2Sim model files, all three search strategies were virtually the same. Even though searching I2Sim models produced the same results set, the ranking of the documents was different because the searches were based on different text content. The attached document strategy searched mdl files, which are text files, directly, while the other two strategies searched the ontology-based simulation models. Consequently, the attached document strategy ranked simulation models lower than the other two strategies.

With regard to MS Office files, the difference among the various searches depended on whether or not they were using text extracted from images. The data set for this case study contained 82 MS Word files (doc and docx), of which only 8 contained images from which text was successfully extracted. In contrast, out of 140 PowerPoint files (ppt and pptx), only 6 did not benefit from the OCR service. Therefore, the OCR service had a greater impact on processing PowerPoint files than on processing Word files. With respect to image files, out of 116 images, text was successfully extracted from 75; however, some of the extracted text did not contain readable words and therefore was not beneficial for searching. Therefore, the OCR service had a greater impact on PowerPoint files than on image files, which can be explained by the common use of diagram-style graphs in PowerPoint presentations.

Figure 7.1: Full-text search S im ulation m ode l fil es MS W or d, P owe rPoi nt, pdf a nd txt f il es

Table 7.2: Search strategies

Search Strategy File type Attached

document

Extracted text Extracted text including text from images

PDF files   

MS Office files 

Does not include text from images



Does not include text from images



Image files   

Text files   

Simulation model files

I2Sim model files (.mdl) 

(mdl file are text files)

 

EPANET model files (.net)   

Transforming simulation models into their corresponding ontology-based representations did not change the result set with respect to I2Sim models, but was essential for including EPANET models in the full-text search. The attached document strategy did not search EPANET models because they are represented in .net binary files; however, the extracted text strategies searched EPANET models by taking advantage of the ontology-based simulation models stored in docText fields.

Note that the attached document search strategy took advantage of CouchDB-Lucene [79], which uses Apache Tika [72] to search the attached documents. This case study also used Tika to extract text from files, and therefore the only major difference between the attached-document and the extracted text strategies was with respect to EPANET model files. Only the extracted text strategy included EPANET model files.

Full-text search can also be achieved by applying text search engine such as Lucene directly on the file system containing disaster-related data; however, such search ignores text contained in images as well as text in images embedded in other documents. In contrast, full-text search in Disaster-CDM includes image text because OCR performed in knowledge acquisition stage extracted text from images. Moreover, direct full-text search on the file system does not include EPANET .net model file as they are binary file. Disaster-CDM transforms EPANET model files into ontology-based representation, and consequently includes them in full-text search. Additionally, storing data in NoSQL data

store facilitates querying file-style data and allows Disaster-CDM to take advantage of scaling and replication capabilities provided by NoSQL store.