Overview of Methodology

As noted previously, the instrument used in this research will contain both selected response items assessing demographic and historical data as well as psychometric items

assessing latent variables. Unfortunately, while some existing instruments examine specific kinds of expertise within certain domains, their application to a more general instrument is doubtful. Also, instruments used to measure the similar concept of information literacy are primarily used in educational settings and often too long for use in general research. While items related to cognitive and affective abilities have a large body of research available, many of the instruments used to measure these traits are proprietary or licensed. Finally, the items used in past research to measure search expertise are often outdated or used inconsistently. This led to a strategy of starting with no pre-selected basis for the instrument to be developed. Since the instrument was expected to contain psychometric items, the suggested steps for scale development were

followed (Wildemuth & Choemprayong, 2009; DeVellis, 2012).

The procedure followed can be seen in Figure 2. In the first two phases, prior research was examined and search expert opinions were consulted to create a definition and model of online search expertise as well as generate a list of potential items for the instrument. The items were then examined by targeted participants to verify their usefulness and clarity. The instrument was then edited based upon those searchers. This edited instrument was then administered to four specific groups of online users in order to test reliability and validity. It should be noted that this process is iterative rather than singular and so is expected to continue into the future.

Figure 2: Flowchart of Research Steps

Literature Review • Past research in expertise examined.

Create working definition and model

Phase 1: Search Experts

• Search experts asked to define online search expertise. Initial Item pool generated.

Phase 2: Focus Groups

• Groups of experts asked to discuss and organize items from Phase 1. Item pool refined.

Created initial instrument

Phase 3: Cognitive Interviews • Targeted searchers asked for reactions.

Instrument Revised

Phase 4: Online Testing • Revised Instrument given to 4 groups online.

3.4.1 Literature Review

Prior research in Psychology, Education, Sociology, Information Literacy, and

Information Science was examined for both conceptual discussions and models of search, but also for how search expertise had been used to categorize past participants. The results were then examined and grouped based upon similarity of concept. The results of this step are reported in Chapters 2 and 3.

3.4.2 Create Working Definition and Model

The concepts from prior research were used to create both a working definition and a working model of online search expertise. The results from this step are reported in Chapter 3.

3.4.3 Phase 1: Search Experts

Search experts were then asked for their thoughts regarding online search expertise, including its definition and use. Each expert was interviewed individually and the results of those interviews were recorded. Concepts were gathered from those interviews and compared to the concepts and groups from prior research to obtain a starting list of concepts for development into items. This process is more fully documented in Chapter 4.

3.4.4 Phase 2: Focus Groups

The nine search experts from the first phase were gathered into groups of three for focus groups to discuss the items generated from their interviews. Each focus group examined a set of index cards, each card containing one of the concepts mentioned in one of their three interviews. All items were from the three search experts in that focus group. They were asked to categorize and organize the cards based upon meaning and were allowed to discard cards or add new cards

as desired. The results from these focus groups were compared to the proposed model and added to the list of concepts from the literature review to create a starting item pool. This process is more fully documented in Chapter 4.

3.4.5 Created Initial Instrument

The list of concepts was examined for the feasibility of creating items for measuring them. Existing items from research were examined and updated where possible. Some items were borrowed from open source Information Literacy tests used by state governments. Other items were created by combining related concepts from the search experts, such as grouping together certain types of experience or certain sources into items.

For the psychometric items, the subscales used to measure latent psychological traits were taken from the International Personality Item Pool (IPIP). The International Personality Item Pool (http://ipip.ori.org/ipip/) is a scientific collaboratory for the development of advanced measures of personality. It contains public domain broad-bandwidth items that were specifically developed for general use in research.

Measurement for non-psychometric items was based upon similar items from existing research. The International Personality Item Pool (IPIP) gave structure to the items requiring a Likert scale as well as guidelines in how to label the points on that scale. Details of this process are more fully documented in Chapter 5.

3.4.6 Phase 3: Cognitive Interviews

This step was expanded in this research process to include both experts and non-experts since this instrument needed to be accessible and meaningful to both groups. Search experts were given the initial draft of the instrument informally to gather design feedback. Cognitive

interviews were then performed with deliberate samples of the population to evaluate the face validity of the instrument, detailed in Chapter 5. This phase deliberately included self-identified experts and non-experts because the online environment has changed so much since many of the studies of search expertise were conducted, making it possible that the environment was no longer as understood previously. Including multiple levels of expertise allowed the items to be examined for multiple levels of ability, including places where items might perform well for one set of searchers, but not as well for another set.

3.4.7 Instrument Revised

The initial instrument was revised based upon both the data from the cognitive interviews and also upon the answers from the participants on the initial draft instrument. Each item was evaluated for its clarity and specificity, as well as its discriminatory power. The results and changes to the instrument are detailed in Chapter 5.

3.4.8 Phase 4: Online Testing

The revised instrument was implemented on the UNC Qualtrics site and made available to four different subgroups: Amazon Mechanical Turk users, members of the UNC Faculty and Staff list, members of the RUSA list, and members of the ASIS&T list. Results from this administration can be seen in Chapter 6. The data from this phase was examined using various statistical methods, depending on the type of item. The demographic and categorical items were first examined using descriptive methods. They were then used to group participants to examine the skill items, self-rated items, and latent variables. The self-rated items and latent variable items were examined using exploratory factor analysis.

Evidence for content-related validity came from the use of expert searchers in initial phases. The use of experts in this way to examine items proposed for an instrument is common in research and often referred to as face validity (Wildemuth, 2009). Additional evidence for was gathered during the focus groups, when the search experts were asked to examine the proposed items and discard any that they believed to be not relevant to online search expertise. Changes in both technology and the online environment indicate that evidence for content-related validity should be gathered throughout the development of this instrument.

Evidence for criterion-related validity is normally assessed by using an existing measure that is related to the latent variable being examined (Wildemuth, 2009), with the idea that the scores should be similar on both the established measure and the measure under investigation. For this research, no existing measure exists for online search expertise by which to formally assess criterion-related validity. However, the research was designed to contain cohorts of participants containing known experts in search. The first such cohort, the Reference and User Services Association (RUSA) is an association whose mission states their commitment to providing reference and information services and their core purpose of connecting people to information services (http://www.ala.org/rusa/about). Their membership includes trained reference librarians with experience searching using multiple online systems. Their training and experience would allow for them to score highly on a measure of online search expertise if such a measure existed. Therefore, their scores on this measure can be viewed as evidence in favor of criterion-related validity.

Similarly, the Association for Information Science and Technology (ASIS&T) cohort focuses on research into techniques, theories, and technologies to improve information access (https://www.asist.org/about/). Their membership is more diverse than the RUSA cohort,

including librarians, information scientists, and others conducting research in information, but are still trained in and focused on information use. Like the RUSA cohort, the ASIS&T cohort would score highly on a measure of online search expertise and their scores can therefore be used as evidence in favor of criterion-related validity.

This method of collecting evidence for criterion-related validity is not ideal, but is necessary due to the lack of established instruments to measure online search expertise. While not ideal, any evidence related to validity is important in the early stages of instrument

development. Continuously collecting this evidence with additional participants will serve to provide more context and a richer understanding of the underlying construct. Rather than using the standard correlation between the experimental measure and the established measure, this research must instead examine the frequency of correct classification of known search experts from these two cohorts.

Construct validity is usually examined through statistical tests using factor analysis, which can be exploratory to examine new items, or confirmatory to examine known items. Exploratory factor analysis is often used in instrument development to determine the underlying relationships between measured variables (Costello & Osborne, 2005), although it has few absolute guidelines in its use. While strict rules for sample sizes needed for exploratory factor analysis do not exist, DeVellis (2012) states that a sample size of 300 is sufficient for most applications. For this research, exploratory factor analysis was used to examine the set of items used to measure the self-reported skill items. Confirmatory factor analysis was used to examine the pre-existing scales for the personality variables. Results of factor analysis allow suspect items to be removed for future versions of the instrument.