Research approach and study design

3.4.1 Overview

The study used a mixed-methods approach to investigate the research questions outlined in the previous section. The study included five elements a) an evaluation of participant background and use of information systems, b) a pre-

instruction assessment of ML, c) an instructional element on metadata, d) a post- interaction assessment of ML and e) qualitative questions about metadata uses. The gathered data was analyzed using both quantitative and qualitative methods. This resulted in a holistic view of how participants used metadata and what their attitudes towards metadata were.

Participant responses were analyzed to determine differences in participant self-efficacy ratings, task ability levels and pre/post instruction efficacy ratings. Qualitative responses were used to provide contextualizing perspectives on participant information use. Participant groups were formed based on responses to the initial set of questions regarding their academic background. The following sections discuss each of the research steps, including the beginning survey, interactions, instruction process and post-instruction evaluation.

3.4.2 Demographic/information use survey

Survey questions gathered information on academic major, number of years of post-secondary education and information on how participants use common digital information systems. These survey questions are in Appendix A and are comprised of questions one through seven. The primary purpose of the survey portion of the study was to provide categorizing information which could be used for comparative analysis following the study. This section of the study gathered

information on participant ability to define metadata and provided information on how they use metadata rich systems.

3.4.3 Metadata literacy initial assessment

Participant level of ML was assessed prior to instruction using two metadata tasks and a self-efficacy instrument. The two metadata tasks assessed

participant ability to identify different types of metadata and are represented in Appendix A as questions nine and ten. The self-efficacy instrument is

represented as question eight in Appendix A and is comprised of seventeen questions which were generated by the researcher. The seventeen questions were designed to map on to three of the seven levels of Bloom’s updated taxonomy. This mapping is available in Appendix B. Questions were adapted from several sources including an IL self-efficacy instrument developed by Kurbanoglu et al. (2006), field experts, and the literature review of metadata and IL concepts. Following question identification and organization, information science professionals and members of the target population were asked to provide feedback on the instrument. These participants were excluded from the full study.

Following the self-efficacy instrument, two metadata tasks involved having participants identify metadata elements by clicking on a screen-shot taken from Flickr.com. Each metadata task had participants identify a single type of

metadata. In the first interaction, participants were asked to identify descriptive metadata elements and, in the second, participants were asked to identify social metadata elements. Participants were asked to identify appropriate elements by clicking on sections of the screen-shot from Flickr. When participants clicked on an area, it turned green. A metadata identification score was calculated for each interaction by adding together the correctly selected and correctly unselected areas of the screen-shot and dividing by the total number of elements in the interaction. Participants were assessed with a single metadata identification task and a metadata creation task following instruction.

The initial assessment of ML allowed the researcher to gather information on actual participant abilities with regards to metadata tasks and participant view of level of literacy with regards to metadata.

3.4.4 Metadata instruction

Following the initial assessment of metadata ability, participants were asked to view two videos regarding digital information and metadata. The first

instructional video was called “The Web is Us/ing Us” (Wesch, 2007). This resource was selected because it has a relatively short length (4:34) and because it covers many of the important elements of metadata and Web 2.0 applications. Instructional elements of the video included how metadata helps streamline web use, what different encodings of metadata look like, and some

ways of using metadata-rich documents. The second video

(http://www.screencast.com/t/QzjepD1R6) was created by the researcher and focused specifically on defining metadata and how it is used in digital image sharing sites. It was one minute and twenty-six seconds in length and included a short description of what metadata is and a discussion of different types of

metadata (e.g. descriptive, social and technical). An example image along with its metadata from Flickr (zyrcster, 2008) were used.

The purpose of these two videos was to provide the participants with a base level of instruction that could help them generalize specific knowledge they had regarding metadata from other environments such as IL instruction or information system use.

3.4.5 Post-instruction metadata literacy assessment

Following the instructional videos, participants were again tested on metadata using two tasks. The first was similar in nature to the pre-instruction task in that it asked participants to identify descriptive metadata elements using a screen-shot from digg.com. The screen-shot included descriptive entries for the website change.gov.

The second task was a metadata creation task. In the second task,

participants were asked to create five metadata tags for a supplied image. This task involved an image of Barack Obama being inaugurated. The image was

pulled from the open collection of USAToday on Flickr (USA Today, 2008) and was a readily recognizable image to the population. The metadata identification task was used to compare pre and post instruction task completion scores, while the metadata creation task was used to investigate how participants created metadata.

3.4.6 Dependent and independent variables

The dependent variables being evaluated in the study were participants task ability level and self-efficacy ratings. These elements were assessed again following the instruction portion of the study. The independent variable being manipulated in this study is the participant’s awareness of ML concepts.

Qualitative data was analyzed to identify metadata use themes and

participant view of metadata in general. In addition, data gathered during the metadata creation task was analyzed to understand more about what kind of metadata the participants create.

3.4.7 Testing and validation

The elements of this instrument have been validated in two ways. First, experts were selected via snowball sampling and were asked to evaluate the extent to which the questions assessed IL and ML. Second, the survey was administered to a small sample of participants (less than 10) to check for survey

flow and logic and to discover any unexpected issues. Participants were asked to complete the survey in addition to providing their impressions. The IL self- efficacy instrument developed by Kurbanoglu et al. (2006) was used as

framework for the self-efficacy instrument. Actual self-efficacy questions relevant to the concept of ML were created by the researcher and other information

science professionals.