Design and Administration of Questionnaires 2 and 3, Round 1

CHAPTER 3: RESEARCH DESIGN AND METHODS

3.8. Design and Administration of Questionnaires 2 and 3, Round 1

participants to rate a list of 100 file formats on the first draft of the endangerment scale:

• Information stored in this file format is already inaccessible.

• Information stored in this file format will be inaccessible in 1-5 years. • Information stored in this file format will be inaccessible in 6-10 years. • Information stored in this file format will be inaccessible in 11-20 years. • Information stored in this file format will be inaccessible in more than 20

years.

• I am not familiar enough with this file format to rate it.

I asked participants to write a brief description of the rationale for their answers. I

collected each response and their incumbent rationale text into one document and shared the

anonymous collected answers with all of participants in the group.

Participants provided feedback on minor spelling errors and inconsistencies with

format names and versions. Most importantly, I received overwhelming feedback that rating

feedback and participant input on how long it took them to give thoughtful ratings, I reduced

the number of formats to 50. First, I removed from the list file formats that at least half of the

pilot participants indicated that they did not know enough about to rate. I was able to remove

45 formats from the list using this criterion. I then removed five additional file formats that

half of participants did not know about, starting with the least frequently appearing formats

according to the NARA count and moving up.

I asked participants to re-rate the file formats after reviewing the responses of their

fellow participants. In this second round of rating the formats in the questionnaire, I reduced

the number of formats to 50, based on participant feedback. After participants completed the

second round, I computed Spearman’s rank correlation coefficient values for each of the file

formats and determined that a third round was not necessary.

For Questionnaire 3, I asked the second group of six participants to rate a list of 21 file

format endangerment factors on a scale for how relevant the factor is as a cause of file format

endangerment:

• Not relevant at all • Somewhat relevant • Very relevant

I received very little feedback from pilot participants on the factor-rating

questionnaire. Two participants indicated some confusion about being asked to answer the

same questions in subsequent rounds of the Delphi. Based on this feedback, I added

introductory text to the questionnaires that explained this more explicitly. All other

participants indicated that the instructions and purpose were clear, and so I made no other

appropriate participants after completing the first questionnaire. I then asked them to review

their fellow participants’ responses, and then re-take the same questionnaire in a second

round. After the second round of questionnaires was complete, I calculated Spearman’s rank

correlation coefficient values based on the procedures described above. Based on the

calculated values, I determined that answer stability was achieved after two rounds, and

concluded the pilot at this point.

For the actual study, Questionnaires 2 and 3 were administered to each of their

corresponding participant groups using the Qualtrics online survey software. I sent links to

the questionnaires to participants using the Qualtrics link generator and email-tracking

module. Participants had twelve days to complete the questionnaires and were issued

reminders after seven days, ten days, and twelve days.

Questionnaire 2. File Format Endangerment Level Rating. In the second questionnaire, I

asked participants to rate a list of 50 file formats according to the degree to which they

believed information encoded in each format is at risk of not being accessible. The choices

for rating file format endangerment levels for each of the formats consist of a six-point scale:

• Information stored in this file format is already inaccessible.

During recruitment, I changed the fifth rating from its original wording of “more than

conversation with one of the recruits. I asked participants to explain the rationale of their

ratings for each of the file formats. See Appendix C: Questionnaire Designs, Questionnaire

2: Rating File Formats, to view the design of this questionnaire. I provided participants with a short file format-rating guide that provided definitions of key terms used in the

questionnaire. The guide appears in Appendix C: File Format Rating Guide for Participants.

Questionnaire 3. File Format Endangerment Factor Rating. The third questionnaire was

designed to collect expert opinion on which factors are most relevant as causes of file format

endangerment. In the questionnaire, I presented participants with the list of file format

evaluation factors compiled from the dozen file format evaluation lists found in the literature.

In this questionnaire, I asked participants to rate each factor on an ordinal scale that

indicates degrees of relevancy of the factor as a cause of file format endangerment:

• Not relevant at all • Somewhat relevant • Very relevant

I also asked participants to provide a brief narrative to explain their ratings for each of the

factor options. I also asked participants to provide a brief narrative to explain their ratings

for each of the factor options. Additionally, I asked participants to suggest factors that they

believed to be a cause of file format endangerment that were not included in the original list,

and their rational for suggesting the factors. See Appendix B: Questionnaire Designs,

3.9. Design and Administration of Questionnaires 2 and 3, Round 2

In document 5974.pdf (Page 105-109)