Introduction
This chapter explains the research data collection and analyses which provided base data and information on the population, and data from collected samples for testing the six hypotheses. The information is provided first as to what data was collected. Secondly the reliability testing of the data collection instrument and the survey
questionnaire is detailed. Explanation is then provided of the examinations of the Web pages using ACheckerTM, A-PromptTM, JAWSTM, KelvinTM, and manual searches to produce data related to Section 508 Web accessibility. Data collected from a secondary sample through a job-advertisement analysis of Web master job advertisements is also described. Lastly the hypotheses tested and attributed results are presented and then summarized.
Data Collection
From the summer of 2008 through spring of 2009, data was collected from respondents of a survey mailing that was sent addressed to the Web masters of education departments of institutions that are accredited by NCATE regarding their Web sites, Web master job, and their organization. The NCATE population of 650 post-secondary
educational institutions and their affiliated education department addresses were found on the NCATE Web site (www.ncate.org) and at each institution’s Web site during the
months of June and July 2008. One hard copy mailing was conducted via the US Postal Service in July 2008 and had 57 responses. In August of 2008, a follow-up e-mailed mailing with the survey as an attachment only produced 1 response. To ensure a
significant sample size, a second, hard copy mailing was conducted January 2009. This last survey mailing produced 40 more responses. In total, 97 Web masters responded to the survey, which was a 14.9% response rate, and constituted the sample of the
population for testing H3, H4, and H5. These Web masters’ education departments’ home pages were tested for Section 508 compliance using ACheckerTM and A-PromptTM and levels of WAB using KelvinTM. Additional data, such as demographics and
qualitative responses, was collected through the survey to provide richer information on the Web masters.
To test H2, data was generated through a job-advertisement analyses that was through finding job postings for positions in the NCATE population that had Web master responsibilities to determine whether or not these ads stated preferences or requirements of Web accessibility knowledge and/or training. There were 43 NCATE institutions found that currently had job advertisements where the position had Web master
responsibilities. The data generated of WAB scores of these education home pages and whether or not Web accessibility language was present in the job advertisements was used to test H2. In addition, the entire population was studied regarding the level of Web accessibility associated with a presence of Web accessibility guidelines/policies to test H1 as well as the complexity of the education home page to test H6.
Survey Questionnaire Reliability
Before analyzing the response data and testing H3, H4, and H5, the survey was tested for reliability. SPSS was used to find the Cronbach alpha coefficients for each of the answers per question group construct pertinent to test a specific hypothesis. The answers to questions 17-28 to test H3 had a Cronbach alpha coefficient of 0.944 (Table 10) and all values in the inter-item correlation matrix were positive (Table 11).
Table 10.
Reliability Statistics for Survey Questions 17-28
Cronbach's Alpha Cronbach's Alpha Based on Standardized Items N of Items .944 .944 12 Table 11.
Inter-Item Correlation Matrix for Questions 17-28
Q17 Q18 Q19 Q20 Q21 Q22 Q23 Q24 Q25 Q26 Q27 Q28 Q17 1.00 .649 .711 .608 .507 .600 .525 .373 .418 .521 .532 .545 Q18 .649 1.00 .742 .582 .561 .670 .639 .375 .518 .658 .651 .666 Q19 .711 .742 1.00 .719 .692 .702 .652 .400 .514 .687 .682 .685 Q20 .608 .582 .719 1.00 .456 .564 .578 .420 .373 .568 .520 .514 Q21 .507 .561 .692 .456 1.00 .588 .557 .423 .457 .547 .584 .653 Q22 .600 .670 .702 .564 .588 1.00 .788 .450 .506 .685 .699 .723 Q23 .525 .639 .652 .578 .557 .788 1.00 .483 .560 .642 .645 .673 Q24 .373 .375 .400 .420 .423 .450 .483 1.00 .407 .433 .402 .455 Q25 .418 .518 .514 .373 .457 .506 .560 .407 1.00 .535 .604 .607 Q26 .521 .658 .687 .568 .547 .685 .642 .433 .535 1.00 .953 .881 Q27 .532 .651 .682 .520 .584 .699 .645 .402 .604 .953 1.00 .924 Q28 .545 .666 .685 .514 .653 .723 .673 .455 .607 .881 .924 1.00
The answers to questions 29 through 32 to test H4 had a Cronbach alpha coefficient of 0.881 (Table 12) and all values in the inter-item correlation matrix were positive (Table 13).
Table 12.
Reliability Statistics for Survey Questions 29-32
Cronbach's Alpha Cronbach's Alpha Based on Standardized Items N of Items .881 .885 4 Table 13.
Inter-Item Correlation Matrix for Questions 29-32
Q29 Q30 Q31 Q32
Q29 1.000 .620 .562 .542
Q30 .620 1.000 .677 .690
Q31 .562 .677 1.000 .854
Q32 .542 .690 .854 1.000
The answers to questions 3 through 10 to test H5 had a Cronbach alpha coefficient of 0.798 (Table 14) and all values in the inter-item correlation matrix were positive (Table 15).
Table 14.
Reliability Statistics for Survey Questions 3-10
Cronbach's Alpha Cronbach's Alpha Based on Standardized Items N of Items .798 .803 8
Table 15.
Inter-Item Correlation Matrix for Questions 3-10
Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q3 1.00 .353 .435 .420 .387 .252 .218 .096 Q4 .353 1.00 .396 .422 .250 .295 .315 .333 Q5 .435 .396 1.00 .420 .429 .414 .413 .220 Q6 .420 .422 .420 1.00 .262 .194 .223 .331 Q7 .387 .250 .429 .262 1.00 .472 .474 .339 Q8 .252 .295 .414 .194 .472 1.00 .336 .266 Q9 .218 .315 .413 .223 .474 .336 1.00 .507 Q10 .096 .333 .220 .331 .339 .266 .507 1.00
Based on Nunnaly (1978), all of these alpha coefficients were high (i.e., greater than 0.70). A high alpha and positive values in the inter-item correlation matrix show strong inter-correlation and reliability (Furr & Bacharach, 2007). Based on the reliability of these multiple-item constructs, further analyses and hypothesis testing occurred. Survey Questions
The survey questionnaire in Appendix B with answers in Appendix N organized 52 questions into six sections on a two-page, double-sided form that was mailed with a cover letter (Appendix A and B). Appendix N provides each question under each of the sections. Section one began after the first question asking employment type and covered the Web master’s technical and organizational skills. The next section addressed Web accessibility pertinent to their Web master role. The third section comprised the level of importance of communicating with end-users. The fourth section addressed contact and policy information specific to Web accessibility issues. The fifth section sought
demographic data and information about the Web master and his/her institution. The last section sought additional, qualitative data and information.
Web masters were provided the opportunity to fill out the survey online on the Web page (www.section508accessible.org/survey) or on paper and sent back in a self- addressed stamped envelope. Their access code, which became their institutional identifier, and login and password were provided at the top of their survey form. In addition, the Web master had the opportunity to check whether they would like to receive a free, confidential Web accessibility report specific to their organization after
completion of the research. The response rate to this question was also gathered and tabulated. Over half (56 of the 97 respondents, i.e. 57.7%) desired to receive a free, confidential report.
Each question, other than questions 21 and 48, were based on other survey research (Chilson, 2002; Lazar et al., 2004; McCullough Stein, 2002; Wade & Parent, 2002) which targeted Web masters as listed in Appendix C. Questions 21 and 48 are based on the World Wide Web Consortium (2006) and Guenther (2005) respectively. Certain questions from the previous surveys were not included as specified in Appendix D. Reducing the number of questions was critical in helping increase the response rate. However, additional questions were included in the survey that were not intended to test hypotheses, but gather demographics, as well as more rich data, and some subjective, responses from the Web master. Questions related to hypothesis testing and those for additional information are noted in Appendix C.
To quantify and analyze the results of this survey, data was converted into
numbers and created in an SPSS file. Appendix N lists the questions with corresponding answers and the values given to each of the answers for SPSS analysis. For example, Likert answers had the following numerical associations: 0 for no answer (i.e., missing
data), 1 for no extent, 2 for little extent, 3 for some extent, 4 for great extent, and 5 for very great extent. The SPSS file configured the variables for missing data, so entire surveys were not thrown out based on one or more answers missing. The quantitative analyses, as previously noted, began with reliability tests of the multiple-item constructs, which were specific questions related to testing H3, H4, or H5. Pearson correlations were run on the Likert questions related to H3, H4, and H5 with the associated education departments’ home page WAB scores (as tabulated by KelvinTM) for hypothesis testing. Descriptive statistics were run on the additional quantitative data. Qualitative data and information were grouped into obvious categories for reporting and discussion.
Descriptive Statistics
This section provides the descriptive statistical results of each question of the survey. Analyses were completed through the use of the statistical program, SPSS. All answers to questions were normally distributed. This section is followed by the summary of the qualitative questions.
Question 1. Q1 in Appendix N represents question one in the survey, check your job classification. It shows the frequency in job classifications of the Web masters: no answer (5.2%), employee (94.8%), consultant (0%), independent contractor (0%), volunteer (0%), other (0%). All who responded were employees. Five of the 97 respondents did not respond to this question (missing = 5, n = 92).
Question 2. Q2 in Appendix N represents question two in the survey, to what extent is it useful for you to have good technical skills? The results were 48.5% very great extent, 39.2% great extent, 11.3% some extent, 1% little extent, and 0% for no
extent (missing = 0, n = 97). The majority (87.7%) thought this was important from a great to very great extent.
Question 3. Q3 in Appendix N represents question three in the survey, to what extent is it useful for you to have good management skill? The results were 30.9% very great extent, 42.3% great extent, 25.8% some extent, 1.0% little extent, and 0% for no extent (missing = 0, n = 97). The majority (73.2%) thought this was important from a great to very great extent.
Question 4. Q4 in Appendix N represents question four in the survey, to what extent is it useful for you to manage projects? The results were 1% no answer, 35.1% very great extent, 43.3% great extent, 18.6% some extent, 1.0% little extent, and 1.0% for no extent (missing = 1, n = 96). The majority (78.4%) thought this was important from a great to very great extent.
Question 5. Q5 in Appendix N represents question five in the survey, to what extent is it useful for you to work effectively in groups? The results of were 28.9% very great extent, 33.0% great extent, 32.0% some extent, 4.1% little extent, and 2.1% for no extent (missing = 0, n = 97). The majority (94%) were equally split of this being important from some extent to a great extent to a very great extent.
Question 6. Q6 in Appendix N represents question six in the survey, to what extent is it useful for you to communicate effectively with others? The results were 55.7% very great extent, 41.2% great extent, 3.1% some extent, 0% little extent, and 0% for no extent (missing = 0, n = 97). The majority (96.9%) thought this was important from a great to very great extent.
Question 7. Q7 in Appendix N represents question seven in the survey, to what extent do you have a supervisory role in this post-secondary educational institution? The results were 9.3% very great extent, 16.5% great extent, 33.0% some extent, 25.8% little extent, and 15.5% for no extent (missing = 0, n = 97). Over half (58.8%) stated that they had only some or little extent in a supervisory role with the rest split between the other categories.
Question 8. Q8 in Appendix N represents question eight in the survey, to what extent is it useful for you to recognize and manage personality problems that interfere with job completion? The results were 14.4% very great extent, 23.7% great extent, 44.3% some extent, 13.4% little extent, and 4.1% for no extent (missing = 0, n = 97). The majority (68%) thought this was important from some extent to a great extent.
Question 9. Q9 in Appendix N represents question nine in the survey, to what extent do you play a central role in determining the Web strategy for your post-secondary educational institution? The results were 1% no answer, 19.6% very great extent, 24.7% great extent, 33.0% some extent, 12.4% little extent, and 9.3% for no extent (missing = 1, n = 96). The majority (57.7%) had some extent to a great extent involvement in Web strategy with the rest split between the other categories.
Question 10. Q10 in Appendix N represents question ten in the survey, to what extent do you think that your Web master work has contributed to the strategic Web goals [of your post-secondary educational institution]? The results were 2.1% no answer, 14.4% very great extent, 35.1% great extent, 40.2% some extent, 5.2% little extent, and 3.1% for no extent (missing = 2, n = 95). The majority (75.3%) thought their work contributed to the strategic Web goals from some extent to a great extent.
Question 11. Q11 in Appendix N represents question 11 in the survey, to what extent do you think you have been successful in your position as a Web master? The results were 0% no answer, 13.4% very great extent, 51.5% great extent, 34.0% some extent, 1.0% little extent, and 0% for no extent (missing = 0, n = 97). The majority (85.5%) thought they were successful as a Web master from some extent to a great extent with over half being a great extent.
Question 12. Q12 in Appendix N represents question 12 in the survey, to what extent is your Web site subject to the US Section 508 rules on Web accessibility? The results were 5.2% no answer, 22.7% very great extent, 32.0% great extent, 26.8% some extent, 7.2% little extent, and 6.2% for no extent (missing = 5, n = 92). The majority (58.8%) thought their Web site was subject to US Section 508 rules between some extent to a great extent.
Question 13. Q13 in Appendix N represents question 13 in the survey, to what extent do your university/college policies address Web site development? The results were 4.1% no answer, 11.3% very great extent, 35.1% great extent, 34.0% some extent, 11.3% little extent, and 4.1% for no extent (missing = 4, n = 93). The majority (69.1%) thought their institutional policies addressed Web development from some extent to a great extent.
Question 14. Q14 in Appendix N represents question 14 in the survey, to what extent do your university/college policies address Section 508 Web accessibility? The results were 7.2% no answer, 9.3% very great extent, 25.8% great extent, 35.1% some extent, 14.4% little extent, and 8.2% for no extent (missing = 7, n = 90). The majority
(60.9%) thought their institutional policies addressed Section 508 Web accessibility from some extent to a great extent.
Question 15. Q15 in Appendix N represents question 15 in the survey, to what extent do your university/college policies address Web accessibility development tools and/or techniques, such as the use of templates, Bobby, ACheckerTM, or other automatic verification tools, or assistive technologies, such as screen readers, e.g., JAWS? The results were 6.2% no answer, 5.2% very great extent, 20.6% great extent, 33.0% some extent, 21.6% little extent, and 13.4% for no extent (missing = 6, n = 91). The majority (75.2%) thought their institutional policies addressed Web accessibility tools from a little extent to some extent to a great extent.
Question 16. Q16 in Appendix N represents question 16 in the survey, to what extent do your university/college policies address regular or periodic Web site analyses to ensure Section 508 conformance? The results were 9.3% no answer, 3.1% very great extent, 13.4% great extent, 29.9% some extent, 22.7% little extent, and 21.6% for no extent (missing = 9, n = 88). Over half (52.6%) thought their institutional policies addressed regular or periodic Web site conformance from a little extent to some extent and 21.6% stated no extent.
Question 17. Q17 in Appendix N represents question 17 in the survey, to what extent when making updates to your Web site, do you consider the importance of making the site accessible to all users? The results were 4.1% no answer, 24.7% very great extent, 34.0% great extent, 27.8% some extent, 6.2% little extent, and 3.1% for no extent (missing = 4, n = 93). The majority (86.5%) thought their precedence of making the Web site useful to all ranged from some extent to a great extent to a very great extent.
Question 18. Q18 in Appendix N represents question 18 in the survey, to what extent are you knowledgeable with U.S. Section 508 of the Americans with Disabilities Act (ADA)? The results were 3.1% no answer, 6.2% very great extent, 28.9% great extent, 42.3% some extent, 15.5% little extent, and 4.1% for no extent (missing = 3, n = 94). The majority (71.2%) thought their level of knowledge of this ranged from some extent to a great extent.
Question 19. Q19 in Appendix N represents question 19 in the survey, to what extent are you familiar with creating Web sites that are accessible for users with visual impairments? The results were 4.1% no answer, 7.2% very great extent, 30.9% great extent, 34.0% some extent, 12.4% little extent, and 11.3% for no extent (missing = 4, n = 93). The majority (64.9%) thought their level of creating Web sites accessible to users with visual impairments ranged from some extent to a great extent.
Question 20. Q20 in Appendix N represents question 20 in the survey, to what extent is the Web site that you are currently overseeing accessible to users with visual impairments? The results were 6.2% no answer, 5.2% very great extent, 30.9% great extent, 30.9% some extent, 14.4% little extent, and 12.4% for no extent (missing = 6, n = 91). The majority (61.8%) thought that the level of their Web site that was accessible to users with visual impairments ranged from some extent to a great extent.
Question 21. Q21 in Appendix N represents question 21 in the survey, to what extent are you familiar with designing Web sites for mobile device accessibility (e.g., iPhonesTM, personal digital assistants (PDAs), or cell phones, etc.)? The results were 4.1% no answer, 4.1% very great extent, 6.2% great extent, 21.6% some extent, 33.0% little extent, and 30.9% for no extent (missing = 4, n = 93). The majority (63.9%) thought
their level of familiarity of designing Web sites accessible to mobile devices ranged from no extent to little extent.
Question 22. Q22 in Appendix N represents question 22 in the survey, to what extent are you knowledgeable regarding available software tools that check your Web site for accessibility and provide useful feedback? The results were 5.2% no answer, 5.2% very great extent, 18.6% great extent, 35.1% some extent, 18.6% little extent, and 17.5% for no extent (missing = 5, n = 92). Over a third (35.1%) thought their level of knowledge using tools for accessibility checking was some extent with a tie of responses (18.6%) between a level of little extent and a great extent.
Question 23. Q23 in Appendix N represents question 23 in the survey, to what extent have you used any of the following Web-based accessibility tools: online Bobby, WebXACT, ACheckerTM, or a similar online tool? The results were 5.2% no answer, 4.1% very great extent, 16.5% great extent, 26.8% some extent, 12.4% little extent, and 35.1% for no extent (missing = 5, n = 92). The leading response (35.1%) thought their level of using online Web accessibility tools was no extent with the second leading response (26.8%) being some extent.
Question 24. Q24 in Appendix N represents question 24 in the survey, to what extent have you used any of the following non-Web-based accessibility tools, e.g., A- PromptTM, desktop Bobby, Dreamweaver with accessibility checks? The results were 4.1% no answer, 2.1% very great extent, 13.4% great extent, 30.9% some extent, 18.6% little extent, and 30.9% for no extent (missing = 4, n = 93). A tie (30.9%) occurred between the level of using non-Web-based tools of no extent and some extent.
Question 25. Q25 in Appendix N represents question 25 in the survey, to what extent have you tested your Web site using a screen reader, e.g., JAWS? The results were 5.2% no answer, 1.0% very great extent, 3.1% great extent, 17.5% some extent, 16.5% little extent, and 56.7% for no extent (missing = 5, n = 92). Over half (56.7%) thought their level of using a screen reader to test Web accessibility was no extent.
Question 26. Q26 in Appendix N represents question 26 in the survey, to what extent are you knowledgeable with the first set of guidelines (Priority 1) originating from the Web accessibility Initiative (www.w3.org/wai)? The results were 8.2% no answer, 3.2% very great extent, 13.4% great extent, 18.6% some extent, 21.6% little extent, and 32.0% for no extent (missing = 8, n = 89). Over half (53.6%) thought their level of knowledge about Priority 1 Web accessibility guidelines ranged from no extent to little extent.
Question 27. Q27 in Appendix N represents question 27 in the survey, to what extent are you knowledgeable with the second set of guidelines (Priority 2) originating from the Web accessibility Initiative (www.w3.org/wai)? The results were 7.2% no