Introduction
A mixed methodology was employed in this research, which was based on the conceptual framework derived from the Web accessibility literature and theoretical models. Figure 7 and Table 2 represent the conceptual framework and the hypotheses tested. This research originated first through questioning the levels of present Section 508 conformity of NCATE home pages and the levels reported by the studies by Chilson (2002) and McCullough Stein (2002). These studies had specifically evaluated Section 508 Web accessibility compliance of NCATE institutional and education departments’ home pages. A comparative analysis between the current levels of conformance and these 2002 results was part of this research’s early inquiry. However this became only a component of the study, further examination of other Web accessibility issues
represented by the hypotheses was warranted based on the literature. Therefore, the issues studied, and associated methodology, have evolved.
As discussed, a majority of governmental and public educational Web sites are still not compliant with Section 508 standards even though this legal mandate was
enacted in 1998. Some Web sites have become even less accessible for multiple reasons, one being increased Web site complexity (Flowers et al., 2000; Parmanto & Zeng, 2005). Additionally many studies have found that testing the level of Web accessibility cannot
rely solely on automatic accessibility verification tools, such as Bobby, a widely used tool (Brajnik, 2000; Byerley & Chambers, 2003; Hackett & Parmanto, 2005; Kelly et al., 2004). This tool was utilized in the studies by Chilson (2002) and McCullough Stein (2002), as well as other researchers (Brajnik, 2000; Diaper & Worman, 2003; Takata et al., 2004). Another issue in understanding Web accessibility noncompliance is that more researchers have been advocating that the guidelines used to create Web accessibility are ineffective (Choi, Yi, Law, & Jacko, 2006; Kelly et al., 2005; Phipps & Kelly, 2006). Even if the industry standard guidelines (i.e., WCAG) have been utilized, the Web master still needs to be trained and educated to understand the results to ensure Web accessibility (Curran et al., 2007; Diaper & Worman, 2003; Lazar et al., 2004). Furthermore,
organizational policies and hiring practices to assist in ensuring Web accessibility compliance have been lacking (Byerley & Chambers, 2003; Chilson, 2002; Hudson, 2002; King et al., 2005; McCullough Stein, 2002).
From the literature review, it was apparent that replicating the studies conducted by Chilson (2002) and McCullough Stein (2002) may only show the differences in levels of compliance and organizational Web accessibility policies. This would not have provided new insight as to why their levels of accessibility are more or less compliant to Section 508. Thus the aims had been enlarged in this study to assist in further
understanding the problems with Section 508 conformity based on the literature.
1. Levels of Web accessibility and Section 508 compliance were evaluated of the census and samples generated from a population of 650 NCATE accredited college and university education home pages through the use of automatic verification tools, ACheckerTM, A-PromptTM, manual checks, JAWSTM and KelvinTM.
2. Organizational policies/practices were identified that may affect Section 508 compliance.
3. Web master profiles were formed through survey data and information gathering to identify Web masters’ education/training, communications with end-users, and administrative-level strategic decision-making skills that may affect their production of accessibly-compliant Web sites.
4. Web complexity was analyzed and evaluated using KelvinTM to show affect on Web accessibility.
5. Other possible factors were identified that may affect Section 508 compliance.
Ethical Considerations
Prior to any research activities of this study taking place, ethics in research training and an internal review of the proposed research were employed. To ensure ethical conduct in research, both the researcher and mentor successfully took
Collaborative Institutional Training Initiative (CITI) modules which educate researchers on the history, laws, and ethical behavior and activities in research. An Institutional Review Board (IRB) application was also submitted and accepted and the proposed research was approved prior to conducting this study. Throughout the research, the requirements were fulfilled as provided in the approved IRB application. Web master and institution anonymity were ensured in all of the study’s activities. Only aggregate data and analyses were represented in the research study to ensure this anonymity. Any specialized, confidential reports provided to opting Web masters were confidential and destroyed after provided to each of these Web masters. The data was de-identified after data was collected and coded for SPSS analyses and after sending the Web master confidential reports. The de-identified data and research information will be held for
Rationale
A quantitative research methodology was employed in this study utilizing a survey instrument and other validated tools inclusive of qualitative inquiry to assist in determining levels of Web accessibility and to understand why or why not compliance was met. The reason for a multiple methodological approach was to address the literature which showcased Section 508 conformance to be a complicated problem in
understanding and solving. This research incorporated primary sources to address the Web accessibility problem through secondary instruments: surveys, ACheckerTM, A- PromptTM, JAWSTM, KelvinTM, and a job-advertisement analysis.
The first step was to generate the base of information through the analyses of the NCATE education department home pages with ACheckerTM (version 0.8.9) on whether these home pages were Section 508 compliant. The use of the validated Web
accessibility verification tool, ACheckerTM (and A-PromptTM to verify ACheckerTM
results), and the use of JAWSTM testing to analyze Section 508 compliance were based on previous studies. The complementary testing using JAWSTM provided additional insights in the need for Web accessibility testing tools beyond automatic verification tools, such as ACheckerTM, A-PromptTM, Bobby, and so forth. The researcher used a survey based on Stewart et al. (2005) to analyze Web pages with the JAWS TM tool as well as tested each conditional pass item in need of manual testing as recommended by ACheckerTM.
Hypotheses 1 and 2 were tested investigating the presence of organizational support provided through Web accessibility policies/guidelines and Web master job advertisements seeking Web accessibility skills/training respectively. This was
conducted through the researchers’ Web site searches and acquiring the policies and job advertisements. The job-advertisement analysis was completed based on Wade and Parent’s (2002) methodology to determine organizational importance of hiring Web masters with specific skill sets, in this case Web accessibility skills, via hiring practices.
To assist in testing H3, H4, and H5, a Web master survey was mailed to gather Web master demographics and skills, education, training and Web user communications that related to Web accessibility, as well as the presence of organizational policies. This survey comprehensively involved previous validated Web master survey questionnaires (Chilson, 2002; Lazar et al., 2004; McCullough Stein, 2002; Wade & Parent, 2002) to produce quantitative and qualitative data and information. The Web master survey component encompassed a mailer inclusive of a letter of introduction (Appendix A), Web master survey (Appendix B), and SASE. Two of the same mailings were sent out to the population. The first generated 57 responses. The second mailing generated 40 more responses for a total of 97 respondents in that sample. After Web master responses were generated and analyzed from the survey component of the research, a correlation analysis was conducted of H3, H4, and H5. Additional information was also generated from the Web master survey responses from the qualitative questioning. To test H6, University of Pittsburgh researchers’ WAB and Complexity Scoring and the effect on Web
accessibility (Parmanto & Zeng, 2005) was conducted using KelvinTM. This determined WAB scores to determine the levels of Web accessibility to use as measures for all hypotheses. Web master and institutional anonymity were ensured as previously discussed. Figure 8 represents the research methodology.
Gathered NCATE contact information: 1. Collected NCATE institutions’ URLs of
education department’s home page information and web master’s name and contact info.
2. Completed Web accessibility analyses on population.
Gathered and entered survey data and information: 1. As hard copy and electronic data arrived,
entered/imported info into an Excel and SPSS files. 2. After a one month period after mailing the hard copy
survey, sent a reminder e-mail with letter and attached survey to Web masters. Conducted a second mailing in January 2009.
Completed job-advertisement analysis:
Found NCATE Web master job postings on NCATE Web sites and job search engines. Completed job- advertisement analysis and compared to results of Web master survey responses - based on Wade and Parent’s (2002) job-content analyses.
Analyzed the quantitative data with SPSS: 1. Tested reliability of survey questions specific to
each hypotheses 3, 4, and 5 using Cronbach alpha analyses.
2. Use t-Tests to test hypotheses 1 and 2. 3. To test hypotheses 3, 4, 5, and 6, Pearson
correlation analyses were conducted.
REPORTING
1. Results 4. Concluding remarks and future 2. Implications research recommendations 3. Limitations
CENSUS & SAMPLES TESTING After the two mailings, a significant sample from survey responses was generated (N = 97). Created initial surveys:
1. Created hard copy Web master survey and a secure online alternative
(https://www.section508accessible.org/surve y) for Web masters to submit data and info using a username/password
2. Created JAWSTM analyses survey questions – this survey was completed by the researcher when utilizing JAWSTM
Reported qualitative data and information: The qualitative data from the Web master survey was compared with previous qualitative data generated by other Web master survey research. Generated information for future studies.
Created and completed a bulk mailing with corresponding Web survey information: 1. Created spreadsheet with contact info 2. Finalized mailing materials
3. Completed mailing between June and July, 2008, and January 2009.
Figure 8. Methodology for Web accessibility flow chart. For this research. J.A. Smith, 2009.
Completed field tests with non-population higher educational institutions:
1. Field tested Web master survey questions (both hard copy and online. Edited the survey based on responses.
2. Field tested tools, ACheckerTM, APromptTM, JAWSTM for Web accessibility.
3. Field tested using WAB and Complexity Scoring methodologies.
4. Field tested using job-content analysis methodology.
Completed tests of NCATE education department home pages:
1. Tested using ACheckerTM and A-PromptTM Section 508 guidelines on census and sample of education department’s home pages.
2. Used JAWSTM to answer Stewart et al. (2005) survey questions and manual HTML checks recommended by ACheckerTM for conditional passes to further assess home page accessibility. 3. Used KelvinTM for WAB and Complexity Scoring to
analyze Web page complexity. Formalized proposal and IRB application
for approval:
Completed all necessary CITI/IRB modules and produced final draft proposal and IRB application for mentor, committee, and IRB for approval (revised and resubmitted, as necessary), and completed conference call.
AFTER APPROVALS RESEARCH BEGAN
Secondary Survey Tools and Data Gathering Instruments
Secondary survey instruments were utilized to conduct primary research. The Web master survey was based on pretested surveys and another for use in the JAWSTM home page testing by researcher. The Web master survey utilized the questions derived from Chilson’s (2002) and McCullough Stein’s (2002) research related to organizational Web accessibility policies. The Web master demographics and experience and skill set data were generated through survey questions from two other studies by Lazar et al. (2004) and Wade and Parent (2002). The Web master survey is represented in Appendix B. Appendices C and D show which questions were attributed to each previous study and this study’s associated hypotheses (Appendix C), as well as why certain questions from previous studies were not included in the survey (Appendix D). The survey captured information on Web master education, training, skills, user communications, and administrative-level strategic decision-making responsibilities as well as organizational Web policy data generated from the Web master survey tools previously discussed (see Appendix C). The Web master skills, education, and training were central to both Lazar et al.’s (2004) and Wade and Parent’s (2002) survey research. However, the Web master administrative/organizational skills were more focused in Wade and Parent’s survey research. Additionally, Web master communications with the end-user were important to both of these studies as well as many of the studies in the literature review.
The pretested survey questions for the JAWSTM home page testing was based on the study conducted by Stewart et al. (2005) which can be found in Appendix E. As previously stated, assistive technology can assist Web masters in ensuring Web
compliance to Section 508. The assistive technologies were central to Leung et al.’s (1999) work and the literature, particularly with many studies utilizing screen readers, which is why JAWSTM was employed. JAWSTM has been a popular screen reader technology for Web navigators with visual impairments and was utilized in the literature by Web masters and users with disabilities. This assistive technology program was used by the researcher to answer the Stewart et al. (2005) questions rather than users with disabilities. The reason the researcher completed the questions rather than users with disabilities who are blind and can navigate with JAWSTM was the unavailability of this population for this study. The testing was a heuristic test as the researcher has expert knowledge in the use of this screen reader. Furthermore, research studies have
recommended the use of assistive programs and means by the Web masters themselves to discover Web accessibility issues if the user with the disability may not be available for testing the Web site (Axtell & Dixon, 2002; Byerley & Chambers, 2003; Flowers et al., 2000; Kaplan-Leiserson, 2000). Also to ensure the conditional passes generated by ACheckerTM were actual Section 508 passes, JAWSTM and analyzing the Web pages’ HTML code were done to conduct the ACheckerTM recommended manual checks.
The measurement instruments including the aforementioned secondary survey tools were chosen based on the literature and models. Each hypothesis and its
measurement tool for testing are described later in this paper. ACheckerTM and A- PromptTM were used and are two of many online automatic accessibility tools used frequently in this field of research. They have been created and administered by the University of Toronto (Adaptive Technology Resource Centre, 2008). The research
initially was to utilize Bobby as the automatic Web accessibility verification tool as it has been widely utilized in the research. However, online Bobby transformed into
WebXACT, which is no longer available (Watchfire, 2008). Additionally, according to Diaper and Worman (2003), A-PromptTM had provided better results than the commonly used Bobby tool when checking priority one compliance that reflects Section 508
compliance. A-PromptTM was a tool created by the University of Toronto, which released its new and improved tool, ACheckerTM. ACheckerTM has been deemed to be a better tool to use in checking for Section 508 compliance (Adaptive Technology
Resource Centre, 2008). ACheckerTM was recommended by the project manager of both A-PromptTM and ACheckerTM for this type of research (Appendix F). A-PromptTM was used as an additional tool to verify Section 508 results by ACheckerTM. A-PromptTM is a desktop program, whereas, ACheckerTM is online, but both provide results on conditional passing and failing of Section 508 guidelines.
Another secondary tool that was utilized in this study was KelvinTM to retrieve home page WAB scoring and complexity analyses, which have their history with the University of Pittsburgh. WAB score calculations were tested by Hackett et al. (2005), Hackett and Parmanto (2005), Parmanto and Zeng (2005), and Zeng and Parmanto (2004). WAB provides the level of Web accessibility on a numerical basis (most cases values ranged from at least one to values over one hundred) rather than a pass (1) or fail (0) test. These scores were used for correlation analyses for H3, H4, H5, and H6 to provide richer data in the analyses compared to pass (1) or fail (0).
The last secondary tool was job-advertisement analyses by Wade and Parent (2002) and generated data and information rating the skill usefulness of Web accessibility designing and programming by higher education through the analysis of Web master job advertisements found from the NCATE population. These job postings were retrieved mostly on the NCATE Web sites, but also through search engines (Google.com and Monster.com).
Validity and Reliability
The choice of the self-administered, mailed Web master survey questionnaire was due to convenience, cost, and the high rate of use in the research community as discussed by Robson (2002). This survey further followed the reasoning provided by Robson (p. 229-230) in that the questions used were fixed and part of a quantitative design (e.g., yes/no/not applicable or Likert). However, a few questions posed by Lazar et al. (2004) were qualitative in nature for information generation that provided further insight regarding issues related to Web accessibility. The survey was also the means for collecting data that was standardized from a large population (N = 650) to help obtain a random sample from those who responded. The Web masters were employees,
volunteers, or contractors of the population of organizations accredited by NCATE as listed by NCATE at their Web site: www.ncate.org. This was a cross-sectional design as the process of sending, receiving, and collecting the data from the mailers occurring once in July/June of 2008 and a second time in January 2009, but was considered analyses at a single point in time to “discern patterns of association” (Bryman, 1989, p. 104) within that population or sample of a population. (Robson, 2002, p. 229-230).
Experiments were not feasible for acquiring data from the Web masters in their place of work. The survey was the tool to gather data and information from the NCATE Web masters otherwise largely unavailable for this study. The internal validity of the survey was based on this study’s use of secondary instruments, pretested by other researchers with Web masters as survey respondents. The questions from these survey tools were worded to evade ambiguity or misunderstanding, which assists internal
validity (Robson, 2002, p. 230). Field testing on higher education Web masters not in the NCATE population using the survey instrument was done to ensure this. Sampling of the population was externally valid by contacting every NCATE education Web master explaining the study to the education Web masters, confirming that their specific results would remain anonymous and they would be provided a confidential report specific to their organization, if they desired. All responders (i.e., NCATE education departments’ Web masters) who provided survey responses would create the random sample to test H3, H4, and H5. This procedure assisted in generating generalizability so that results could be applied to other NCATE or higher educational institutions.
By field testing and providing the survey mailings to the NCATE population and basing questions on pretested surveys (Chilson, 2002; Lazar et al., 2004; McCullough Stein, 2002; Wade and Parent, 2002), reliability of the survey data was high. Robson (2002) stated, “Reliability is more straightforward. By presenting all respondents with the same standardized questions, carefully worded after piloting, it is possible to obtain high reliability of response” (p. 231, para. 5). Although this research did not conduct a pilot study (i.e., prior study on the population), field testing was done using other post-
secondary institutions not in the NCATE population studied. Also, the Web master survey consistently utilized the same questions, as well as had each question being