Having developed the survey instrument at the beginning of this research with reference to the literature, it is reviewed and commented on by the expert panel. The overall reliability and validity are then evaluated in terms of item quality with a pilot test, and the final version of questionnaire is then developed and finalised for survey.
A panel of experts, consisting of eight hospitality faculty members and industry professionals, is invited to review the survey instrument. The aim of having this panel is to review the survey instrument and to validate that the survey instrument does seek to collect data to answer the research questions. Questions are re-organised for relevance. Recommendations are offered by the panel to improve the clarity and to increase the scope of the questions. Modification of questions, wording changes and content are then amended following suggestions from the panel to ensure that the questions are suitable for the research and can reflect the current training situation o f the Hong Kong hotel industry.
There are reasons for selecting the members of the expert panel in order to increase the creditability of comments. Professor Cathy Hsu is the research expert in hospitality academic; Dr Andy Lee is the I.T. research expert; Dr Simon Wong has expertise in human resources of the Hong Kong hotel industry; and Dr Alice Hon specialises in research methodology. For industry partners, Ms Daisy Wong and Ms Claire Yau have strong experience in hotel human resources operations; and Ms Serena Chan and Ms Sherine Mak are training experts in high-end hotels in Hong Kong. By both industry and academic standards the comments of the above panel are considered credible and the final version o f the questionnaire valid.
Name Position Organisation
Professor Cathy Hsu Associate Director
and Professor
School o f Hotel and Tourism Management, The Hong Kong
Polytechnic University
Dr Simon Wong Assistant
Professor
School of Hotel and Tourism Management, The Hong Kong
Polytechnic University
Dr Andy Lee Assistant
Professor
School of Hotel and Tourism Management, The Hong Kong
Polytechnic University
Dr Alice Hon Assistant
Professor
School of Hotel and Tourism Management, The Hong Kong
Polytechnic University
Ms Claire Yau Assistant Director
o f Human Resources
Four Seasons Hotel Hong Kong
Ms Daisy Wong Director of
Human Resources
Kowloon Shangri-La Hotel
Ms Serena Chan Training Manager The Peninsula Hong Kong
Ms Sherine Mak Training Manager The Marco Polo Hong Kong
Table 3.3 Profile o f Expert Panel M e m b e r
3.5.2 Validity
The researcher's goal of reducing measurement error can follow several paths. In assessing the degree of measurement error present in any measure, the researcher
must address the validity of a measure (Hair, Jr. et a i, 2010). Validity has been defined by Oppenheim (2000) as the degree to which the scale measures what it intends or supposes to measure. Pallant (2005) added that because validity usually seeks to measure an abstraction, it must take place indirectly and can be established only through empirical tests. Saunders, Lewis & Thornhill (2003) describe validity as being concerned with whether the findings are really what they appear to be about. Broadly speaking, validity refers to the extent to which differences in observed measurement scores reflects true differences in the characteristics being measured (Oppenheim, 2000). Validity, described by Fink and Kosecoff (1996), refers to the accuracy with which the questions represent the characteristics they are supposed to survey.
In conventional usage, the term validity refers to the extent to which an empirical measure adequately reflects the real meaning of the concept under consideration. Carmines and Zeller (1979) discussed three types o f validity: criterion validity, construct validity and content validity. First of all, criterion validity is sometimes called predictive validity. Generally, behaviour may serve as a gauge of criterion validity for the many attitudinal measures found in social science research (Carmines & Zeller, 1979) while Field (2009) defined criterion validity as whether the instrument measures what it claims to measure. Smith and Albaum (2005) explained that in pursuing the objective of criterion validity, the researcher attempts to develop or obtain an external criterion against which the scaling results can be matched. Construct validity, meanwhile, is related to the logical relationships among variables while content validity refers to the degree to which a measure covers the range of meanings included within the concept (Carmines & Zeller, 1979). Content validity refers to the adequacy with which a measure or scale obtained a sample from the intended universe or domain of content while construct validity involves testing a scale not against a single criterion but in terms of
theoretically derived hypotheses concerning the nature o f the underlying variable or construct (Pallant, 2005).
In this study, three abovementioned approaches were taken to ensure that the three types o f validity were achieved as much as possible. First and foremost, all of the constructs of the variables in the study were adapted from previous studies in the literature, except that some revisions were made in the process o f the pilot test. Secondly, the validity was underpinned by inviting personnel of the Hong Kong hotel industry as well as the School of Hotel and Tourism Management to comment on attributes of the constructs. Part of the purpose of the pilot survey was to ensure that each attribute was presented with focus, brevity and clarity. In addition, any possible types of instrumental bias and error were avoided so as to enhance the instrumental validity. Finally, substantial literature was reviewed to ensure that construct validity, which was based on logical relationships among the variables, was enhanced. Balian (1994) reported that content validity is a subjective form of validity evaluation. It consists o f opinion and judgment as the method to derive valid test or survey items. In this research, to ensure the survey instrument's validity, content validity was established through evaluation by a panel of experts. Specifically, the panel expert was asked to evaluate every survey item against the research questions in terms o f its objectivity and whether the item measured the component it was supposed to measure. After the completion of the survey revision, the researcher also conducted a pilot test which will be explained in the pilot test section. Content validity ensures that the measure includes an adequate and representative set of items that measure the concept. The more the scale items represent the domain of the concept being measured the greater the content validity. To ensure content validity only items specifically related to concepts where included in the questionnaire. This is based on the identification of attributes of concepts that is based upon attribute identification by
authors in the literature review. In addition, the questionnaire is subject to a pilot test targeted at the sample population and who are specially asked to comment on the constructs and construct items. Construct validity, on the other hand, testifies to how well the results obtained from the use of the measure (questionnaire) fit the theories around which the test is designed. It is common for the results of a test to be correlated with and compared to the results of tests conducted previously and thought to be valid in this respect. However, in this research, such a test of validity is not possible as no previous data from similar measuring instrument is available, i.e., adoption of computer-based training in the Hong Kong hotel industry. Another possible measure of construct validity is to administer the measuring instrument to a sample population who are not representative of the intended population. Due to time constraints, again, this test is not applied.
3.5.3 Reliability
If validity is assured, the researcher must also consider the reliability o f the measurements (Hair, Jr. et al., 2010). When scales are selected to include in the research it is important to find scales that are reliable. Reliability is an assessment of the degree of consistency between multiple measurements of a variable (Field, 2009). Saunders et al. (2009) describe reliability as the extent to which data collection techniques yield consistent findings while Borg and Gall (1989) described the reliability of an instrument as the level of internal consistency or stability of the measuring device over time. This characteristic, which concerns the purity and consistency o f a measure to repeatability, is always a matter of degree expressed in the form o f a correlation coefficient, and is tested, for the purpose of this study, with the Cronbach's Alpha (a) coefficient utilising SPSS. Cronbach's Alpha coefficient, which is considered to be one of the most commonly used indicators of internal consistency, is computed in terms of
the average inter-correlations among the items measuring the concept (Pallant, 2005). The calculation of the correlation between two splits, or two halves, o f items yields a coefficient between 0 and 1. 0 means no correlation and therefore no internal consistency; and 1 means a perfect correlation and therefore complete internal consistency.
By contrast, reliability means freedom from random error. It is a matter of whether a particular technique, applied repeatedly to the same object, would yield the same result each time. Babbie (1995) stated that the problem of reliability is a basic one in social science measurement, and the suggested test-retest technique for dealing with it. The test-retest method indicates that sometimes it is appropriate to make the same measurement more than once. The measurement method is reliable if the response in both times is the same. Smith and Albaum (2005) describe test-retest technique as an examination of the stability of response. In this study, it was not feasible and economical to use the instrument to question the same respondents more than one time because of time constraints; the respondents were not willing to respond to the same measure, e.g., twice within two months; management o f the responding companies would not support the option to do so because it might have taken up too much of employees' time to complete questionnaires; it would be expensive to conduct an extra measure; and there was the potential problem that the first measurement may have had an effect on the second one.
Threats to the reliability include 1) participation bias/error, and 2) response sets. For the participation bias/error, characteristics of a respondent, for example, age, may affect his/her response. Or the respondent may attempt to give the response desired by the researcher. Since the researcher is not known to the majority o f the respondents, this survey is conducted anonymously and there are no leading questions in this survey. The bias therefore can be kept to a minimum. Other attempts to avoid bias or participant
error included keeping all scales used the same, providing hyperlinks to technical terms and jargon, consistent formatting, questions of similar length, providing feedback on progress to avoid fatigue and avoiding sensitivity. Nevertheless, a response set is the tendency for a respondent to answer a series of questions on a certain direction regardless of their content. This is a potential problem as once a participant begins to agree with a set of questions they select this option without truly reading the question. A technique to avoid this is to reverse one o f the questions. However, as this sometimes may confuse the respondents, this technique was not used in this survey.
In this study, internal consistency coefficients (Cronbach's Alpha) for reliability were determined for the subscales o f Questions 14,18, 20 and 23 of the survey tool that were intended to measure the potential factors that impacted the adoption o f computer- based training. These questions adopted a 5-point Likert scale. The following table indicates the reliability of acceptance scale as measured by Cronbach's Alpha:
Question Concept Construct Cronbach's
Alpha
1 4 Response on current
training
N/A .678
18 Frequency ofCBT adoption Front of the House .911
Back of the House .925
20 Impact of CBT Benefit from technology .629
Employee support .882
Hotel support ■497
2 3 Barrier of CBT adoption Employee attributes .908
Hotel support .813
External environment ■744
Educational measurement experts agreed informally that a minimum accepted level of Alphas is .65 for instruments that provide information about a group of individuals (Frisbie, 1988). Pallant (2005) stated that ideally, this coefficient should be above .7, outlining the sensitivity to the number of items in the scale. Sekaran (2003), in addition, suggested that the closer the Cronbach's Alpha to 1.0, the higher the internal consistency reliability. However, Pallant (2005) further explained that with short scales, for example, scales with fewer than ten items, it is common to find quite low Cronbach value, e.g., .5. Even Briggs and Cheek (1986) recommended an optimal range for the inter-item correlation o f .2 to .4. The Cronbach's Alpha of every construct of Q18 and Q23 are larger than .70, which implies that the internal consistency reliability is high. In Q14, as there are only 10 statements while the Cronbach's Alpha is .678, as per Pallant (2005), this Cronbach's Alpha is acceptable. In Q20, according to Pallant (2005), the Cronbach's Alpha of two constructs, i.e., benefit from technology and employee support, is high. Meanwhile, the Cronbach's Alpha of 'Hotel support' in Q20 is .497 which is comparatively low. This is due to the fact that there are only two statements in this construct; hence, this Cronbach's Alpha cannot truly reflect the consistency. Consequently, the results from the above table indicated that the measure was reliable with appropriate Cronbach's Alpha.
3.5.4 Pilot Test
Pilot test is a small-scale test o f what the survey is to be, including all activities that will go into the final survey (Smith & Albaum, 2005). Fink and Kosecoff (1996) stated that the purpose of a pilot test is to produce a survey that is usable, provide needed information and ensure clear language. The pilot test of this research was conducted in mid November 2009, the main purpose of which was to identify and eliminate possible problems before the main survey, to investigate if respondents faced any difficulties in
the completion of the questionnaire and to check if the questionnaire was too long, in order to allow the revision and refinement of the instrument. This pilot survey was conducted with 30 managers and executives of five Hong Kong hotels in order to obtain feedback on the validity and appropriateness o f the questions. Based on their comments, the questionnaire was further refined. Through this pilot test, the researcher obtained comments regarding the appropriateness of language, presentation and clarity o f the questionnaire. The questionnaire was then found meaningful and the participating subjects provided some positive feedback and comments. Based on these comments, minor revisions were made in wording and layout to improve readability. The results of this pilot test showed that some expressions and grammatical aspects o f the questionnaire needed to be changed to be more comprehensive and to improve clarity. In addition, the respondents suggested reducing the number o f questions which seemed to be repetitive.