Reliability and Validity

Even though retrieving the natural (or relatively natural) verbal speech is valuable for sketching the learners’ acquisition profile in my research, this may raise doubts as to whether the received data represents a true reflection of the actual linguistic processes in the speakers’ minds (Seliger and Shohamy, 1989). As such, the reliability and validity of the research process (including approaches and results) are usually called into question.

Reliability reflects the extent to which a measurement leads to consistent interpretations about a certain instance. The key concerns of the research reliability are: whether the applied methods are reliable, whether these methods could result in the effective data elicitation and whether these independent methods could interact and then lead to identical research results with the same group of research subjects. Therefore, many researchers now utilize the methods which have been employed and verified numerous times in similar research frames or in studies over many years (Chaudron cited in Doughty and Long, 2005; Norris and Ortega cited in Doughty and Long, 2005).

In general, the data in the current study was presented in the format of implicational scaling, which is suitable for ranking linguistic features into a hierarchy (Pienemann cited in Pienemann and Kessler, 2011). Implicational scaling aims to interpret the variables and quantify the relative frequencies of nonstandard variants of all variables (Hatch and Lazaraton, 1991). In order to measure the reliability of the scale, it is thus necessary to calculate its scalability, which ‘can be done by comparing the number of exceptions with the total number of cells contained in the database’ (Pienemann cited in Pienemann and Kessler, 2011: 53). Hatch and Lazaraton (1991) have suggested the

129

coefficiency of scalability which is over 90% can be viewed as significant. According to the current study which includes 15 grammatical rules contained in the CSL table, the results have shown an implicational table with 100% scalability in most cases, except one with 99.2% scalability (Joe’s case).

Additionally, the researchers are recommended to run a pilot study before the actual study is carried out, which allows the insertion of changes, revision and modification of the research procedure. Otherwise, the researcher is taking the risk that the collected data may not point to a valid and reasonable conclusion in the end. If a pilot study is constrained, it is suggested to the researcher that he/she should concentrate on the process of data collection, adjust the research procedure or remove the items that are causing the problem (Seliger and Shohamy, 1989).

A pilot study has been undertaken when designing particular tasks for the Chinese L2 learners aligned within the procedural skills. These tasks are not simply delivered; instead, they are designed and adopted based on the theoretical literature, early studies in PT and task-takers’ language input and output. This pilot study helped me to reconsider the original purpose of each task and then re-evaluate its effectiveness. The revised tasks could better elicit required grammatical structures or necessary speeches among task-takers, which therefore reduced the risks of failure when used for other learners.

Another way to maintain reliability of the study is to extensively train the researchers in order to avoid the mistakes and minimize the problems which can occur in the research procedure. Seliger and Shohamy (1989) have highlighted that training is able to help improve certain skills of the research conductors, such as data recording, question raising and sufficient note-taking. In order to effectively conduct my study, I have taken a series of research methodology courses and training offered by Newcastle University, in which I have been taught different techniques and skills in how to carry out a successful research.

Reliability can ensure the accurate data elicitation, while validity provides information on the extent to which the procedure measures what it is meant to.

130

Validity is a matter of degree. Even though it cannot really be proved, it is still necessary to obtain evidence through the examination of the consistent data (Seliger and Shohamy, 1989).

Validity is divided into two aspects: the internal and external. The internal validity refers to the control of the method selection procedure, such as the task effectiveness and subject selection fit in one research. As explained previously, the tasks selected in my data collection are not random; they have been constructed through literature analysis and previous research designs. Many PT-based studies have proved the validity of the task types used initially by Pienemann (1998c). Therefore, these task types were adapted and then applied to Chinese L2 learners. Besides, the research subjects are all Chinese L2 learners with a variety of language learning backgrounds, which could also guarantee the internal validity of this research design.

In contrast, external validity refers to the degree of practical application of the current research to other similar situations (Hatch and Farhady, 1982). My research is designed according to Zhang (2001 and 2008) and Gao (2005). Even though it is difficult to measure and test the application of my research to other similar contexts due to limited research resources and time, the external validity of this study has to some extent been maintained through these previous studies. In the insightful perspective, the validity of this study has been examined through the longitudinal data collection and the triangulated research methods, alongside interviews, elicitation tasks and questionnaires; further action is taken in textbook analysis and in the CSL task-based teaching plan.

To summarize, as the operator of my study, I am not always alert that the results are sometimes biased, so there are independent experts – two professors in different linguistic perspectives – to supervise my working results and overcome such issues. In fact, my data analysis stage consisted of two steps: the technical preparation and data reorganization, and the distributional analysis. In the distributional analysis, the grammatical tagging of various items was performed and documented. Details will be explained in the next Chapter.

131