The Translation Task: The Adopted Marking Method

The method adopted to mark the translation task was similar, to some extent, to that already described in the previous section for the GJ task. They do differ from each other in certain ways, however. Given the nature of the translation task, which is designed to test the production of grammaticality, not the intuitive knowledge of grammaticality and ungrammaticality the way the GJ task did, the participant’s translation to each sentence was marked on the basis of the following two marking criteria:

1. Whether he or she produced the relevant syntactic structure for the analysis or not – namely, whether the construction produced contains an embedded adverbial or complement clause or instead contains a clause or structure irrelevant to the analysis. Note that native speakers in such sentential contexts usually use a complex sentence with either an embedded adverbial or complement clause¹⁰²

2. Whether he or she managed to supply the pronominal functional category and/or the agreement morphology under investigation in this study – namely, whether he or she provided the required overt embedded subject pronoun and the subject–verb agreement inflection

102 It was noticed in the pilot studies that native speakers on rare occasion used the gerund form with some of the embedded adverbial clauses in the GJ task when correcting the ungrammatical sentences. Such translated construction will be considered as correct target translation. This is because using the gerund form in this particular position indicates that the learner has realised that English cannot have a null pro subject in this syntactic position.

This marking method was carried out, again using Microsoft Excel 2010, so that each translated sentence was evaluated according to the following five marking options that cover all the learners’ possible translation performances:

1. Target-like structure – correct suppliance of the investigated form (CS) 2. Target-like structure – incorrect suppliance of the investigated form (IS) 3. Non-target-like structure/unexpected translation (DS)

4. Unanalysable translated sentence (US) 5. Missing translation (NA).

The rationale behind implementing this marking method was again to ensure that the performance for every participant is accurate enough to meet the purpose the test was designed for – to reflect the participant’s IL. To meet this purpose, the results must be reliable. Ideally, the principles of this marking method enhance the reliability of the data obtained by making it possible for the researcher to properly deal with the data that can bias or invalidate the findings of the study. For this reason, all translational performance that neither can add information about the learner’s IL (i.e., unanalysable-translated sentence [US] or missing translation [NA]) nor is structurally relevant to the study (i.e., non-target-like structure [DS]) will be removed from the statistical analysis. The following table illustrates which of these possible translational performances are including in the statistical analysis and which are not.

Table 4-11. Possible translational performances: included performances vs.

excluded performances

Classification

Participants’ Possible Translational Performances

CS IS DS US NA

Included Performances  

Excluded Performances   

Key:

CS: Target-like structure – correct suppliance of the investigated form IS: Target-like structure – incorrect suppliance of the investigated form DS: Non-target-like structure/unexpected translation

US: Unanalysable translated sentence NA: Missing translation

Furthermore, this marking method allows the researcher to exclude participants who did not complete the task as expected in a convincing, systematic way, following

the principles of the 20% excluding criterion formulated in the GJ task. Consequently, any participant whose excluded performance (US, NA, DS) exceeded 20% of his or her translational performance (4+ out of 15 sentences) was removed from the analysis.

These answer codes (CS, IS, US, NA, DS) used in the marking process need to be transformed into numerical scores for the purpose of statistics. This process of transforming answers into numbers was done by following the same process performed to deal with the participant’s reactions to the grammatical and ungrammatical sentences in the other task, where initially the participant was given one point for each evaluated translational performance, regardless of its structural relevance or correctness. After that, these points for each type of the five possible translational performances were calculated for every participant using Microsoft’s Excel 2010 programme. Then, to discover whether a participant should be excluded from the analysis according to the 20% excluding criterion, the participant’s scores for the three excluded translational performances (US, NA, DS) were added together.

The following tables offer an overview of these marking, calculation, and exclusion processes:

Table 4-12. Method used to mark the translation test

Participant No. Translational Performances to the Test Sentences

S1 S2 S3 … S13 S14 S15

FN P18 CS CS IS … NA CS CS

FN P36 CS CS CS … NA NA NA

FR P119 CS CS CS … CS CS CS

FR P175 CS CS CS … NA NA NA

AR P258 CS CS CS … CS CS CS

AR P283 IS CS CS … CS CS NA

Key:

CS: Target-like structure – correct suppliance of the investigated form IS: Target-like structure – incorrect suppliance of the investigated form DS: Non-target-like structure/unexpected translation

US: Unanalysable translated sentence NA: Missing translation

Table 4-13. Calculation procedures used to exclude performances and participants in the translation task

Participants

Number and Percentage of the Excluded Translational Performances

Exclusion Null Subject

(Non-TS, UTS, MT) Percentage

FN P18 1 07 % Included

FN P36 6 40 % Excluded

FR P119 0 00 % Included

FR P175 4 27 % Excluded

AR P258 0 00 % Included

AR P283 1 07 % Included

These tables summarise the different processes involved in marking the translation task. Table 4.12 shows how the different translational performances were initially marked, whereas Table 4.13 illustrates the implemented participant-excluding criterion (i.e., participants FN-P36 and FR-P175 were removed from the analysis as their excluded individual performance [US, NA, DS] exceeded 20% of their total translational performances). Accordingly, it implicitly exhibits that only the CS and IS translational performances will be part of the analysis. Hence, it could be argued that this coding and marking procedure not only prepares the data for the next descriptive and inferential statistical step but also increases the reliability of the data, allowing conclusive conclusions and generalisations to be drawn from this study.