Data Coding and Analysis

In preparation for analysis, all working memory and proficiency-related data for this study were collected, scored and stored on a password-protected laptop computer. All proficiency interviews were digitally recorded, with the data then coded and analyzed by the researcher. Fluency data were measured as per Yuan and Ellis (2003): 1. The data were first analyzed for the number of syllables uttered per minute by dividing the total number of syllables uttered during the monologue by the number of seconds the

participant spoke and then multiplying by sixty; 2. The number of meaningful syllables uttered per minute (MSM) was calculated by repeating this procedure but subtracting the total number of repeated or reformulated syllables. The final results were compared with NS data collected from the four NS participants, and were tallied as a percentage of the average number of MSM uttered by an average native speaker (see Table 3.2). The number of MSM uttered by NS participants fell within a range between 210 and 270 (the lowest NS participant scored 220 MSM, and the highest 258 MSM). For example, a speaker with level three fluency utters an average of between 30 and 40 percent of the minimum number of MSM uttered by a native speaker, while a level four speaker utters between 40 and 50 percent. A level eight speaker utters between 80 and 90 percent of the minimum number of syllables observed in NS speech. Any participant scoring within ten percent of a native speaker was described as ‘near native’, and assigned a score of 8.5 for coding purposes. The performance of a speaker who scores within five percent of the range of NS participants could therefore be described as ‘native-like ’.

108 108

Table 3.2 Participant Fluency as a Percentage of Native-level Fluency

% of Native Level 10 - 20 % 20 - 30 % 30% - 40 % 40% - 50% 50% - 60% 60% - 70% 70% - 80% 80% - 90% Above 90% Near- native Native Level Level 1 2 3 4 5 6 7 8 8.5 9 Range (MSM) 21- 41 42- 62 63- 83 84- 104 105- 125 126- 146 147- 167 168- 188 189- 209 210- 270

The coding of the accuracy data for Section Two (the elicitation task) was achieved by assigning a total number of points for each elicited construction. Points related to specific morpho-syntactic and semantic aspects of the learner’s performance. Six points were assigned to each relative clause structure and five points were assigned to the subjunctive construction with chtoby. In addition to the number of items and

inflections assigned to each structure, an additional point was related to syntax, and another related to semantic content. Examples are shown in 3-1 – 3-3.

(3-1) Relative pronoun kotoryj

O kakikh amerikantsakh govorit Valerij? About which(LOC/PL) Americans(LOC/PL) speak(3S.PR) Valerij? ‘Which Americans is Valerij talking about?’

1 (infl.) 1 (item) 1 + 1 (1 item + 1 infl.) 1 (syntax) 1 (meaning) ob amerikantsakh, s kotorymy On pogovoril vchera. about Americans (LOC.PL) with who (INS.PL) He speak (3S.PF.PT) yesterday ‘He is talking about the Americans with whom he spoke yesterday.’

(3-2) Relative pronoun construction to, chto

O chyom khochet rasskazat’ Andrej Vitaliyu?

109 109 ‘What does Andrew want to tell Vitalij about?’

1 + 1 (item + infl.) 1 + 1 (item + infl.) 1 (syntax) 1 (meaning)

o tom, chem Zanimaetsja Boris

about that(LOC/SG) what(INST/SG) do(3S.IM.PR) Boris

‘…about (that) what Boris is doing’

Total possible points for each test item above: 6 (3-3) Subjunctive complementizer chtoby

Chto khochet Larisa?

What(NOM.SG.) want(NOM.SG.) Larisa

‘What does Larisa want?’

1 (item) 1 (infl.) 1(Past) 1(syntax) 1(meaning) Ona khochet, chtoby Masha priekhala na dachu She want(3SG.PR) so that (SUBJ) Masha(NOM) come(PAST/F)

‘She wants Masha to come to the dacha.’ Total possible points for the test item above: 5

The items above were scored differently (six possible points for relative clause constructions and five for the subjunctive construction with chtoby), since relative clauses in Russian often have a greater degree of complexity, as seen with the presence of an additional item in each of these examples (the preposition s in the kotoryj clause, and the demonstrative to in the to, chto construction). Two additional points were added to each structure, one for native-like syntax and one for retention of meaning. For example, if a participant responded by omitting the correct inflection on kotoryj, the basic syntactic structure remains correct, while the meaning has changed. Two points were therefore subtracted (one for a missing inflection and one for a change in meaning), and the item is scored as four out of six. In the event of a missing targeted item, such as the

complementizer chtoby or the relative pronoun kotoryj, two points were immediately subtracted from the score, one for the missing item and one for a change in meaning,

110 110

provided the syntactic structure remained, e.g. Ona khochet, chto Masha priekhala na

dachu ‘She wants that Masha came to the dacha’ would be scored as 3 points, with one

point taken off for the missing item and one for the change in meaning. A total of 34 possible points were included in this section of the test and scoring utilized the same 9- level numeric system. As with scoring the fluency monologue, to calculate the level of accuracy, the total number of points scored was divided by the total number of points possible (34) to arrive at a percentage of native-level accuracy; the result was scored as a percentage of native-level accuracy as demonstrated by the NS participants of the study, i.e. 10 – 20 percent as level 1, 20 – 30 percent level two, 30 -40 percent level three, and so on. (the score level ranges in terms of percent of native-level accuracy are identical to those for fluency depicted in table 3.3). Scores ranging within ten percent of native-level accuracy were assigned a score of nine minus or ‘near-native’ accuracy.

The general proficiency or ability score resulting from Section Three of the interview was rated according to the level criteria provided in the oral proficiency guideline (see Appendix C), and as with sections one and two, nine possible levels (plus an additional, ‘near native’ level) were included in the assessment criteria for this section of the test. This part of the interview was not included in the statistical analysis related to the interaction between working memory and learning conditions to produce accuracy and fluency, due to the general nature of the score on this part of the interview. The final overall proficiency or ‘native-likeness’ score was calculated as an average of the

combined fluency, accuracy, and general ability measured in part three of the test. The resulting overall degree of native-likeness was then used as one of three dependent

111 111

variables (fluency, accuracy, overall proficiency) as part of the analysis for the predictive validity of working memory under different learning conditions.

The second purpose for the general interview portion of testing, namely recording the number of potentially fossilized forms uttered by participants, was coded in a manner similar to the analysis of fluency. For this procedure only those participants with over four years L2 Russian experience were included in order to ensure that participants would have a minimum amount of exposure to the types of structures considered for the

analysis. To attempt to eliminate the possibility that the learner was in the process of learning an observed erroneous structure, any items that learners attempted to reformulate or that did not recur at least three times in speech were eliminated from consideration. Due to the phonological nature of Russian unstressed and stressed inflections as well as verbal morphology and use, it was not necessary to control for any potential phonological effects on accuracy in indentifying potentially fossilized forms: phonology does not interfere with Russian morphosyntax as it does with other L2s, such as English. To determine the degree or amount of potential fossilization in participant speech, the total number of forms uttered repeatedly by a speaker was divided by the total number of words uttered in the sample and then multiplied by 100. The final repeated error rate was recorded numerically. For example, a sample of participant speech that includes 380 words and evidences a total of eight different types of repeated errors would yield a total score of 2.1 potentially fossilized forms per 100 words.

Prior to any analysis of the data, qualifying candidates were divided into three groups related to learning conditions reported on the Language Background

112 112

naturalistic learning (FN), and naturalistic learning followed by formal instruction (NF). Based on the goal of comparing subjects who have invested similar amounts of time in their pursuit of proficiency, each group was then further divided into subgroups based on the duration of learning experiences. The learning condition groupings consisted of three subgroups, which were divided in terms of amount of experience as follows: a) below 50 months of exposure (less than four years); b) 50-110 months (between four and nine years); c) 110-170 months (between nine and fourteen years); d) 170-230 months

(between 14 and 19 years) of exposure; and e) over 230 months (19 years). The data was then tabulated and recorded in Microsoft excel to prepare for analysis.

For the Naturalistic/Formal (NF) and Formal/Naturalistic (FN) conditions, an additional procedure was utilized in order to further isolate any potential effects of the amount of formal experience on components of proficiency and potential fossilization rate: participants were divided in terms of the percentage of overall experience

represented by formal training. For this procedure, participants were categorized according to the following percentages of formal instruction: a) between two and eight percent of total experience; b) between eight and fourteen percent; c) between 14 and 22 percent; c) over 22 percent of experience comprised of formal training.

Prior to statistical analysis, a general analysis was performed in order to arrive at an overall picture of four major characteristics: 1) the level of attainment achieved for the different components of proficiency for all learners collectively; 2) the differences in attainment of components of proficiency between the three learning conditions in relation to the amount of experience and the extent, if applicable, of formal learning experiences (as a percent of overall experience; 3) the predictive relationships between WM and the

113 113

major aspects of proficiency for all study participants grouped according to amount of experience; and 4) the predictive validity of WM for the specific aspects of proficiency with study participants divided by the three learning conditions in terms of the amount of learning experience.

Based on these procedures, participants were first grouped together in order to evaluate in general the average attainment of all components of proficiency for specific amounts of experience, as follows: a) between one and four years; b) between four and nine years; c) between nine and fourteen years; d) between 14 and 19 years; and e) over 19 years of experience. Furthermore, for each of the specific learning condition-related analyses, study participants were divided in three ways: 1) into the three different learning condition groups; 2) in terms of the overall amount of L2 learning experience, with groups consisting of individuals with similar amounts of total learning experience, and 3) into subgroups according to general profiles of working memory as well as experience in order to discern predictive relationships between WM and the components of learner proficiency for learners with similar amounts of experience, as well as

differences between the conditions. For example, learners whose average working memory was level 2 and whose experience was between one and four years under naturalistic conditions were compared with learners who also scored level 2 in working memory and whose amount of experience was similar under a different learning context.

Statistical Analysis Procedures

Each aspect of proficiency was scored numerically with nine possible scores for regular study participants, ranging from level one to level nine (native-level proficiency) for each of the three categories of fluency, accuracy, the general ability proficiency score,

114 114

and the overall combined native-likeness measure. Learners could therefore receive combined accuracy/fluency scores of different values (e.g. 3/2). For the general ability score in Section Three, to be assigned a particular level, a learner had to possess a majority of the characteristics of that level according to the test guidelines. Scores for accuracy and fluency were regarded as separate dependent variables for statistical analysis. The general ability score was analyzed separately solely for the purpose of determining the fourth, native-likeness score, which consists of the average between fluency, accuracy and general ability scores. The derived native-likeness score was only used in analyzing the predictive relationships between WM and overall proficiency, since the criteria for the underlying general ability score based on Section Three of the test are more general than the fluency and accuracy scores, though the derived NL score normally closely reflected the combined accuracy/fluency scores achieved.

The score related to the presence of potentially fossilized forms was also analyzed separately as the fourth dependent variable, potential fossilization rate (FR). This

variable was included with other dependent variables in both the regression UNIANOVA to test for the predictive relationship between WM and measures of SLP, and in the moderated regression designed to test for potential interactions between WM and learning conditions. Independent and dependent variables are depicted in Table 3.3.

Table 3.3 Independent and dependent variables

INDEPENDENT VARIABLES

Working Memory (WMC) (Coded as a percentage)

Learning Condition (/ = followed by)

Low Naturalistic Learning

Average Naturalistic/Formal High Formal/Naturalistic DEPENDENT VARIABLES Accuracy Levels 1 – 9 Fluency Levels 1 – 9 Native-likeness average Fossilization Rate

115 115

The objective of the three research questions in this study involved a) an attempt to discover predictive relationships between specific measures of WM and attained levels of key aspects of proficiency as well as potential fossilization, b) the effects on the major aspects of attained proficiency of an interaction of WM with different learning

conditions, and c) the effects of the different learning conditions and the two different types of sequencing of conditions on both the major aspects of proficiency and the occurrence of potentially fossilized forms. In light of past research that indicates strong predictive correlations between WM and proficiency (e.g. Daneman, 1991; Harrington and Sawyer, 1992; O’Brien, Segalowitz, Collenteen and Freed, 2006; Sawyer and Ranta, 2001), the first procedure adopted in this study was to test for the predictive validity of WM in each learning condition. To examine these predictive relationships between WM and the different aspects of proficiency, logistic regression was utilized. Participant scores for working memory were recorded as a percentage and analyzed in two different ways: first, WM scores were classified as either low, average or high and loaded as factors, and secondly, the scores were loaded as covariates with learning experience. These two different methods were utilized in order to both compare the results to look for any inconsistencies and to compare and verify whether the patterns found in the general analysis were reflected in the regression. The dependent variables for participant

proficiency consisted of the three components that were tested individually: an accuracy score related to the elicitation task, a fluency score derived from the fluency monologue, and a score related to the overall proficiency or “general ability” derived from the general oral proficiency interview section of the test (Section Two). A fourth dependent variable related to overall proficiency, called a “nativelikeness” score was also derived by

116 116

averaging the fluency, accuracy and general ability scores together. Finally, WM was also tested for predictive relationships with the dependent variable related to the number of potentially fossilized forms occurring in participant speech, or fossilization rate (FR). A separate regression analysis was performed to test for effects of WM on the frequency of occurrence of such forms.

Another goal of the present study was to describe the interaction between working memory, which is a predictive or explanatory variable, and other environmentally-

conditioned variables, namely learning conditions. Such an analysis suggests the idea that the strength of the effect of WM is potentially affected by learning condition. In other words, the analysis tests whether the strength of the effect of WM on proficiency is modified or moderated in any way by the nature of the learning condition variable, or potentially vice versa. Along the lines of previous aptitude interaction research, an analysis of the effects of interactions of the two independent variables on the major elements of proficiency therefore required a moderated multiple regression procedure (MMR) (Overton, 2001). Furthermore, according to Overton (2001), the need to provide corrections for heterogeneous error variances also demands a follow-up analysis, and Tukey post-hoc test was therefore performed following initial analysis of the data.

The third goal of this study was to test for potential correlations of the different learning conditions and their two different combinations with major aspects of participant oral proficiency, including the occurrence of potentially fossilized forms. Two different analysis methods were used for this portion of the experiment. First of all, the amount of overall participant experience under the two combined learning contexts was analyzed for correlations with the tested measures, and second the amount of formal training as a

117 117

percentage of overall experience was loaded to check for correlations between such percentages and test results for the three components, the nativelikeness score and the observed rate of potentially fossilized forms. Multivariate regression was also used for this procedure. Combined with both the analysis of the predictive value of WM and the analysis of potential interactions between WM and learning conditions, the results present an intriguing picture of the effects such relationships have on the nature of proficiency in L2 Russian.

118 118