Reliability Analysis

A first step in analysing the data was to reduce the scores on the forty-three questionnaire items to composite scores for eleven CMC variables. The questionnaire was subjected to reliability analyses. Reverse-phrased items were reverse-scored before running the analyses. For three of the variables, there was high consistency among item scores: Cronbach’s α = .74, α = .85, and α = .76, as shown in Table 2. The corrected item-total correlations were above .3 for all items of these variables, and the alphas did not improve if items were deleted. Mean scores were computed for these variables. Yet for seven variables, the alphas were lower than the commonly accepted .7, some corrected item-total correlations were below .3, and some alphas improved when deleting items. This can be attributed to the number of scales and the inconsistency of scores among different social media within a single variable. Analysing those questions separately rather than together within umbrella variables was deemed unfeasible, given the unduly high number of predictor variables this would yield for the regression analyses. To keep the variability within the data visible,51F52 we computed sum scores for variables with α < .7.

Table 2. Reliability analysis results for the questionnaires.

CMC variable No. of

questions α Score

Variety of CMC use 7 .35 sum

Frequency of CMC use: average amount of time

spent per day 6 .53 sum

Frequency of CMC use: number of messages sent

per day 4 .53 sum

Exposure to CMC: number of messages received

per day 3 .21 sum

First experience with CMC: age of first acquiring

mobile phone or using CMC software 5 .74 mean

Intensity of CMC use: size of social network via

CMC 4 .65 sum

Use of textisms in CMC 5 .85 mean

Understanding of textisms in CMC 1 - i

Mobile phone ownership 3 .54 sum

Mobile phone dependency 3 .53 sum

Use of predictive or corrective software in CMC 2 .76 mean

Note: alpha scores > .7 appear in bold.

i _{Since this variable was measured using only one question, no α was computed.} 3.3.2 Analysis of Writing Quality

As explained above, text quality is a multifaceted notion that should ideally be analysed in context. Still, we need to objectively determine the quality of our participants’ writing products in isolation here. Considering the valid objections that could be raised to a one-dimensional analysis, we adopted a multidimensional approach to text quality for this study: following Spooren (2009), the texts were analysed with quantitative linguistic measures at different levels. Our analysis was facilitated by T-Scan, software that can analyse Dutch texts (Pander Maat et al., 2014; Pander Maat, Kraf, & Dekker, 2016), into which the essays were entered after they had been typed out and formatted as required by the tool. T-Scan was selected because it is state-of-the-art, frequently updated, and can be accessed freely. We came across no other software that was able to provide information on Dutch texts on such diverse levels and to analyse texts for so many features. The T-Scan output contained 411 variables for each text. Out of this large set, a selection was made of 27 variables that were deemed relevant for school writing. Some T-Scan variables measure more or less the same concept; in that case, we picked one as a representative for such a group of variables to avoid multicollinearity: for instance, for TTR_wrd (type-token ratio for words) and TTR_lem (type-token ratio for lemmas), we only selected the former. These 27 were divided into six categories – measures of length, structure, diversity & density, verbs, nouns, and other parts of speech. The following list of variables features their original T-Scan names (underlined) plus their definition:

Length measures:

1) Zin_per_doc: number of sentences per essay 2) Word_per_doc: number of words per essay

3) Wrd_per_zin: number of words per sentence (average) 4) Let_per_wrd: number of letters per word (average) Structural measures:

5) Bijzin_per_zin: number of subordinate clauses (finite + infinitival) per sentence 6) D_level: D-level [developmental level]

7) AL_gem: average of all dependency lengths per sentence 8) AL_max: maximal dependency length per sentence Diversity & density measures:

9) TTR_wrd: type-token ratio (for words)

10) MTLD_wrd: measure of textual lexical diversity (for words) 11) Inhwrd_d: density of content words [lexical density] Verbal measures:

12) Pv_Frog_d: density of finite verbs 13) Ww_mod_d: density of modal verbs

14) Huww_tijd_d: density of auxiliary verbs of time 15) Koppelww_d: density of copula verbs

16) Imp_ellips_d: density of imperatives and elliptical constructions 17) Lijdv_d: density of passive forms

Nominal measures: 18) Nw_d: density of nouns

19) Pers_vnw_d: density of personal and possessive pronouns 20) Nom_d: density of nominalisations

21) Spec_d: density of proper nouns, names and special words Other parts of speech measures:

22) Bijw_bep_d: density of adverbials 23) Vg_d: density of conjunctions 24) Lidw_d: density of articles 25) Tuss_d: density of interjections 26) Interp_d: density of punctuation 27) Afk_d: density of abbreviations

All measures of ‘density’ computed the average number per 1,000 words. The length measures took into account the length of the text (number of sentences, 1, and words, 2), of sentences (no of words, 3), and of words (no of letters, 4). These features have been identified to be effective in determining the level of a text (Hacquebord & Lenting-Haan, 2012). The structural measures gauged the extent to which complex constructions were used in the text, such as subordination (5), which is a common indicator of complexity (Shaw & Liu, 1998). D-level (6), which stands for ‘development level’, is a measure of sentence structures based on a classification and rank order of sentence types in eight increasingly complex developmental levels (Rosenberg & Abbeduto, 1987; Covington, 2006). Two more structural measures include dependency length (AL, ‘afhankelijkheidslengte’, 7-8), i.e.

the distance between a head (of a sentence or phrase) and its dependent, such as a finite verb and the corresponding subject. The greater this distance, the more complex it becomes to process a sentence (Gibson, 2000). The diversity & density measures assessed the variation in word choice and the proportion of content words (vs. function words) in a text. The type-token ratio (TTR, 9) is a classic measure, calculated by dividing the number of types (different words) by the number of tokens (total number of words). The measure of textual lexical diversity (MTLD, 10) is the average length of sequential word strings in a text that maintain a TTR above a specified threshold, so it is insensitive to text length (McCarthy & Jarvis, 2010). Lexical density (11) was operationalized as the number of content words, i.e. nouns, verbs, adjectives, and adverbs, per 1,000 words (e.g. Johansson 2008). The verbal measures counted the density of different kinds of verbs – finite verbs (12), modal verbs (13), auxiliary verbs of time (14), copula verbs (15), imperatives and elliptical constructions (16), and passive forms (17). Passive verb constructions are commonly considered to be more complex than active constructions (Chomsky, 1965; Gazdar et al., 1985). The nominal measures, similarly, counted the density of various kinds of nouns – all nouns (18), personal and possessive pronouns (19), nominalisations (20), and proper nouns, names and special words (21). The use of nominalisations is generally seen as making a text more formal or impersonal (Shaw & Liu 1998), whereas personal pronouns and proper nouns make a text less distant to the reader. The final set were measures of other parts of speech, besides verbal and nominal ones, namely adverbials (22), conjunctions (23), articles, (24), interjections (25), punctuation (26), and abbreviations (27).