Data Analysis

Analyses were on-going and started during the interview with the researcher following up ideas as they arose and during the observation with the inclusion of comments. Field notes and interview recordings were typed up and examined to discover points of similarity and discrepancy with developing models (Brown & McIntyre, 1993; Stake, 1995). This included noting expected and unexpected

(Cole, 1994) events and ‘active reflexivity’ (Mason 2002), that is reflection on the researchers own performance or influence on responses. The researcher electronically tagged transcripts with comments using Microsoft Word [Appendix C1] to enable ideas and reflections to be shared with her supervisors. This provided discussion points for shared analysis and enabled the researcher’s view to become part of the data. As a quality assurance measure periodically the researcher and her supervisor would tag a transcript separately and then compare analyses. There was a high degree of consistency in the resulting observations. As the research progressed interpretations were explored and modified against further data and research foci revised or developed as necessary.

From this initial tagging process and from the literature codes began to emerge as the data grew. Data sorting software, Atlas.ti 4.2, was used to help organize and code data. By the latter period of data gathering ‘umbrella’ codes had been identified which linked to four overriding themes: teachers, teaching, learners and ICT skills [Appendix C3]. These were used for initial coding of first-level categories (Miles and Hubberman, 1994). This initial coding provided groups of data to further explore from differing perspectives, that is changing the spotlight to seek different patterns. By this process some codes subdivided, for example ‘Aims’ was found to have two sub-codes of motivational aims and teaching aims that is the wider aims that prompted learners to attend the course and the more immediate aim within the classroom. Within sub-codes, groups or categories could also be defined, for example several different motivational aims were expressed, videlicet skills needed for work, leisure, to help one’s children, to keep up to date, to gain qualification. Thus as the analysis progressed these initial codes were modified and developed and a code plan developed [Appendix C2, C3] and the data continually reviewed and recoded. This constant revisiting of data enabled the researcher to compare what different people did or said, what themes were discussed and how concepts were understood (Rubin & Rubin, 1995).

Grids, diagrams and charts to aid cross-sectional or contextual organization of data (Riley, 1990) were also used. This made data quicker and easier to read and helped the researcher organize her thinking (Mason, 2002). Data was mapped according to concepts and characteristics developed from the literature. So for example the ‘motivational aims’ were mapped on the grid used in Figure 1.3, while the observational data was mapped against characteristics of epistemological orientation. This kind of procedure was not intended to define or pigeonhole teachers or methods as it was anticipated that teaching activity would extend across a range of theoretical characteristics and would not neatly fall into defined patterns or category. The aim was rather to reduce the data (Miles & Huberman, 1994), enabling the researcher to make comparisons and explore areas of emphasis or neglect as a prelude to seeking explanations and further data analysis.

Exploratory Data Analysis, techniques, that is processes and methods for exploring patterns and trends in data using graphs, tables and statistics (Myatt, 2007) were also employed. This statistical approach was not intended to produce definitive evidences as in positivist paradigms. The ‘fuzzy’ nature of data sets within qualitative research means relevant objects can have varying degrees of membership to a set (Ragin, 2000) dependent on interpretation (Diamantopoulos and Schlegelmilch, 2000), so cannot be definitively defined. Results can only be interpretations and, as Hartwig and Dearing (1979) suggested, need to be approached with skepticism and openness. But they can be a useful additional tool to ‘data mine’, that is identify ‘hidden’ patterns and relationships (Myatt, 2007), enabling the researcher to find out more about the data (Hartwig and Dearing, 1979). In keeping with Kincheloe’s (Kincheloe and Berry, 2004) call for researchers to use every possible tool, aids such as frequency distribution, scatter graphs and visual representations were developed. For example the responses to the skills ranking activity was mapped according to teacher variables such as ICT background, length of ICT teaching, nature of employment, course focus, and teaching style [Appendix C4(i)], and then according to level of qualification [Appendix C4(ii)]. The aim was to seek

hidden patterns linking perceptions of ICT and teachers’ backgrounds with practices. These graphical representations were then used in conjunction with interview responses to develop a more holistic and complete research picture.

The documents connected to examinations, in particular, become evident as an area requiring a means of graphically representation to aid comparison. As the research developed qualification syllabuses and test objectives emerged as a strong influencing factor on teachers’ classroom practices. The nature of the test objectives was perceived as having the potential to impact on teaching style and outcomes. The researcher sought a tool to analyse the various test objectives to a) define what types of knowledge dimension was evident; b) what levels of cognitive process were emphasized; c) if and how skill level might influence knowledge dimension and/or cognitive process and; d) whether the differing qualifications varied in their approaches.

As part of their learning taxonomy Anderson and Krathwohl (2001) offered a two-dimensional grid, the Taxonomy Table, by which teachers might classify and organize objectives [Figure 4.3] to help them more clearly define learning outcomes. Objectives could be allocated a point on grid by examination of the verb and noun. The verb, the action, related to cognitive process while the noun related to the knowledge dimension. For example in an objective such as ‘The learner will be able to apply formatting options in a table’ the verb would be ‘apply’ and the noun ‘formatting options in a table’ would indicate knowledge of the procedures to apply formats. Thus the objective placement would be cell 3C on the grid. Airasian and Miranda (2002) suggested that the Taxonomy Table might also be used to analyse state-wide assessment objectives, so this seemed a starting point for a possible analysis tool.

Cognitive Process Dimension Knowledge Dimension 1 Remember 2 Under- stand 3 Apply 4 Analyse 5 Evaluate 6 Create A Factual B Conceptual C Procedural X D Mets-cognitive

Figure 4.3: Example of objective mapping using Anderson and Krathwohls’ Taxonomy Table (2001:28)

The skills and knowledge model for user ICT was envisaged as three- dimensional [Figure 1.8]. The knowledge dimension having two elements: the practical and the thinking, covering six cognitive process categories. Some of the qualification criteria, in particular City and Guilds e-Quals and OCR CLAIT similarly presented assessment objectives, or outcomes, as practical activity underpinned by cognitive knowledge (e-Quals, 2001a, OCR, 2005a), which seemed to support the idea of two knowledge dimensions in ICT user activity. The above objective, from this perspective, needs to be examined by the verb and the noun but the latter has two elements the practical and the thinking, which may be implicitly or explicitly expressed. The practical element would refer to the procedural knowledge needed to highlight, select and apply formatting while the thinking element referred to the underlining knowledge of the purposes of formatting. A 3-dimensional, rather than Anderson and Krathwohl’s 2-dimensional, grid was therefore required [Figure 4.4].

Figure 4.4: Three-dimensional grid to track user skills objectives

By assigning numerical values, for example 1-6 for the cognitive process and 1- 3 for the two knowledge elements [Table 4.1] objectives could be quickly allocated a position on the grid according to the level of cogitative process and knowledge dimension. Thus the above objective would possibly score ‘3’ for ‘apply’, a ‘2’ for the procedural steps to complete the task, and ‘2’ for understanding of formatting concepts. This would enable each objective to be mapped and positioned on the grid. Then using 3-dimensional mapping software a graphical representation could be created to enable comparison. Several modules from differing awarding bodies and differing levels of award were examined and mapped. Then a verification process with a second assessor was conducted. It quickly became clear from this process that there were concerns involving a) ambiguity of verbs; b) vagueness of outcome; c) misalignment of objective and test activity; d) lack of clear differential and; e) interpretation of content.

Ambiguity of Verb: Anderson and Krathwohl (2001) had encountered problems with allocating meaning to verbs and warned readers to consider the verb-noun combination. This technique quickly revealed the extent to which the verbs within objectives had varied or confused meanings. For example the word ‘create’ was found to have at least four different meanings. The first, as in ‘Create a document’, meant ‘open’ a new blank document and was concerned with remembering basic practical knowledge. The second involved ‘creating’ a document from specific given instructions that is ‘reproducing’ a document. The interpretation here depended to some extent on the level of practical skill. This ranged form typing in text, which simply involved remembering basic practical skills, to recreating a document by recalling a number of procedural routines. The third, concerned objectives such as ‘create a chart from a table’, meant applying a change to given data which usually required knowledge of procedures and routines. Finally, ‘create’ could be used to describe an original document generated to complete a problem solving task. Thus ‘create’ could score 1, 3 or 6 on the cognitive process scale depending on the context of use.

‘Create’ was by no means an isolated case. Many other actions such as ‘add’,

Cognitive process Practical element Thinking element

Remember 1 Practical Knowledge 1 Factual knowledge 1 Understand 2 Procedural Knowledge 2 Conceptual knowledge 2 Apply 3 Tactical Knowledge 3 Meta- cognitive knowledge 3 Analyse 4 Evaluate 5 Create 6

all found to have multiple meanings. Classification problems were particularly noticeable when trying to define whether a verb implied remembering or applying practical skills. This may be in part because Anderson and Krathwohl’s definition of ‘apply’ itself had a duality, defined as ‘executing’ or ‘implementing’ [Appendix A3]. Applying a procedure to a familiar task would imply a lower level of cognitive process than applying a procedure in an unfamiliar situation. The former requires recall of procedure while the latter may require analysis or evaluation of task to decide what procedure was required. It might be expected that judgements could be made about the nature in which the verb was used but vagueness of objectives compounded this duality. For example ‘use formulae to calculate percentage’ might vary from the formula being given so the learner only has to type it into a cell to the learner developing and applying the formula for himself. Interpreting the verbs’ level of use proved quite problematic and required reference to test texts and assessment material.

A further complication occurred due to the nature of ICT vocabulary. Words like ‘group’, ‘sort’ or ‘preview’ were found to be used in ways that required a lower cognitive level than might normally apply. Typically to group or sort items would imply an element of understanding or analysis but when applied to an electronic activity, for example ‘Sort lists’ or ‘Group drawing objects’ then it involves remembering procedural routines which could require little conceptual knowledge to complete. Similarly ‘preview’ might typically be used in the context of critique or evaluating but in an ICT context is often used simply to mean ‘show a page as it will appear in print’ and only requires remembering basic practical knowledge.

Vagueness of outcome: Often objectives were vague and non specific, for example ‘create graphs’. There is no indication here of what procedures or level of skill was expected. To interpret such objectives required multiple sources, for example test papers and marking criteria. But even then the learning outcome was not always clear due to the prescriptive and procedural nature of the tests.

Misalignment of objective and test activity: Even where the underpinning knowledge was expressed at times there appeared to be a misalignment between the stated objective and what was tested. For example using ‘create graphs’, objective 3a Unit 2 ‘Manipulating Spreadsheets and Graphs’ in CLAIT Plus (OCR, 2005a), the underpinning knowledge for exploded pie charts was expressed as ‘understanding the importance of emphasizing particular data …….. distinguish between pie charts, line graphs and bar/column charts’. The test task example read ‘Using the Total number of guests from each Country create an exploded pie chart …….. Pull out the smallest sector of the pie chart so that it is further away from the rest of the chart’ (OCR, 2005b). While the marking criteria referred to penalisation for ‘failure to select the specific graph/chart type’ (OCR, 2005a). The intention of the objective did not appear to match the examination task or assessment criteria. Ultimately the test was assessed on whether the learner could select the exploded pie chart option from a list option, and was related to procedural activity only. The learner could simply have followed instructions, without any understanding of the rational for using a pie chart as opposed to other graphical representations, or of the importance of emphasizing data, and they would still have passed the test. The underpinning knowledge was assumed and was never tested directly. This misalignment meant that each objective could not be taken at face value but needed to be cross referenced against the appropriate test questions and marking criteria to ensure comprehensive interpretation of meaning.

Lack of clear differential: In some cases, for example OCR Text Processing Levels 1-3, there was very little difference in the wording of the objectives for different levels of qualification. Here the qualification aims and list of knowledge, understanding and skills for each level was almost identical (OCR, 2006). The verbs and nouns used remained essentially the same and differentiation was only possible by exploration of test task or assessment criteria.

Interpretation of content: Without care and cross referencing it was difficult to ensure that objectives were interpreted honestly. On the one hand, there was a

danger that taken at their face value some objectives may be allocated greater depth of cognitive meaning while others may be undervalued. Wrong allocations would undermine the reliability, id est consistency, of the findings.

The process revealed some interesting and valuable data; the problems outlined may themselves be of significance in influencing teaching practice and are discussed in more detail in later discussion. But the level of ambiguity and lack of clarity highlighted the possibility of lack of consistency in interpretation thus cast doubts on the reliability (Guba, 1978; Schensul et al, 1999) of this method of analysis. Distortion was possible due to inconsistency, misinterpretation and variability of scoring both for an individual analyst and between analysts. The concentration on the verb/noun relationship perhaps put too much emphasis on the vocabulary used rather than the context. This quantitative approach had limitations and another analysis method was required that included a more holistic approach.

An alternative way to analyse and compare test material was therefore required. The researcher returned to Clarke and Englebright’s (2003) tri-fold knowledge for effective computer use; Functional knowledge, Structural understanding and Generic skills. Functional knowledge was seen as those skills needed to carry out the task, both the low level practical skills and the higher level procedurals skills. Structural understanding allowed the learner to predict how to do things, involving both factual knowledge and the higher level conceptual understanding of systems and structures. Generic skills went beyond the immediate computer skills and knowledge and emphasized the tactical knowledge and meta- cognitive knowledge required for applying skills and understandings. By adopting this tri-knowledge dimension the need to define the level of skill was removed making judgement easier and more reliable. Competency goals could be directly allocated to one of the three knowledge dimensions.

The test tasks also had an application dimension. For Anderson and Krathwohl (2001) working within academic contexts the level of application was not

differentiated, but here the application level and context becomes far more crucial. All activity in user ICT is concerned with application but the nature of that application could vary according to the cognitive process dimension and level of learner autonomy in completing the task. The ambiguity of the verb ‘apply’ needed to be addressed. During the qualifications analysis, three levels of application had become evident. Level one applied to executing; carrying out procedures to familiar problems. Activity depended on remembering routines and procedures to complete a directed task from which there could only be one correct answer. Level two was concerned with implementing; applying procedures to unfamiliar problems. Here activity required understanding and perhaps analysis of a problem to complete a task successfully. The learners may be given task direction but needed to complete an element for themselves, for example construct the correct formula to be entered in a spreadsheet. There was only one correct result but the means of achieving the result could vary. Level 3 was concerned with problems in context. Activity was concerned with problem solving independently and learners’ solutions were individual and unique, see table 4.2.

Application Level Application type Cognitive process Example Level 1 Application in familiar problems

Remember Italicise the whole of the second paragraph Level 2 Application in unfamiliar problems Understand Analyse Create a formula to calculate the profit for the pears and oranges Level 3 Application in context Evaluate Create Produce a spreadsheet specification from the user description

Thus tasks could be plotted on a grid [Figure 4.5] according to the tri-knowledge

Level of

Learner Control

Application Dimension Level 1 Level 2 Level 3

Skill Cognitive Process Dimension Remember Understand Analyse Evaluate Create Functional Structural Knowledge Dimension Generic

Figure 4.5: Grid used to record test task objectives

dimension and tri-level application using Anderson and Krathwahl’s noun and verb approach but with a greater emphasis on task context. The analysis of objectives alone had proved to be unreliable but this new approach used the tasks, that is the test questions, as the analysis focus with reference back to the stem objective and forward to the assessment criteria in a triple-examination process. It was anticipated that there would be a relationship between the level of application and the type of knowledge. Functional knowledge relating to level one application and remembering, structural understanding to level 2 activity, and generic skills would be required for level 3 activity. It might also be

concluded that as skill level increased the level of autonomy would rise, that is learners would be able to problem solve independently as their functional skills improved. This revised approach was again trialled by the researcher and then with colleagues and was found to be simpler to use and more reliable. There was a much higher degree of agreement on the placement of task objectives which enabled comparisons to be made on differing tests types and levels.

As with data gathering certain ethical issues emerged concerning analysis. The researcher needed to be constantly aware of the place and influence of values and remain critically subjective, open and sincere so as not to prejudge situations to fit personal agendas. During this research, the researcher has