Concept Mapping Protocols

Two important components must be considered when using concept mapping for evaluation: ―a task that students perform to demonstrate and record their knowledge, and a scoring system which a researcher or teacher uses to evaluate the students‘ knowledge‖

(Stoddart et al., 2000). Protocols were developed to address both of these components along with a protocol for training students unfamiliar with concept mapping in how to construct maps. Each of these is discussed below in the following order: 1) training students to make concept maps; 2) the design and administration of the concept mapping exercise on both questionnaires; and, 3) the Concept Map Quality Scoring Rubric and Protocol (CMQSRP).

In order to increase the validity of concept mapping data it is critical to train students in how to construct concept maps prior to the pre-questionnaire (Stoddart, 2006). This training was planned into the first week of the intervention. On my first day in the classroom I introduced concept mapping as a way of recognizing interconnections and lead the students through an example of a concept map on a topic with which I thought they would be familiar due to the emphasis on sunburn prevention in New Zealand:

ozone depletion. I projected the image of this concept map with the data projector and talked the students through the map from top to bottom, pointing out the difference between concept words and linking words and what makes up a proposition.

After that science-based map, I invited the students to join me in making a map on the whiteboard on what makes up the economy of their town. I provided short list of potential words and asked the students to brainstorm other words. I called on students individually

to suggest linking words and propositions while I constructed the map on the board. The teacher had suggested doing another concept map on fast food, a topic he said was familiar to the students and of interest to them. Although we did not have time to do the second concept map on that day, I prepared a PowerPoint slideshow concept map on fast food, which I presented to the students one link at a time, and asked them why I might have chosen the linking words I did. On the third day I presented the students with the task of creating a concept map on the causes and effects of climate change. As the students worked in small groups I circulated around the room to encourage full participation and answer questions. Following this exercise, I presented my own concept map in the form of a PowerPoint slideshow and walked the students through it step by step. On the fourth day of class I administered the pre-questionnaire.

Concept mapping tasks used for assessment or evaluation can take a number of forms from constrained to open-ended and degrees between the two. Constrained tasks restrict participants to a limited number of words or a fill-in-the-blank format. Open-ended tasks may supply a few prompt words at most, but no other restrictions on map makers.

Intermediate tasks may supply a list of concept words, sometimes called a ‗parking lot,‘

but place no restrictions on how the map is drawn (Stoddart et al., 2000). For this study I chose an intermediate approach that provided students with 16 concept words in a parking lot, but encouraged them to add more words that they thought would be appropriate for the task of making a concept map ―to show what you know about a sustainable system for producing food.‖ The 16 words were whittled down from an original 24 that were piloted with the year 11 science class. Unfamiliar concepts were dropped after the pilot as Stoddart (2006) advises: in pre- and post-test situations, the words provided should have meaning for students even before the teaching unit. The only restriction placed on the mapping process itself was a starting node at the top of the page that said, ―A sustainable food system.‖ The format and instructions for the exercise were identical to the training map exercises described above. As explained in Section 4.6.2, the students were given oral instructions on concept mapping and reminders of how to form a proposition with examples of linking words immediately before the administration of the pre-questionnaire.

The final step of using concept maps for evaluation involves scoring (Miller et al., 2009;

Stoddart, 2006; Stoddart et al., 2000). The intermediate approach involved in this study allowed for some flexibility in collecting both quantitative and qualitative data. Both approaches can be used to evaluate understanding of a topic and changes in understanding after a course of instruction (Edmondson, 2000; Kinchin, Hay, & Adams, 2000; Miller et al., 2009).

As described below, quantitative map scores were calculated by counting: the number of concept words used; the overall number of links; and, the number of cross-links. In addition, they were scored for sustainable propositions as explained below. Data was only analyzed for students who constructed a concept map on both questionnaires.

The first step in calculating scores involved counting the number of concept words used by each student in his or her concept maps out of the 16 words provided. The total number of concept words used by all students who completed the concept mapping section of both questionnaires were added together and divided by 16 times that number of students and expressed as a percentage. The rare cases in which students provided additional concept words for their maps were considered only during qualitative analysis.

Next I counted the number of links used by each student in his or her concept maps. In the case of the post concept map where the starting node - ‗A Sustainable Food System‘ - was inadvertently excluded in the copy given to the students, each student was credited for one link in addition to those they formed. Additionally, I counted the number of crosslinks used by each student in his or her concept maps. Crosslinks establish interrelationships between different map segments.

Finally, concept maps were scored for the number of sustainable propositions they contained. Two concept words connected by a linking word or phrase forms a proposition. A proposition is a unit of meaning constructed in cognitive structure. As described in Section 4.4.1.2, a list of sustainable propositions was developed using expert maps from the Executive Board of Permaculture in New Zealand.

On the student maps, each proposition was analyzed for whether or not it trended toward sustainability. As explained in the theoretical framework, this would indicate evidence of

sustainable thinking among students. Only those propositions that were considered to reflect sustainable thinking were counted in the tally. Propositions not included in the tally fell into a number of categories summarized in Table 4.2: unsustainable, common knowledge not necessarily sustainable or unsustainable, inaccurate/irrelevant.

Table 4.2: Scoring for Sustainable Propositions on Student Concept Maps

Category Score Example

Accuracy Sustainable Food can be organic Unsustainable Transportation needs petrol Common Knowledge Plants benefit from sunlight Inaccurate/Irrelevant Nitrogen kills weeds

Propositions that maintained the status quo for conventional agriculture were considered unsustainable in most cases. Potential sustainability-related issues that students could have identified in this particular concept map included: use of fossil fuels (both on farm and in transportation – ‗food miles‘), soil fertility, insect and weed control, meat-centered diet (eating lower on the food chain), and water conservation.

As introduced in Section 4.4.1.2, a Concept Map Quality Scoring Rubric and Protocol (CMQSRP) was developed (see Appendix D). Developing a CMQSRP is important for scoring ―the quality to content, knowledge, and skills seen in conceptual organization on pre- and post concept maps‖ (Miller et al., 2009, p. 369). The CMQSRP was used to establish quality scores for each comparison. In this study the expert maps used as comparison were constructed by a number of experienced permaculturists in New Zealand and supported by national and international literature on sustainable agriculture.

4.6.2.2 SOLO Taxonomy

As described in Section 4.4.1.1, the SOLO Taxonomy is scored at five levels:

prestructural, unistructural, multistructural, relational, and extended abstract (Biggs &

Collis, 1982). My challenge was to design an exercise for the pre- and

post-questionnaires that reflected these levels while relating to the topics of study during the intervention and to the broader concept of sustainability. The decision was made to focus the exercise on agriculture, including the role of human labor and fossil energy. In this way, the opportunity would be presented for students to recognize increasing levels of relationships within the commonly acknowledged aspects of sustainability: economic, social, and environmental. The decision was made to include the information for the exercises in two graphs because reading graphs is a basic scientific literacy skill. As students progressed to higher SOLO levels they would use the cumulative information in the graphs to answer successive questions.

As noted in Section 4.4.1.1, some authors have identified the criteria for categorization as a weakness in the SOLO Taxonomy (Chan et al., 2002; Chick, 1998). To address this potential weakness, I took a literal, quantitative approach to the categorization of each SOLO level and the increments between them. For example, to achieve SOLO Level 2 a participant need only recognize one simple and obvious connection (Atherton, 2001). As seen in Appendix C, the first question of the SOLO exercise required students to look at a graph showing the percentage of people working in agriculture from 1961 to 2004 and to indicate what changed about the number over time. This task was designed to evaluate whether a student could achieve a unistructural level (SOLO Level 2) of understanding for one piece of information: the direction in which the graph trended over 45 years. If the student did not answer the question correctly, or left it blank, that student was considered not to have understood the task and was assigned a prestructural level (SOLO Level 1) of understanding (Biggs, 1999; Biggs & Collis, 1982; Chan et al., 2002).

The second question referred to the same graph but asked the students to compare two pieces of information: the percentage of people working in agriculture in high income countries versus low income countries. A correct answer was scored as multistructural (SOLO Level 3), as it indicated a student‘s ability to compare two pieces of information (Biggs, 1999; Biggs & Collis, 1982; Chan et al., 2002).

The third question referred to a graph showing a close relationship between the prices of corn, wheat, soybean and crude oil from 2000 to 2009, including a steep spike and decline between 2007 and 2009. The question asked why the prices of corn, wheat and

soybean went up and down so much during 2007 and 2008, and to give reasons for the answer. The question was designed to identify a student‘s relational level (SOLO Level 4) of understanding: indicating that they could recognize the relationship of multiple pieces of information to a greater whole (Atherton, 2011; Biggs, 1999; Biggs & Collis, 1982;

Chan et al., 2002).

The highest level, extended abstract (SOLO Level 5), was evaluated by asking students to use the trends in the two graphs to comment on their impacts on the three dimensions of sustainability: social, economic and environmental. In other words, as agriculture relies more on oil and less on working humans, what social, economic and environmental changes might occur? This question was designed to challenge students to apply the information from the previous incremental steps to issues not addressed heretofore in the exercise (Biggs, 1999; Biggs & Collis, 1982; Chan et al., 2002). While taking this type of quantitative approach to the SOLO Taxonomy may not be possible in all instances, it was deemed appropriate in this case because it could be applied to the example graphs that I used on the questionnaires and because it addressed the potential weakness of selecting appropriate criteria for categorization. While there will always be grey areas in qualitative research, thick description such as that provided in this section is a way to address potential weaknesses. Nowhere, potentially, is thick description more important than with observation data as discussed in the next section.