Code
coded the remaining 80% of the transcripts. A total of 312 pages of coded transcripts were produced.
As the transcription and coding processes progressed, a research journal was created (Galman, 2007) to track important process ideas and emergent themes.
See below for an extract of the sixth grade transcript. The EDP codes were placed directly after the timestamps and the secondary EDP-related codes were placed at the end of each segment for clarity. The building moves, as well as the discourse, were
transcribed and inserted using curly brackets immediately after the timestamps. Brief notes, if any, were places in parenthesis. Longer notes were placed in the research journal.
The relationship between segments and codes is interesting and a somewhat complex one. Certain segments typically indicate a possible EDP phase transition. Here are some examples.
• {moving} - typically indicates EVALUATE or RESEARCH
• {connecting} and {searching} - usually indicates BUILD
• {no_activity} - indicates WAIT or PLAN
• {gesturing} - usually indicates PLAN
See Figure 26 for an illustration of the relationships between segments and codes.
Figure 26. Segmenting and coding example.
The first BUILD code contains two physical segments that both indicate building as well as a verbal segment that, in this case, also indicates the same EDP phase.
However, the second BUILD code shows a simultaneous and different verbal code of PLAN while the subject is connecting parts. Note that the overlapping code starts with
“2:” to help with the later computer program extraction of the codes. The subject then stops connecting and is only planning. The last part of the example shows a RESEARCH
counts the evaluation of a side build as part of research. This final part also has an example of the verbal track overlaps and extending beyond the physical track. The code extraction and visualization process (described later) handles all these cases accurately.
Before that is discussed, however, the methodology used for secondary codes is briefly described.
Secondary EDP Coding
Note in the transcript above that secondary EDP codes were placed at the end of segments. Most secondary codes have a value of +, =, or - indicating a positive, neutral, or negative effect. Three subjects were fully coded with secondary EDP codes.
Code Checking, Extraction, and Importing Into EXCEL
Two “little programs” were developed (based on the pilot study program) in the Python programming language (Summerfield, 2010) to extract the timestamps and codes from the transcripts. The two programs are a code scanner and a code extractor. See Appendix F - Code Scanner Program and Appendix G - Code Extraction Progrm for the actual Python code. The code scanner checks for valid codes and common errors. The extractor program creates four output files for each transcript:
Main Codes - timestamps and main EDP codes,
Sub-codes - timestamps and EDP codes and EDP sub-codes,
EDP Related Codes - timestamps and EDP-related problem solving and causal reasoning codes.
Error log - file of any errors encountered. These are also shown on the screen.
Here is an example of some of the error detection output of the code extractor program.
timeStamp Error in line [00:13:32] {moving} [RESEARCH] [2:PLAN] Girl 06:
If I can put this together-
Unexpected S Phase expecting store in line [00:13:33] [2:END]
timeStamp Error in line [00:13:34] [WAIT] Researcher: Let me press this for you. [HELP]
timeStamp Error in line [00:13:50] [2:PLAN] Girl 06: And then so I could do it a little lower, so it wouldn't fall.
Unexpected S Phase expecting store in line [00:13:54] {connecting} [2:END]
[BUILD-REBUILD]
When coding errors were detected in either program, they were corrected and rechecked. The improved error detection in this study improved the validity of the data and the reliability of the results.
The main code files were then imported into Microsoft Excel. See below for a sample extract of the main codes Excel file.
Time Elapsed Code Code
The phase code number is needed for later analysis using Excel, specifically to produce the EDP timelines. Elapsed times for each phase for the main and sub EDP codes were calculated in Excel. Once the EDP data was imported into Microsoft Excel,
Visualization Production
Once the data was imported into Microsoft Excel, a number of different types of visualizations were produced: finished model design data graphs, EDP timelines, and EDP count, frequency, and duration graphs.
Finished model design data graphs. Graphs were produced using Microsoft Excel that show the data about the finished model such as number of parts, parts used (motor or no motor), originality rating, functionality rating, EDP rating, and other attributes of the finished amusement park ride models by gender and by grade level.
Because no significant differences were found by gender or grade level but significant differences did seem to be occurring, additional graphs were produced by LEGO
experience and EDP rating, which were added as new, possibly significant, independent variables. LEGO experience was rated using a questionnaire that was filled out by second grade parents and sixth grade students. See Appendix H - LEGO Experience Questionnaire for the questionnaire and rating scheme. The finished model design data showed attributes of finished design. However, a big part of this research was to understand the difference engineering design process of elementary students.
EDP timeline graphs. Using the EDP data in Microsoft Excel, individual graphs
were produced that show the engineering design process phase by the elapsed time.
Sample data and the corresponding sample EDP timeline are shown in Figure 23 and Figure 25 respectively.
EDP count, frequency, duration graphs, and other data. Additionally,
individual graphs for each subject were produced that show the count of each EDP phase,
the time spent in each EDP phase, and the average duration (in seconds) of each EDP phase. See the Results Chapter for examples.
While it was initially planned to produce additional visualizations of aggregated EDP and secondary EDP data by the two initial, independent variables of grade level and gender, these were not done because significant differences were not being seen. A summary rubric was created that rated the factors that did seem to the driving the differences seen in EDP timelines and final model quality. These factors are the build complexity, the LEGO structural knowledge possessed by the student, three specific executive function skills, and three specific domain specific EDP skills. [The process of creating and using the summary rubrics is explained in more detail in the next section.]
Figure 27 shows the overall relationship between the different data produced and
analyzed in this study. Moving up in diagram indicates the timing of the data production process and also the increased level of abstraction from the actual sessions.
Figure 27. Overall study data taxonomy.
Now that that data and derived data process has been fully defined, a short description of the analysis process used is given.
Data Analysis Process
First, the finished model data graphs were examined for significant differences by gender, grade level, LEGO experience, and EDP rating. The next and more complex step looked at the frequency and distribution of events in the EDP timelines of the twelve students. This methodology is called inductive contrastive analysis (Goldman, Erickson,
Lemke, & Derry, 2007). Basically, patterns were searched for in the EDP timelines of students. A similar approach has been using by others in studies of the design process of undergraduate students (Atman et al., 2007, 2005; Atman & Bursic, 1998) and in
novice/expert engineering studies (Crismond, 2001). As an example, Atman et al. (2007) found what they considered an ideal EDP timeline shape by looking at EDP timelines of expert practitioners and the undergraduate engineering students in the study.
The next step was to again look for patterns in the EDP count, frequency, and duration graph of the students by a number of various factors. Since the original
independent variables of gender and grade level were not showing significant differences, both the EDP timeline and EDP count, frequency, and duration graphs were labeled with a variety of potential independent variables (also called factors and described in previous section above) and sorted by each variable in turn to see if patterns emerged. One example of the twelve visualizations created for this purpose is shown below.
Figure 28. EDP timeline summary.
Figure 29. EDP count, frequency, and duration summary.
When a pattern did emerge, a summary rubric (see Table 1 - Summary Rubric) was used that measured the seven most relevant factors for each design and design process and appropriate visualizations were created that show some of the relationships between the seven factors. The Results and Discussion chapter will describe the relationships that were found.
CHAPTER 5 RESULTS
Warm Up Task Results
Recall that the warm up task served two purposes: to help students understand and execute the talk-aloud protocol and to serve as a check on the students’ skills, process, and knowledge when compared to the main task. For the latter, there was a close correlation between the tasks.
Figure 30. Warm and main task ratings.
Girl 3 was the exception. However, as we shall see, she had good EDP skills but lacked structural knowledge and general executive functions process skills that caused problems building the complex amusement park ride build she chose but did not cause
0.0