Data Analytic Plan 1 Research Question One

What, if any, APT moves are associated with the presence of unique epistemic frame elements in group discourse during design team meetings?

To answer this research question I fit a series of logistic regression models to test the relationship (α-level, p<.05) of APT moves, controlling for all other types of

conversational moves and individual player characteristics (NOVICE and MALE), on the probability that evidence of each epistemic outcome would be present in an utterance associated with each move. Models tested these relationships at three levels of analysis

(i.e., “segments”) - the sample, design cycle (1 and 2), and conversation level (1, 2, 3, and 4). Final models were determined through likelihood-ratio tests and evaluated for

goodness-of-fit. A sample model22 for this analysis, for each epistemic frame outcome followed this format, where “EFrame_x” represents each of the 20 possible epistemic frame outcomes (e.g., Skill of Data), and “Discode_i” represents each of the unique conversational moves used in my analysis:

To facilitate meaningful interpretation of results, I then computed and interpreted the average marginal effects (AME)23 for each conversational move retained in the final logistic models to determine the practical (“real”) effects of each move relative to evidence of each epistemic outcome (i.e., the predicted probability of the evidence occurring based on the discrete change in the presence of the conversational move (i.e., from “0” (not present) to “1” (present)). Said another way, the AME statistic indicates the predicted probability of evidence of the outcome for each predictor variable in a regression model.

3.4.1 Research Question Two

Are there differences in the connections among different epistemic frame elements that groups make in their discourse with respect to APT? What, if any, group-level characteristics are associated with these differences?

22 _{I also tested for interaction effects in all models (NOVICExMALE; NOVICExDiscode_i;} MALExDiscode_i), though none were found to be significant.

23 _{Another commonly used statistic to do so is the marginal effects at the means (MEM), or “conditional}

effects,” which is a statistic of the practical effect of an independent variable in a regression model when all other variables are equal to their means. However, some researchers have argued that the use of AME provides a more realistic estimate of these effects given that the sample means used in the calculations of marginal effects can refer to nonexistent observations in the data in large sample sizes, as are found in this study (Bartus, 2005; Bockarjova & Hazans, 2000).

log p(EFrame _ x =1 p(EFrame _ x = 0 ! " # $ %

I answered my second research question by conducting Epistemic Network Analysis (ENA) using a publically available, web-based analytical tool developed by EGG/GAPS (http://epistemicgames.org/ena/) to measure the development of complex STEM thinking by quantifying the co-occurrences of epistemic frame elements in discourse (Choi et al., 2010). Arastoopour et al. (2014) provide a concise overview of ENA:

Because the learning that takes place during a practicum can be characterized by the connections between elements of a professional frame, ENA measures when and how often students make such links during their work. ENA creates a network model (similar mathematically to a social network model) in which the nodes of the network represent the skills, knowledge, identity, values, and epistemology from a domain. The links between these nodes quantify how often a person (or group of people, depending on the model) has made connections between these elements at some point in time. In this way, ENA models the development over time of a student’s epistemic frame—and thus quantifies their ability to think and work like professionals (p. 215-16).

Using the ENA tool and the epistemic discourse data, for each conversation (i.e., room), for each group, the co-occurrences of each pair of codes was calculated and used to create epistemic network models for analysis of differences in the associations of particular conversational moves or about different groups’ discourse. In what follows, I briefly describe how these visual models are created24 as well as how they are interpreted.

Network models are graphical projections of the epistemic discourse data in higher-dimensional space based on the singular value decomposition (SVD) of adjacency matrices, which reveal the structure of connections between discourse elements (Orill et al., 2013). To generate these representations, the ENA tool uses a mathematical

approach similar to PCA. The tool first creates an adjacency matrix of all possible co-

24 _{For a more thorough discussion of the mathematical theories and equations underlying ENA see: Shaffer}

occurrences of codes for each group, in each room. These matrices are then converted (i.e., unwrapped) into adjacency vectors and summed for each group. Then, a singular value decomposition (SVD) is conducted on the matrices by rotating the vectors in space to show the greatest variance. In this space, each group’s vector containing the code co- occurrences is then represented as a point in high-dimensional space (i.e., ENA space), and can be interpreted by examining the loadings (rotation) matrix for each dimension (Arastoopour et al., 2014). Finally, each network model has a unique “center of mass,” derived from its composite score, which is used for significance testing.

Four visual aspects of epistemic network models can be interpreted and compared across models – space/region, size, line weight, and node size (Shaffer, 2014;

Arastoopour et al., 2014; Wesley Collier, Research Assistant with Epistemic Games Group, personal communication, October 2, 2014).

1) ENA “Space”: Network models show the relative location of connections made in discourse (i.e., "where" connections are in “ENA space”) and

elements that are clustered together in space are more “related" than elements further apart in the models. These models are therefore one way of

characterizing the nature of epistemic discourse in conversation (i.e., what the “talk” was about).

2) Network Size: Network models that are denser, or broader, are those that