Data Sources 1 Site/Setting

This study analyzed data collected from the epistemic game for engineering, Nephrotex8, designed and implemented by the Epistemic Games Group (EGG) and the Games and Professional Simulations Research Consortium (GAPS) at the University of Wisconsin-Madison (http://edgaps.org/gaps/). Nephrotex is a virtual internship that simulates professional practice in engineering in order to support undergraduate students’ development of an engineering epistemic frame that could lead to increased motivation for students to persist in their commitment to study and practice engineering

(Arastoopour et al., 2014). In Nephrotex, students take on the role of interns at a fictional biomedical design firm working on a team with 3-4 other interns. In this role, students are tasked with designing, testing, and building an innovative device for production that satisfies a number of competing interests of the company’s internal consultants

(Arastoopour et al., 2012; Chesler, D’Angelo and Shaffer, 2012; Arastoopour and Shaffer, 2013).

Students engage in a number of different types of activities during their virtual internship (see Appendices A and B), including:

1) conducting individual research using simulation-embedded resources;

2) working with their team in a virtual design space to conduct two design-cycle-test cycles in which they develop and test hypotheses, generate device proposals, and analyze results;

3) receiving and interpreting feedback regarding their device performance; and

4) participating in a final, public presentation of device designs (Chesler et al., 2013; Arastoopour et al., 2014; Nepthrotex, 2014).

Throughout the experience, students interact with an in-game supervisor (via email) and an in-game mentor/advisor (via chat and email) who provide support and guidance.

3.2.2 Participants

The participant sample is drawn from two implementations of the Nephrotex virtual internship, summarized in Table 3.2, below.

Table 3.2

Summary of student demographics in the Nephrotex sample (n=273)

Sample A Sample B Combined Sample

Male 151 (77%) 59 (76%) 210 (77%) Female 44 (23%) 19 (24%) 63 (23%) Total 195 78 273 Asian or Pacific Islander 12 (6%) 5 (6%) 17 (6%) Black 4 (2%) 2 (3%) 6 (2%) Hispanic 1 (1%) 0 (0%) 1 (0%) Prefer not respond 2 (1%) 7 (9%) 9 (3%) Other 3 (2%) 1 (1%) 4 (1%) White 164 (84%) 61 (78%) 225 (82%) Mixed/Multiple 9 (5%) 2 (3%) 11 (4%) Total 195 78 273

The first (Sample A) is from the Fall 2012 term with students of undeclared majors (n=195) enrolled in an introduction to engineering course at a large, public university in the mid-west. The second (Sample B) is from the Spring 2013 term with students in declared engineering majors (n=78) in an advanced course at a large, public university in a mid-Atlantic state. All students in each sample were randomly assigned to groups of 4- 5 students for the first design activity cycle (Design Cycle 1) (n=55) and randomly re-

assigned to new groups for the second design activity cycle (Design Cycle 1)

(n=55). Thus, the total sample includes 273 students organized into 110 unique groups. The sample includes 210 male (77%) and 225 white (82%) students. While these demographics are skewed toward white males, they reflect enrollment trends in

undergraduate engineering courses in the United States in 2012 with a few notable differences. For example, this sample over-represents the national average of white students (66%) while under-representing enrollment for Asian-American (12%) and Hispanic (9%) students (Yoder, 2012).

3.2.3 Data sources

Three data sources, obtained from EGG/GAPS, were used as the basis for this study and are described below.

Discourse data

Discourse data was obtained through digital capture of chat-logs during design- team meetings (i.e., text-based utterance data). The data used in this study was drawn from four 30-40 minute long meetings (n=210) where the focal activity is on

collaboration, decision making and reflection related to the design problem. The first two meetings occur during the first design activity cycle (Rooms 7 and 9, n=110). In the first meeting (i.e., “Conversation 1”), students are expected to share findings from their independent reviews of the literature and then discuss and rank the five device

“attributes” they feel would yield the best device design. In the second meeting of this design cycle (i.e., “Conversation 2”) used in my analysis, students shared with their teammates the five device prototypes they each designed and then worked together to decide on the “top 5” devices the team came up with to submit for testing. The second

two meetings used in my analysis occurred during the second design activity cycle (Rooms 11 and 14, n=110). In the first meeting in Design Cycle 2 (i.e., “Conversation 3”), students meet for the first time in their new groups to share performance results from their prior teams’ devices and then proposed five new devices to test. In the last meeting (i.e., “Conversation 4”), students were expected to come to consensus around the team’s best device to be submitted for evaluation. As reference, a summary of all of the focal activities in each “room” in the virtual internship is presented in Appendix A.

Epistemic discourse coding

The coding scheme for engineering epistemic discourse, developed by EGG/GAPS, is comprised of twenty codes derived from ABET9 criteria (2011) and guided by epistemic frame theory for professional practice (see Appendix C). For instance, a student utterance coded as Epistemology of Data indicates that there was evidence that she justified a design decision by using data (i.e., graphs, results tables, numerical values, research, etc.) and an utterance coded as Knowledge of Client indicates that there was evidence that she made reference to the health, comfort and/or safety of the patient. The scheme was applied to the data using a validated auto-coding process that identifies key words and character strings in each utterance to create a binary code for evidence of all twenty codes (1=present, 0=not present)10. Previous validation studies between human coders and the auto-coding system obtained Cohen’s kappa coefficients between 0.80 and 0.98 for all categories (Arastoopour et al., 2014).

9 _{Accreditation Board for Engineering and Technology}

10 _{It is important to note that an utterance can be coded for multiple epistemic codes (i.e., a single utterance}

Pre- and post-survey responses

Students responded to 34 Likert-scale survey questions (see Appendix D) administered before and after the Nephrotex simulation. Questions in the survey were drawn from the Pittsburgh Freshman Engineering Attitudes Survey (PFEAS), a scale shown to be internally reliable and structurally valid as a measurement of college

freshman attitudes toward engineering (Besterfield-Sacre and Atman, 1994; Hilpert et al., 2009; as cited in Arastoopour et al., 2014). Questions in the survey asked students about their associations with engineering careers, their perspectives and beliefs related to the work, characteristics and proclivities of engineers, and their commitment to pursuing a career in engineering. Based on my review of the literature, I identified six survey items (of 31 answered by students in both the Novice and Advanced samples) for use in my analysis, which will be detailed in the following section.