Evaluation

We wish to determine the degree to which our instruction enhances students’ strategic/tac-tical thinking about communication, and to what extent students find this communication-focused course content authentic and useful. To assess student perceptions and feelings, we conducted an open in-class discussion and a written survey. To assess the growth in think-ing about communication, we employed the grounded theory technique of emergent theme identification. While these initial evaluative efforts indicate promise in our approach, our sample size is small, and more work is needed to evaluate its effectiveness thoroughly.

168

6.2.5.1 Class Discussion

Towards the end of the final sprint, the students were asked about what they had learned about communication and teamwork from their previous courses and then they were asked to share some positive and negative aspects of the communication activities.

There was only one communication based freshman course that all the students in the class had taken prior to the Team Software Project course. Some of what they remembered from the course was risk management and rhetoric. A few other students had taken some other courses where they had worked in teams through some part of it. Some students shared that they had learned some things about writing documentation as part of some other courses.

When it came to sharing what worked for them about the communication activities in class, most students said it helped them be “more aware” of their practices and made them “focus on what was working” thereby leading to improvements. Many students said that they liked that the communication activities seemed to have an “industry focus” where they learned about communication in their discipline. When asked about what they would have liked to see changed about the course, some students said that they would have liked to see “more complex forms of communication”, especially ones they did not have any experience with like attending a conference call and would have liked to do some role playing where they are “put on the spot” like they might be in industry and they could “practice spontaneously giving presentations more”.

6.2.5.2 Survey

A survey was administered in the last week of the final sprint and consisted of 10 questions.

Students were asked on a 7 point, Likert like scale whether they agree or disagree with the statement in that question. On the 7-point scale, 1 stood for Strongly Disagree and 7 stood for Strongly Agree, with 4 standing as Neutral. Out of the 30 students enrolled in class, 29 took the survey. Table 6.3 on page 170 describes the questions asked, the median, mode, standard deviation and variance and Margin of error with a confidence interval of 95 There were three main motivations behind the questions: asking about the students’ un-derstanding and ability to understand and perform the communication based activities;

assessing the degree to which students considered good communication important to the software process; and assessing the degree to which the communication based activities helped the team improve their process. The responses indicate that our instructional mate-rial is effective at building awareness of the importance of communication in real software

Table 6.3

Survey questions and analysis, with margin of error for a 95% Confidence Interval

Survey Questions Mode Median

Margin of Er-ror

"I understood how to perform the

communica-tion based activities" 6 5.71 0.38

"I was able to understand the rationale behind

the communication based activities" 4 4.79 0.54

"I was able to use the communication patterns

template easily" 6 4.79 0.59

"The communication based activities made my team more aware of our communication prac-tices"

5 4.86 0.45

"The communication based activities helped my

team improve our communication practices" 5 4.57 0.56

"What I learned about communication in this course was more relevant to my field than other courses about communication"

7 5.43 0.56

"This course has made me realize the impor-tance of communication in the software indus-try"

7 5.5 0.54

"Improved communication leads to a better

software making process" 7 6.32 0.30

"Improved communication leads to a better

software product" 7 6.39 0.30

170

development (median of 7=Strongly Agree). There is also indication of some success in conveying the pattern approach as a natural way for analyzing communication – though we must take care to assess whether the perceived ease of using patterns is associated with the rote responses we encountered in some of the submitted material. The survey does indicate a need for more support in building rationale for the communication activities (median of 4=Neutral); we must take greater care to explain why we are engaging in them. Finally, the survey responses indicate a moderate level of agreement (median of 5=Somewhat Agree) that the activities helped to improve teams’ communication process.

6.2.5.3 Written Assignments

In this paper we are restricting our grounded analysis to the assignments that were submit-ted collectively by each group. The assignments were coded for emergent themes associ-ated with communication reflection and design. Themes that emerged include: identifying flaws in their communication strategy; identifying improvements they can make to their communication choices; critiquing communication choices of others; discussing the im-pact of communication on their project; and identifying the audience or style of their com-munication choices. Table 6.4 on page 172 lists some frequently observed codes, arranged roughly by level of sophistication.

We made multiple passes through the submitted group assignments to code them and con-sidered each group’s journey. We noticed that even though all the teams were given the same instructions and adhered to the same project timelines, their project practices differed and their reflections differed a lot. However, we did notice that the depth of reflection in-creased over the semester as the teams had more practice reflecting, as indicated by a greater number of coded reflections in the final project story assignment compared to Sprint plans earlier in the semester and in the quality of their written assignments.

We are particularly interested in the evolution of each team’s attitudes and abilities regard-ing communication. At this early stage, we concentrate on individual scenarios: positive scenarios as confirmation that our approach can work, and less successful scenarios as an indication of how to improve it. As a “best case” scenario, we share the story of one team that truly seemed to embrace the spirit of reflection on communication. This team fol-lowed the structure of the Daily Stand-up meeting very faithfully and enforced the roles and responsibilities of the team members. Early in Sprint 2, the team noted that their meet-ings were working out quite well for them but “there’s always room for improvement” and that they needed to “keep on track” better and not “spend too much time trying to answer questions” [CRITIQUE].

Table 6.4

Sample of codes used in group assignments

Code Description

IMPACT Students explicitly discuss the impact or impor-tance of communication in their project

PRIOR Students relate their current communication ex-periences to their prior ones

EXPLAIN Students explain or defend their communication choices

CRITIQUE Students offer critiques of their communication choices

ALTERNATIVES Students identify improvements or choose alter-natives to the communication practices

AUDIENCE Students discuss the audience of the communi-cation act

STYLE Students discuss the style of their communica-tion act

As a solution, the team decided to keep a shared task document to allow team members to better prepare themselves for meetings and to eventually convert the task document into a task list (instances of considering [ALTERNATIVES]). The team also reported time wasted and confusion with scheduling and reminding members of meeting times. In addition, to reliably inform members of updates to documentation and project code, the team made a policy of sending out an email when any member makes changes that might impact code that another member is working on (instances where they mention how they had to change their communication practice, carefully considering [AUDIENCE]). The team noted that even in the second sprint, “many good instances of documentation ... were overlooked ... and only seen when they were mentioned”. In Sprint 2 and 3, the team reported that their user testing experience helped them learn a lot about their user’s needs (instances of considering the [USER]) and make changes to their software accordingly.

172

The team attributed its success to its strict adherence to Scrum practices in a formal, struc-tured manner (identifying that the [STYLE] of communication was important). In fact, in terms of complexity of delivered product, scope of project and completion of original goals, this student team would be considered the most successful. They also submitted the most reflection rich assignments (in terms of instances of reflections coded in their assignments).

In their final reflection, as the Project Story assignment, they describe their earlier practices and their shortcomings, followed by the alternatives they adopted and how they panned out. On coding their group story, we found several occurrences of identifying room for improvement in their current communication strategy, and some occurrences of attention to audience and style of their communication and product.