Computing Time Spent Using Programming Environment Events

McKeogh and Exton developed Eclipse Watcher [41], an Eclipse plug-in that logs students’ programming activities such as editing (insertions and deletions) code and navigating between files, the timestamp of these programming activities, the line number of a specific edit, and the file being edited. They used this logged data to compute the time spent editing on each line of code and the time spent editing and navigating. The goal of their work was to determine if these metrics could be used to determine the lines of code and files that are causing developers to have difficulty. This information could be used to find complex code and refactor it.

To compute the time students spent editing and navigating, the timestamp of the previous programming activity is subtracted from the timestamp of the current programming activity. A problem with this approach is that periods of inactivity could lead to inaccurate calculations of time spent. To overcome this problem, the timestamp of the current programming activity is disregarded if it occurs more than five minutes after the timestamp of the previous programming activity. This threshold was chosen arbitrarily. The time spent programming equals:

∑

where n is the number of programming activities, x is the timestamp of a programming activity,

xi is the ith timestamp of a programming activity, xi+1 is the ith + 1 timestamp of a programming

activity, and y is the number of minutes. To illustrate, consider the example log we created shown in Table 2.3 and Equation 3. The total time spent editing and navigating is 12 minutes.

Table 2.3: Our example of a programming activity log captured by Eclipse Watcher [41]. Id Timestamp Programming Activity Line Number File Name 1 3:40 pm Editing 1 Test.java

2 3:41 pm Navigating n/a n/a

3 3:50 pm Navigating n/a n/a

4 3:51 pm Editing 1 Test.java

5 3:52 pm Editing 1 Test.java

6 3:53 pm Editing 2 Test.java

7 3:56 pm Editing 2 Test.java

8 3:59 pm Navigating n/a n/a

9 4:10 pm Editing 2 Test.java

This paper did not give any indication of how to compute the time spent editing on each line of code in a file, but it provided a table, Table 3 taken from [41], which shows the amount of time a student spent editing on each line of code in a file. Rows represent a file and columns represent five lines of code. Based on Table 2.4, we assume that time spent on each line of code is computed by a) modifying Equation 2 to only subtract the timestamps of edit events if both occur in the same file and on the same line, b) summing the computed time of edits with the same file name and line number, and c) summing the result of the previous step for every five lines of code. For example, using Table 2.3, if we perform steps (a) and (b), the total time spent editing code on line 1 is 2 minutes and on line 2 is 6 minutes. If we perform step (c), the total time editing lines 1 through 5 is 8 minutes.

Table 2.4: Amount of time editing each line of code [41]. Rows represent a file and columns represent five lines of code.

To evaluate their tool, they logged three student programmers with varying levels of experience while working on a maintenance task. Their preliminary results show that their tool could be used to explain the navigation behavior of the students. More specifically, the student with the most experience spent the least amount of time navigating code. One possible reason is that the most experienced student understood the code faster than the inexperienced students.

Norris et al. [46] also used time spent to understand student programmers’ difficulty, but instead of focusing only on students’ navigation behavior, they focused on students’ compilation

and edit behavior. To realize this goal they created ClockIt [46], an extension of the BlueJ programming environment that a) monitors compilations, edits (insertions and deletions), and the number of lines of code and comments written, b) computes the number and percentage of failed compilations, time spent on an assignment, and project growth, and c) graphically displays this information to students and instructors through a web interface. This paper gave no indication of how time spent is computed, but we assume they use Equation (2). Project growth is computed daily using the following equation:

(4)

To determine the usefulness of these metrics, they logged and computed metrics for 75 students working on lab assignments. However, they only reported the results of three of those students. Their results show that a) the student with the least amount of time spent writing code wrote the least amount of code and got the worst grade of the three students and b) two of the students spent nearly the same amount of time writing code, but the student with the best grade wrote less code. The student with the best grade encountered fewer compiler errors and did not encounter the same types of compiler errors as the other two students. These metrics could be used to answer questions common to instructors teaching a CS1 course. Examples of questions are: a) how much time are students spending on their assignments? and b) what types of errors do students make?

Retina [42], a related tool, provides preliminary answers to these questions. It monitors students’ compilations and also their run time errors to a) estimate the amount of time students spent on assignments, b) predict how much time students would take on assignments, c) compute the total number of compilations, d) compute the total number of compilation errors, e) compute

the percentage of successful compilations, and f) compare these metrics for an individual student against the class’s average of these metrics.

Time spent is estimated by grouping compilation events that occur within a fixed time, e.g. 30 minutes, into individual programming sessions and summing the time spent on each individual session for an assignment. The time spent programming for a session can be computed using Equation 3 if we set y equal to a fixed time. Thus, the amount of time spent programming for all sessions equals:

∑

where n is the number of sessions, x is the time spent programming computed for a session using Equation 1, and xi is the ith session. The example log we created in Tables 2.5a, 2.5b, and 2.5c

show that a) a student had two programming sessions for an assignment , b) the same student spent 80 minutes programming during the first session (Table 2.5a) and 25 minutes during the second (Table 2.5b), and c) the sum of both programming sessions for an assignment is 105 minutes (Table 2.5c).

Table 2.5a: Student’s programming activity log for session 1.

Table 2.5b: Student’s programming activity log for session 2.

Programming Activity Timestamp

compilation event 3:40 PM compilation event 3:45 PM compilation event 3:54 PM compilation event 4:00 PM compilation event 4:15 PM compilation event 4:38 PM compilation event 4:50 PM compilation event 5:45 PM

Total Time Spent: 80 minutes

Programming Activity Timestamp compilation event 5:49 PM compilation event 6:04 PM compilation event 6:13 PM compilation event 6:14 PM compilation event 6:55 PM Total Time Spent: 25 minutes

Table 2.5c: The sum of the time spent for sessions 1 and 2 on an assignment. Time spent Session 1 80 minutes Time spent Session 2 25 minutes Total Time 105 minutes

Retina estimates the amount of time it would take a student to finish an assignment based on how much time (a) students in a previous offering of a course took on their assignments and (b) students in a current offering of the course took on previous assignments. Let Sc denote all

students in the current semester, Sp all students in the previous semester, j the student whose time

is being estimated, and n the assignment for which time is being estimated. The estimated

amount of time it would take j to finish n is computed by a) averaging the amount of time Sc took

on previous assignments, b) ranking j’s average time spent on previous assignments with respect to Sc, c) averaging and ranking the amount of time Sp spent on all assignments, and d) finding the

time it took similarly-ranked Sp to complete n. For example, a student with Id of 2 in Table 2.6a

a) ranked 2nd in terms of average time spent on all previous assignments, b) spent a similar average time on assignments as a student with an Id of 3 in a previous semester (Table 2.6b), and c) is predicted to spend approximately 107 minutes on an assignment, which is the same time that a student with an Id of 3 spent on that assignment (Table 2.6c).

To evaluate Retina, they a) logged 48 students doing class assignments and b) asked a small number of students and three instructors to comment on the usefulness of the tool.

determine that their classmates also struggled with the material. Instructors used to tool to a) determine the types of compilation errors that caused students to have difficulty, b) anticipate the questions students would ask, and c) tailor their help based on students’ compilation errors. They focused on compiler errors because student programmers often struggle with getting their programs to compile correctly [37].

Table 2.6a: Average time students spent on all

assignments.

Table 2.6b: Average time students taking a previous offering of a course spent on all assignments.

Table 2.6c: The time students spent on an assignment in a previous offering of the course.

Current Semester

Student Id Average time spent on previous assignments 1 165 minutes 2 175 minutes 3 177 minutes 4 179 minutes Previous Semester Studen t Id Average time spent on previous assignments 1 150 minutes 2 165 minutes 3 172 minutes 4 185 minutes Previous Semester Student Id Time spent on assignment in previous semester 1 95 minutes 2 105 minutes 3 107 minutes 4 113 minutes