Adoption of Pair Programming in the Academic Environment with Different Degree of Complexity in Students Perspective– An Empirical Study

(1)

Adoption of Pair Programming in the

Academic Environment with Different

Degree of Complexity in Students

Perspective– An Empirical Study

V.Venkatesan1*, A.Sankar2*

*Department of Mathematics and Computer Applications, PSG College of Technology, Coimbatore, INDIA [email protected], [email protected]

Abstract

Agile software development is a light weight process which is a solution for the heavy weight processes. EXtreme Programming is one of the Agile Methods. Pair Programming is one of best practice of eXtreme Programming. . It is a technique which concentrates not only on the software but also the developer who working on the software. Pair Programming is best suitable to produce efficient software. It accentuates the practice of two persons working together at a single computer terminal, to design, code and test computer programs. We studied the effects of pair programming in student context on knowledge transfer and enjoyment of work. We analyzed whether the task complexity affects the effort differences between solo and pair programming. We conducted experiment to evaluate the efficiency of the knowledge transfer of the students when they adopt pair programming. The feedback from the student show the positive result that pair programming improves knowledge transfer and enjoyment of work. Through the experiment we also analyzed whether there is dependency between the task complexity and students capability. The result of this experiment shows that there is significant difference in the duration of task completion, the effort to perform the task correctly between solo and pair programming.

Keywords — Pair programming, Agile Software Development, Extreme Programming, Pair rotation

1. Introduction

Agile software development is a philosophy which is a response for the traditional way of developing the software [Beck, et. al., 1999]. Agile methodologies seek to focus on people and to utilize the power of software in software development. In traditional methods, work has been start with the documentations which contains set of requirements. Agile process reduces the number of rules. The highest precedence is to satisfy the customer and to reduce the time taken for decision making [Cockburn, A. and Highsmith, et. al., 2001c]. EXtreme Programming (XP) is one of the agile methods. It is actually a deliberate and disciplined approach to software development. This methodology is designed to deliver the software based on customer needs when it is needed. It also emphasizes team work in which managers, customers and developers are all part of a team dedicated to delivering quality software [4]. XP was created in response to problem domains whose requirements change.

(2)

2. Pair Programming

. Pair programming is a practice in which two programmers work side-by-side at one computer, continuously collaborating on the same design, algorithm, code, or test. One of the programmers is called the driver, responsible for entering software design, source code and test cases, while the second programmer called the navigator examines the driver’s work, offering advice, suggesting corrections and assisting with design decisions.

Pair Programming has been revealed to improve productivity and quality of the software. The roles of pair programming are driver and navigator where the driver writes the code and the navigator helps the driver and looks for errors. Several experiments of Pair programming on software development in industries have been conducted and its efficiency has been evaluated. The results show that pair programming improves the productivity by 15% [Kalimuthu Vivekanandan, Subbaraya Kuppuswami, et. al., 2004a]. By the analysis of Arisholm, Gallis, Dyba, Sjoberg et. al. (2007). PP has many benefits over individual programming, including significant improvements in functional correctness. The navigator watches for defects, thinks of alternatives, looks up resources, considers strategic implications, and asks questions. Through these activities, the navigator contributes by identifying tactical and strategic deficiencies in the design or code. The main objective of pair programming is to provide better quality and knowledge transfer. When doing pair programming partners are witnesses to each other’s strengths and weaknesses. Confidence is a factor in successful pairings, that is, partners must attain a level of comfort with each other. A person will pair differently with different partners. Pair programming is a dynamic and adaptive process. Teams should evaluate what has to be done and who should pair to work on the tasks. After the tasks are done, new activities are planned and new pairings arranged. Also there are some drawbacks in pair programming based on the use of pair programming by the developer. Choosing of pair is the most unnoticeable factor in pair programming. Most of the studies say that if random pair is chosen bad result might be yield. Rotation of pairs should be done for knowledge transfer. Also paring with the same person is not good in pair programming.

Every so often, the driver and navigator switch roles, this allows the programmers to vary their work routine, learn and practice new skills, and contribute to the work in more than one manner. There is no set rule as to how often the programmers should switch roles [Hema Srikanth, Laurie Williams, Eric Wiebe, Carol Miller, Suzanne Balik. et. al., 2004b]. It should occur at a natural transition point in the design or coding activity. Sometimes this may be after only one or two hours, or sometimes it may be after days. In organizations where there is more than one pair, pairs often rotate partners after the completion of a task or subtask. Pair rotation allows the programmers to learn different areas of the system, and learn from the skills of different partners. When two programmers are first paired, there is often a jelling period during which time each is adjusting to the working habits of the other, learning the other’s strengths and weaknesses, and learning how to communicate effectively [Hema Srikanth, Laurie Williams, Eric Wiebe, Carol Miller, Suzanne Balik et. al., 2004b] While productivity is somewhat less during this period, it soon increases as the pair becomes comfortable working together.

Working with a more experienced partner can be a great way to learn about programming, designing, planning, standards and project details. Knowledge will spread easily throughout a team that uses pair programming. Experienced members of the team will benefit by seeing the project through other eyes. A successful pair-programmer not only knows and adheres to the formal guidelines, but also is able to resolve intra partner stress through practical communication.

There are variations in the way that organizations structure their workplace environment to support pair programming. The most common configuration is to use one monitor, one keyboard, and one mouse. The programmers sit next to each other so that they each have a clear view of the monitor. Control can be passed by simply sliding the keyboard and mouse, or less optimally by sliding chairs. In order to facilitate sitting next to each other and sliding control, pair programmers often situate their chairs and equipment around open, rather than closed, corners, as illustrated in Figure 2.1

Fig.2.1 Pair Programming Workspace Layout

(3)

Programming Language subject, their performance and confidence towards developing good programs with the content of knowledge transfer. The sharing of knowledge when solving Non programming mathematical problem has also analyzed by using the concept of pair programming.

Our research involved experiments which has been conducted in two phases: 1) Programming Experiments, 2) Non Programming Experiments. Pair programming has been indicated that it affects students’ performance, confidence and knowledge.

3. Existing Empirical Studies on Pair Programming

Several experiments have been conducted to analyze the efficiency of pair programming. The experiments were conducted both in industrial environment and in educational institutions. The experiments conducted have been based on the parameters: time, quality, cost, knowledge transfer, confidence of the developer, learning enhancement. In this session we summarize some of the important empirical studies on pair programming.

An experiment conducted at the University of Utah compared 13 individuals using the Personal Software Process (PSP) with 14 pairs using the Collaborative Software Process, which includes PP. The students had to complete four assignments in 6 weeks. The experiment compared the time to complete the assignments (in number of hours from start to finish), the cost (indicated by the number of programmer hours) and the quality of the solutions (indicated by the number of passed test cases) between the two groups. After an initial “adjustment period” (the first assignment) the pairs spent, on average, only 15 percent more programmer hours than the individuals. Consequently, the pairs spent less time than the individuals, but at increased cost [J. T. Nosek, et. al., 1998]. All pairs also passed more of the test cases than the individuals. Thus, it is difficult to draw conclusions about the cost benefits, because the individuals might have produced similar quality if they had spent 15 percent more time. However, in terms of trust and morale, more than 90 percent of the students using PP reported that they enjoyed it more than programming alone. Almost 95 percent of the students stated that they had more confidence in their solutions when programming in pairs.

Another experiment have been conducted by Nawrocki and A. Wojciechowski with 21 students, comparing three different groups: (1) six students using PSP, (2) five students using a modified version of XP with individual programming, and (3) 10 students (in five pairs) were using a modified version of XP including PP. [J. Nawrocki and A. Wojciechowski et. al., 2001a]. They found almost no difference in the development time (indicated by the number of hours from start to finish) between the three groups. Furthermore, there was almost no difference in quality (indicated by the number of lines of code and number of re-submissions due to defects in code). Consequently, the results suggest doubled costs for the pairs compared with the individuals – without increased quality. The effects of PP on student performance was studied in an introductory programming class. The study consisted of 313 students where PP was compared with individual programming. The results indicated that the pairs produced “better” programs (indicated by the grade obtained on their delivered assignment). The authors suggested that the students who used PP benefited from the information and knowledge transfer process of PP, because fewer students who used PP dropped the course than students who worked individually.

Ciolkowski and Schlemmer had six-person student teams spend 13 weeks and about 700 hours of effort for the projects [M. Ciolkowski and M. Schlemmer et. al., 2002] However, the experiment covered only the programming phases lasting 5 weeks and taking about 240 hours of effort. The researchers were not able to evaluate quality using defect metrics, but analyzed LOC and the coupling factor instead. These metrics showed slightly better values for the pair programming teams. The pair programming teams spent 9% more effort for the programming tasks

4. Research Design

The research consisted of an experiment whose design we describe below. Then we present the hypothesis we aimed to study

4.1 Experimental Design

The experiment has been conducted in two phases: i. Programming course

ii. Non Programming course

(4)

applied to solve mathematical problems.

Hypothesis

Hypothesis 1: There is no significant difference in the duration when the students adopt pair programming and traditional method

Hypothesis 2: The difference in time taken to perform task for pairs and individuals does depend on system complexity

Hypothesis 3: There is a significant increase in effort to perform the task correctly when students adopt pair programming

Hypothesis 4: There is a significant increase in correctness to perform the task correctly when students adopt pair programming

4. 1. 1 Programming Course

The study was applied to 60 students of first year Engineering program. A small introduction about the pair programming and experiment was explained to the students for 10 minutes. The students were supposed to involve in both traditional method and pair programming. The students were allowed to program in C++. First, they were supposed to adopt solo programming for 2 hours. Same problem solving question has given to all the students for solo programming. When they finish the problem it was evaluated immediately and the completion time has been noted down. Students name and their completion time has been noted. When the session time elapsed all the students were asked to stop programming and the programs have been saved in the server for further evaluation.

After break, the next session of programming has been started for pair programming. Students adopt pair programming in this session. Students were asked to choose their partners of their own. Another problem solving question has given to all the students. For each pair they choose the role of driver and navigator. They were asked to frequently change their roles. As in solo programming session, in pair programming session also the completion time has been noted for each pair. Finally, a feedback session was there to get the feedback of the students about the solo programming and pair programming. A feedback form was given to the students to choose their options. The Laboratory session was evaluated based on the parameters in terms of Duration, Quality of the code, and Effort.

Duration: Duration was defined as the elapsed time taken to perform a set of change tasks. Since it is not

meaningful to compare duration for programs that require rework with programs that do not, we only considered duration for subjects whose work was correct.

•

Effort: Effort was defined as the total number of programmer hours taken to develop a correct program. Thus,

effort equals duration for the individuals and twice the duration for the pair programmers. Note that the effort and duration measures are clearly not independent, but we considered both measures to be important as they provide complementary insights of the costs and benefits of pair programming.

Correctness: Correctness was defined in terms of whether or not the final, maintained program possessed the

required functionality, i.e., a binary score.

4. 1. 2 Non Programming Course

Two experiments have been conducted for Non Programming Course. Experiment on Non Programming Course is similar to Programming experiment. In Programming experiments students were allowed to do program for the given problem. Whereas, in Non Programming experiments students were given the mathematical problem for solving. The experiments were conducted in the classroom environment. The study was applied to students of engineering program who have mathematics as a course. A small introduction about the pair programming and experiment was explained to the students for 30 minutes. The students were supposed to involve in both traditional method and pair programming. Each experiment has conducted for two hours. First, they were supposed to adopt solo programming for 1hour. Students name and their completion time have been noted. When the session time elapsed all the students were asked to stop solving and the answer sheets were collected for evaluation.

(5)

answer sheet has been collected for evaluation. Finally, a feedback session was there to get the feedback of the students about the solo programming and pair programming. A feedback form was given to the students to choose their options. The parameters involved for evaluation are the effort and the duration.

4. 1. 3 Experiment arrangements Subjects

The subjects of the case study are categorized in to two groups. One group assigned for Non Programming assignments and the other group for Programming assignments.

Group 1: Students who have Laboratory subjects such as Programming Languages(C, C++, Java)

Group 2: Students who have Mathematical subjects such as Statistics, Numerical Analysis, etc

Variables

The dependent variables were: the effort spent for carrying out the assignments and the achieved quality. The effort was measured by the time spent for accomplishing the tasks: the subjects were required to write the start and the end time in the assignation form for each task. The quality was evaluated by grading the work of the subjects.

Process

The process of the experiment was the following:

 Each subject studied the problem for 5minutes, individually

 At the first phase subjects involved in solo problem solving. Both in programming and non programming task they solved the problem individually. The duration and effort has been recorded. The task lasted 1hour.  At the second phase subjects involved in pair problem solving. The pairs are chosen randomly. As in the

first phase the duration and effort has been recorded. This phase also lasted 1 hour.

4. 2 Choosing of Pair

The main task in pair programming is that choosing the pair. Choosing the right pair yields better productivity. In Organizations, the different options for choosing the pair are used. They are:

 Expert-expert pairing.  Expert-average pairing.  Expert-novice pairing.  Novice-novice pairing.

In educational environment the students can be categories based on their average marks yield by the previous semesters. For the students, a pretest can be conducted to know the knowledge of the students. Based on the marks scored by them in the test the students can be categorized [L. Williams and R. Kessler. et. al., 2003b]. According to the categories they can be assigned with the partners. In some situations we can go for random pairs. But random pair, at sometimes produces fair productivity. Also, if a student assigned with the partner who is enemy to him/her, the knowledge transfer concept will fail. So choosing the right pair provides better performance and it increases the enjoyment of the students when programming.

4. 3 Rotation of Pair

One way of implementing it would be to pair everyone and have five pairs that would work together. We believe a preferred way of implementing pair programming would be to have the pairs switch partners periodically. The advantages of rotating are that the programmers learn more about the whole product by pairing with different people, they team with the person who can help them the most on a particular task, and communication increases significantly [Hema Srikanth, Laurie Williams, Eric Wiebe, Carol Miller, Suzanne Balik et. al., 2004b]. Also, we don’t have to worry as much about pairs becoming dysfunctional and the need to switch around all the pairs because two particular people just can’t get along.

Good knowledge management techniques are important for employers and employees alike. Truly, it is not advantageous to have all knowledge in any area of a system known to only one. In Industry, for the employer, this is very risky because the loss of one or a few individuals can be debilitating. By rotating pairs, knowledge is disseminated. Employers reduce risk and employees can relax when out of the office.

(6)

This familiarity serves to break down many communication barriers. Team members find each other much more approachable [Hema Srikanth, Laurie Williams, Eric Wiebe, Carol Miller, Suzanne Balik et. al., 2004b].They will struggle with questions or lack of information for less time before getting themselves out of their chair and going ask the right person a question – because they know that person quite well.

4. 4 Data Collection

The data from the experiments have been collected based on the duration, effort and quality. After each session the completion time of the task has recorded. The effort of the experiments is measured based on the programmer hours taken to complete the task.

Duration

The completion time of the task by the programmer students are recorded in the problem sheets which has been used for further calculation. Using the data, the mean duration of solo programmers and the pair programmers are calculated

Effort

For a task done by the students, the programmer hours taken for completion is calculated. So from the calculation it has been verified whether there is improved performance done by the programmers when they adopt for pair programming.

Quality

The quality of the task has been evaluated by giving marks for their output and by verifying the code. By this data it has been verified whether the quality of the code has been improved when the students adopt for pair programming.

4. 5 Knowledge Factors

Information and knowledge can be both explicit and tacit. PP may be a good way to share tacit knowledge through the partners’ visual observation of each other’s work. It is, unfortunately, difficult to quantify the “amount” of information and knowledge transfer directly. One potential indirect measure might be the amount of code reuse, and the degree to which coding standards within a project are adhered.

In the Pair Programming teams developers generally had high involvement in more packages than in the Solo Programming teams. Probably related to this, there were generally more developers in the Pair Programming teams with good understanding of each package, and each developer understood more packages well [Laurie A. Williams and Robert R. Kessler et. al., 2000a]. This indicates better knowledge transfer within the Pair Programming teams.

In our research, we have collected feedbacks from the programmers about the knowledge gained because of pair programming. We received positive feedbacks about knowledge transfer. It seems that the use of pair programming leads to fewer defects in code after coding and better knowledge transfer within the development team without requiring additional effort if the learning time can be avoided.

4.6 Threats of validity

4. 6. 1 Threat to Internal Validity

(7)

Differences among subjects

Using a within subjects design, error variance due to differences among subjects is reduced. In this experiment students were skilled enough to deal with the kinds of problems presented by the experimental package.

Learning effects

The subjects executed introductory runs before the experiment in order to familiarize with the practice of pair designing. As discussed above, significant differences in the runs were detected for the effort results. It was due to a greater knowledge of the system acquired by working, and it did not invalidate the results of the experiment. As a matter of fact, the analysis concerned pairs and solos of the same run: the bias did not concern the samples to compare.

Fatigue effect

On average the experiment lasted a time short enough, so that fatigue was not very relevant. As a confirmation, the subjects asked for a longer time to better accomplish the assignment.

Subject motivation

The participants were volunteer, in order to help us in our research.

4. 6. 2 Threats to External Validity

The requirement to use pair programming for all development work was not the most natural and possibly also not the most useful choice. The optimal amount of pair programming may be anywhere between using it for all development tasks by everyone and not using it at all. Pair programming was taught to the students but we did not observe how actively they changed roles and communicated during the pair programming sessions.

In a typical industrial setting pair programming may decrease the amount of interruptions by other people compared to when a person is working alone, but this effect probably did not show up in our experiment. We do not know how well the team members knew each other and what kind of personalities they were. Both of these variables may have affected the success of the teams. Potential participants knew before entering the course that half of the participants must use pair programming, which may have kept away people who are strongly against pair programming.

Subjects

Using students in a controlled experiment is not always a threat for the external validity. It happens when the skills of the subjects are suitable to accomplish correctly the tasks: the case of this experiment.

5. Experimental Result

5. 1 Data Analysis

The data collected has been used to evaluate efficiency of pair programming based on the parameters. It has been found that there is no significant difference in the duration when the complexity of the problem given is easy. When the complexity increased it has be verified that the duration taken for solo programmers is more than the duration for pair programmers. The Figure 5.1 and 5.2 shows the chart of efficiency of duration in pair programming.

(8)

a significant increase by the Pair programmers. The Fig 5.3 shows the chart where the difference in the effort has been visualized.

Fig 5.3 Effort Efficiency

5. 2 Testing of Hypothesis

Hypothesis 1

There is no significant difference in the duration when the students adopt pair programming and traditional method. From the data collected based on the duration, the significant difference of two programming methods have been evaluated.

Null Hypothesis : The duration for pair programming is same as for solo programming.

Alternate Hypothesis: The duration for pair programming is lesser than solo programming.

Hypothesis 2

The difference in time taken to perform task for pairs and individuals does depend on system complexity. The problems with different complexity have given to the students to check whether there is difference in duration because of complexity. Based on this hypothesis has been set as follows

Null Hypothesis: The duration depends on complexity for pairs and individuals

Alternate Hypothesis: The duration does not depends on complexity of the task

Hypothesis 3

There is a significant increase in effort to perform the task correctly when students adopt pair programming. Pair programming’s efficiency is mainly relied on effort in our experiment. The effort is keenly observed so that there could be an improvement because of pair programming.

Null Hypothesis : There is no difference in the effort when the students adopt solo and pair programming

Alternate Hypothesis: Pair programming produced improved effort compared to solo programming

Hypothesis 4

There is a significant increase in correctness to perform the task correctly when students adopt pair programming. The problem has been evaluated to check whether it satisfied the questions

Null Hypothesis : There is no difference in the correctness comparing solo and pair programming

Alternate Hypothesis: Pair programming produced improved compared to solo programming

(9)

in the duration and the program complexity. When the problem given is easy then there is no significant difference in the time taken by the solo programmer and the pair programmer. When the complexity of the program is increased then the time taken will be reduced by the pair programmers than the solo programmers. The Table 5.1 shows the experimental results in which the difference in the duration has been visualized.

Table 5. 1 Comparisions of Solo and Pair Programming in terms of Duration

Duration

Techniques Number of _Subjects Mean Standard _Deviation t - Value

Solo Programming 30 48.3 5.97

0.675

Pair Programming Pairs : 30

Total : 60 42.4 5.53

The Table 5.2, shows the experimental results with the comparison of Solo and Pair Programming in terms of Duration with different complexity.

Table 5. 2 Comparisions of Solo and Pair Programming in terms of Duration (Different Complexity)

Duration (different Complexity)

Techniques Number of

Subjects Mean

Standard

Deviation t - Value

1.7839

Total : 60 30.6 5.53

The Table 5.3, shows the experimental results with the comparison of Solo and Pair Programming in terms of their effort during the experiments.

Table 5. 3 Comparisions of Solo and Pair Programming in terms of Effort

Effort

Standard

3.69

Total : 60 5.34 1.75

The Table 5.4, shows the experimental results with the comparison of Solo and Pair Programming in terms of their correctness of code during the experiments.

Table 5. 4 Comparisions of Solo and Pair Programming in terms of Correctness

Correctness

Standard

3.76

Pair Programming Pairs: 30

Total: 60 4.5 1.89

(10)

6. Conclusions and Future Work

The Pair Programming based on the driver-navigator model has been successful in the learning environment. Its success is noticeable in better performance in computer science programming and non programming mathematical assignments, increased team work in and outside class, enhanced learning, and decreased frustration We have conducted three experiments with students in both programming and non programming course. The data collected from the experiments have been used to analyze about the efficiency of pair programming. The results showed that there is significant increase in the knowledge when using pair programming both in the programming course and non programming course. Furthermore, unlike previous experiments, the effect of PP was investigated in the context of performance and knowledge transfer. If each person in the team paired with only one other person on the team, many benefits would be realized – quality gains, communication gains, teamwork gains, trust building, etc. However, even more can be gained by dynamically assigning the partners based on the task at hand. Two of these additional gains relate to knowledge management and to training. Pairing is an excellent knowledge management strategy because it breaks down communication barriers between teammates. They get to know each other and each seems more approachable when a question arises. Also, by pairing with many different people, developers learn more about many aspects of the system and more about the programming and tool techniques of many different people. Pair rotation also reduces the training time to assimilate a new team member and distributes the burden of training throughout the team. We are also developing software tools that will aid in the analysis of the experimental sessions. This logging can be used to generate automatically a set of measures for comparing development styles, and serves as an analysis baseline that can be extended by hand-coding. Our research on pair programming will continue by performing case studies at companies using pair programming. In Industries it is very challenging to arrange even quasi-experiments, but on the other hand case studies can give valuable qualitative information on how pair programming should be practiced in industry.

7. References

[1] Arisholm. E, Gallis, H., Dyba, T., Sjoberg, D.(2007). "Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise," IEEE Transactions on Software Engineering, vol. 33, no. 2, pp. 65-86, Feb. 2007

[2] Beck, K., (1999) "Embrace Change with Extreme Programming," IEEE Computer, pp. 70-77, Oct. 1999 [3] Beck. K., “Extreme Programming Explained : Embrace Change”, 2000. Addison Wesley.

[4] Ciolkowski .M and Schlemmer. M, (2002) “Experiences with a Case Study on Pair Programming”, In Workshop on Empirical Studies in Software Engineering, 2002.

[5] Cockburn, A. and Highsmith, (2001b) J., "Agile Software Development: The Business of Innovation," IEEE Computer, pp. 120-122, Sept. 2001

[6] Cockburn, A. and Highsmith, (2001c) J., "Agile Software Development: The People Factor," IEEE Computer, pp. 131-133, Nov. 2001 [7] Gallis, H., Arisholm, E., Dyba, T. (2003c). An initial framework for research on pair programming. In: Proceedings of International

Symposium on Empirical Software Engineering, Italy, pp. 132–142.

[8] Hanks, B. (2003d) Empirical Studies of Pair Programming, in 2nd International Workshop on Empirical Evaluation of Agile Processes (EEAP 2003). 2003.

[9] Hema Srikanth, Laurie Williams, Eric Wiebe, Carol Miller, Suzanne Balik. (2004b). "On Pair Rotation in the Computer Science Course," cseet,pp.144-149, 17th Conference on Software Engineering Education and Training (CSEET'04), 2004

[10] Kalimuthu Vivekanandan, Subbaraya Kuppuswami. (2004a). “The Effects of Pair Programming on Learning Efficiency in Short Programming Assignments”, Vol 3-2/2004, pp. 251-266

[11] Laurie A. Williams and Robert R. Kessler. (2000a). All I really need to know about pair programming I learned in kindergarten. Communications of the ACM, 32(5):108–114, May 2000.

[12] Laurie Williams, Robert R. Kessler, Ward Cunningham, Ron Jeffries. (2000b). "Strengthening the Case for Pair Programming," IEEE Software, vol. 17, no. 4, pp. 19-25, July/Aug. 2000.

[13] Nawrocki.J and Wojciechowski.A. (2001a). "Experimental Evaluation of Pair Programming," proc. European Software Control and Metrics (Escom), 2001.

[14] Nosek.J.T, "The Case for Collaborative Programming. (1998). " Communications of the ACM, vol. 41, no. 3, pp. 105–108, 1998. [15] Padberg. F and Muller.M.M (2003a) ”Analyzing the Cost and Benefit of Pair Programming,” Proc. Ninth Int’l Software Metrics