The Instrument Design Process

An iterative process was used in the design of the instrument. There have been at least three major iterations during the design. There were also several minor iterations to improve specific sections. The major steps in the instrument design are listed here in order to guide other researchers in their future studies.

a. Step 1: Search for the Sources of Information

In this step, the sources of information that can be used in the design of the instrument have been identified. Many sources of information were sought. These sources include software practitioners’ interviews, subject books, related standards, best

and worst practice guidelines, journal publications, conference publications, professional seminars, and other relevant written or verbal material that can be found on the World Wide Web (WWW). Another source of information was personal correspondence with some of the survey study participants. Most of these sources are referenced throughout the dissertation.

This search for information was conducted based on two main themes. The first theme was software project management knowledge areas and practices. Most of the relevant information was found in the project management and software engineering literature as well as via interviews with practitioners. However, especially for the human side of software project management, many sources were found in other disciplines such as organizational management, sociology, psychology, etc. The second main theme was how to measure or evaluate the effectiveness of these project management knowledge areas and practices. It is important to note that, especially for the guidance of other researchers, a big portion of the information in this theme did not come from project management or software engineering related literature, but from literature in other disciplines. These other disciplines include sociology, organizational management, organizational behavior, psychology, engineering management, human resource management, sales management and other related disciplines.

b. Step 2: Categorization of Information

All these sources identified in the previous step were carefully reviewed. The relevant information from these sources was extracted for use in the design of the instrument. Then the information was categorized based on relevance. A separate folder was created for each category to place the relevant information in one place.

These sources were also rated based on their relevance and applicability. This was important because most of the sources or studies were only applicable to some extent. In social sciences, it is possible to identify a few recognized questionnaires that are commonly used in studies. For instance, one such example is the organizational commitment questionnaire (OCQ) developed by Porter et al. (1974). This recognized

questionnaire was only applicable to a very limited extent. However, there was value in reviewing such a questionnaire. The development of a few questions and statements in the instrument was influenced by this questionnaire.

On the other hand, there were studies which influenced the development of the instrument to a great extent. One such example is CMMI v1.1 and v1.2. A significant number of questions in some of the project management areas were guided by CMMI.

c. Step 3: Detailed Analysis of Information Gathered

At the end of the first two steps in the design of the instrument, a significant amount of information was gathered. Among the information gathered, naturally there was redundancy. This redundancy, to a certain extent, was considered as an indication of the importance of a certain area, activity or entity. There were also pieces of information which only existed in a few sources. These were also carefully reviewed for inclusion in the design of the instrument. Even though some of this information was referenced in a few sources or the focus of a few studies, valuable insight was attained during the review. At the end of this step, a list of activities and entities was generated for each project management area. These lists contained activities and entities found related to software project management in a short, bulleted, categorized form.

d. Step 4: Development of Questions

The result of the previous steps was the creation of systematic lists for each project management area in the 3PR framework. For each item in these lists, a question was developed.

Careful consideration for the context was significant in the design of the instrument. For example, each section in the instrument corresponds to a project management area from the 3PR framework. This provided a context for the questions and reduced the necessary number of words used in wording the questions.

The questions were worded very carefully. The wording of the questions was kept as simple and straightforward as possible. The wording was very important since the reliability of the responses was very much tied to the quality of these questions.

Questions inquiring about similar issues were closely located within the sections in the instrument. Such localization reduced the amount of context switching required by the respondents while completing the questionnaire-based instrument.

The chosen question types were closed questions and statements with a Likert scale. These types enabled faster response time.

There were also other factors considered during the development of the questions. There are many specific subject books focused on guidelines for the development of questionnaires. These books were consulted extensively whenever necessary. A good one on the topic was authored by Ian Brace (2004) and it is titled “Questionnaire Design.”

e. Step 5: Interface Design

The interface design of the instrument was an important step, thus it is specifically mentioned as a major step. The first versions of the questionnaire-based instrument were more than sixty pages. This length is intimidating for many potential study participants. This length was reduced during major iterations.

Another important issue was the selection of a specific interface for each question. A number of different interfaces were tried. After the pilot studies on the instrument, the latest version of the instrument was finalized.

f. Step 6: Testing and Redesign

Pilot studies were conducted to test the instrument. After these pilot studies were concluded, it was understood that the content of the instrument was satisfactory. Only minor changes were found to be required. On the other hand, the interface was improved significantly. In the pilot studies, the study participants were

carefully observed for their reactions to the instrument. The interface was an important factor that required special attention in order to achieve better results. The interface took its final form after the pilot studies.