SURVEY DESIGN

The empirical data of this study was gathered through a survey administrated to the whole population of the companies.

4.2.1 _{Questionnaire}

The survey questionnaire was constructed based on the thorough analysis of the relevant literature presented in previous chapters. The survey questionnaire was eight pages long and divided into two parts. The first part focused on the background information of the company, also including questions related to the background of the respondent. The second part focused on the front end project itself, which was a unit of analysis in this study. The respondent was asked to select the most recent significant product development project (independent of its success or failure) where the front end phase was already completed. Respondents were asked to answer the questions based on this example front end project.

The questions covered different control mechanisms (independent variables), front end performance measures (dependent variables), and also some contextual information regarding the front end project. In addition, some key figures of the company such as turnover were requested at the end of the questionnaire. The questionnaire included 48 main questions in total. However, these questions additionally included sub questions, resulting in a total of 130 individual questions.536

The majority of questions were multiple choice questions in which the respondent was asked to choose an appropriate answer. Two measurement scales were used. The first scale measured items on a typical five-point Likert scale with the options strongly disagree, somewhat disagree, do not agree or disagree, somewhat agree, and strongly agree. The other scale measured items by focusing on the extent to which a certain issue was applied in the front end project. The options were not at all, to a little extent,

535

Technology Industries of Finland membership catalogue 2005 536

to some extent, to a great extent, to a very great extent. In addition, all scales included a “do not know” option for those informants who were unable to answer the question. The questionnaire took approximately 30–40 minutes to complete. The language of the survey form was Finnish.

The name of the respondent was asked at the end of the questionnaire to enable the delivery of the following advantages: 1) a free front end seminar based on the results of the research project, 2) a summary of the survey results, and 3) a book focusing on front end issues based on the results of the research project. Confidentiality was emphasized in this section and contact information was asked for only if the respondent found some of the above advantages attractive. Only nine (6.6%) of the respondents refused to give their contact information and so anonymity was not a problem. In addition, 92% of respondents requested on or more of the three benefits promised, which illustrates the motivation to participate in this survey.

Before sending the questionnaire it was tested both with academics and practitioners. As suggested by Fowler537, the questions designed were subjected to a critical systematic review. The questionnaire was discussed with four experienced researchers including the instructor and the supervisor of this thesis. The purpose was to improve the wording of the questions and increase their accuracy, understandability and validity. In addition, the questions were discussed with a survey research expert from the methodological point of view. The improved questionnaire was field pre-tested with five practitioners representing different companies538. Practitioners filled in the questionnaire, which was followed by a short interview with the purpose to clarify whether the questions were clear, understandable and relevant, whether the survey covered all relevant areas under the investigated phenomena, as well as whether the instructions to answer the questionnaire were clear. Field tests lead to minor modifications of some questions, but also to one unclear question being changed for a better one. 537 Fowler 2002 538 Ibid.

4.2.2 _{Mailing process}

The communication process included three separate types of contact with the company representatives. First contact was made via mail and consisted of a cover letter (Appendix A) emphasizing the importance of the survey, response instructions (Appendix B), the eight-page questionnaire (Appendix C), and a pre-paid return envelope. This package was addressed to the 888 respondents personally. The cover letter included a definition of a suitable respondent; if the addressee was not in the required position, the person was advised to forward the survey to the right person in the organization. Three weeks after the mailing, a second contact was made as suggested by Dillman539. This was done with an e-mail reminding the respondent about the survey. A different contact method was used to make contact effective with non-respondents more effective540. The e-mail also included an electronic copy of the survey as well as a link to the electronic version of the questionnaire on the web. Final contact was made by a phone to 50 randomly selected non-respondents to increase the sample size.

The questionnaire was sent to 888 respondents. A total of 21 companies indicated that they did not carry out product development activities or the questionnaire was returned because the respondent was no longer with the company. This resulted in a maximum sample of 867 companies. Of these companies, 137 returned the completed questionnaire, which leads to a response rate of 15.8%. The response rate can be considered as acceptable in the light of the long questionnaire and the fact that the questionnaire was targeted toward the director-level group where time resources are always scarce. Out of the 137 returned questionnaires, three companies participated in the survey with two business units, and in addition the respondents indicated that they had a company-wide approach to controlling front end projects. One business unit was randomly selected from each of these three pairs and removed from the final sample as inclusion of both business units in these pairs would have naturally biased the sample541. In addition, one uncompleted answer was removed because the returned questionnaire included only 26% of the requested data. The final usable sample for statistical testing was 133.

539 Dillman 2000 540 Ibid. 541 Birnberg 1988

When surveys rely on the responses of a single informant, special attention should be paid that the informant is knowledgeable in the survey domain542. Even though controversial opinions of applicability of a single informant have been presented543, it was considered to be a suitable approach in this study. The questionnaire was sent to the R&D director, research director, technology director, CEO, BU director or R&D- responsible person in each company, who were considered to be the key informants in the investigated phenomenon. As can be seen from Table 7, the great majority of respondents (92.5%) held one of these positions. The respondents had 5.7 years of experience (range: 0–30) in their position on average and 12.8 years of experience (range: 0–40) in the organization on average.

Table 7. Organizational position of respondents.

Position

Number of

respondents Percentage

R&D Director, Development Director, Research Director 52 38.8%

R&D Manager, Research Manager, Development Manager 34 25.4%

CEO, BU Director 20 14.9%

Technology Director, Technical Director 17 12.7%

Others 10 7.5%

Total 133 100.0%

4.2.3 Missing value analysis

Only 1.56% of the data of used measurement items were missing, which indicates that the returned questionnaires were completed thoroughly. The missing values were visually inspected to find possible patterns of missing data, but no such patterns were found. The influence of missing data seemed to be insignificant and random. In a few cases, the answer was unclear and it was impossible to correctly interpret it. In these cases, the answer was left blank. Some respondents presented answers as a range (e.g. 1–3) in some descriptive background questions. In these cases, the average figure was used. Listwise exclusion of data was used in the case of missing data.

542

Campbell 1955, John and Reve 1982 543

4.2.4 _{Non-response analysis}

A potential problem in mail surveys is the possibility of bias that results from low response rates544. The response rate in this study was 15.8%, which is reason to study a possible response bias. One method to investigate the response bias is to compare early and late respondents of the survey545. Armstrong and Overton have suggested that late respondents, who responded because of the increased stimulus, are relatively similar to non-respondents546. Possible response bias was analyzed by testing a difference in turnover, number of employees and R&D intensity (percentage of turnover to R&D) between early (63 companies) and late (70 companies) respondents. The results of this test are presented in Table 8. No statistically significant differences were found between early and late respondent groups. The results indicate that response-bias is not a problem in this study and the sample can be considered to be representative of the target population (see Chapter 4.5.2). This supports the findings of Krosnick, who concluded that a low response rate does not necessarily mean that a survey includes a non-response error547.

Table 8. Independent sample t-test of difference between early and late respondents in terms of selected variables.

Analyzed Early respondents (N=63) Late respondents (N = 70)

figures Mean (Std. Dev.) Mean (Std. Dev.) t df Sig. (2-tailed)

Turnover 1661.55 (5258.80) 391.79 (780.84) 1.88 63.38 .06 % of turnover to R&D 5.11 (12.93) 9.53 (28.37) -1.11 86.23 .27 Number of employees 5412.97 (15617.33) 1982.57 (4514.20) 1.68 71.31 .10