3 Methodology and Research Design 45
3.4 Validity and limitations 53
The principle purposes of this study should contribute to our understanding of public policy on sustainable growth, by focusing upon the diffuse, indirect and long-term effects. Hence, the discussions of validity and causality should be seen in this context.
The research design agreed upon is based on capturing the long-term developments within research and industry. Within the overall
developments, we address specific issues, using multiple indicators, data, and methodologies.
The first issue is the choice of focusing upon the specific programs and projects that were financed by STU, Nutek and VINNOVA. The choice was made in consultation with experts. For Innovative food, we accessed all the specific project applications and reports, and could therefore focus upon the project level. For Medical technology, we had lists of programs and
projects, but this was a much longer list, further back in time, and with less detailed information on each project. We decided to focus upon the larger picture of research environments and interactions with industry.
Some selection bias was introduced, in that we did not examine projects that were rejected. One pragmatic reason was that one could have relatively easily found the rejected projects through archive studies, but only in
Innovative food. The medtech project applications were in archives, and extensive resources would have had to be devoted. For this report, the aim was to analyze the more systemic effects, and the task already complex. Therefore, we chose to focus upon financed programs and projects. Moreover, the major research groups and companies were financed.
The second issue is whether and how the Innovative food case corresponds to our interpretation of an analysis of chain-linked effects. It is worthwhile to point out difficulties that may limit the effect analysis in relation to these programs, as compared to VINNOVA’s definitions found in Chapter 1. Many of the projects are still fairly recent in time with some ending as late as in 2008 and where final reports have not yet been made. Provided this time-period of the programs it may still be too early to see some effects such as, for example, patents and products. In VINNOVA’s own terminology therefore, the analysis becomes more one of effect evaluation than one of effect analysis (VA 2007:14). The positive side to the recent ending of the programs is that people interviewed tend to have a good remembrance of the projects.
Moreover, the two programs are fairly small scale in the sense that there are relatively many projects, but with relatively small amounts of funding provided each project. This holds especially for the first program, whereas the second program also funded some larger projects. However, this holds reasonably true for the Medical technology area as well.
A related issue is the additional financers of research, which is explicitly addressed in Chapter 4. As a result of the study reported here, we have discovered that most projects do not start nor end with the funding granted them within these two VINNOVA programs. Many projects have a prior history within earlier programs funded by VINNOVA or other funders, and also proceed after the formal closure of the specific projects of interest here. Each project that VINNOVA finances, then, is often part of a research group’s larger research theme, where the funded project interacts more or less with other projects. This follows quite naturally because, as one interviewee put it, “you rarely apply for money for something which you don’t know at all or have been into before”.
Many projects therefore build on prior projects, and the constellations of many projects often go back a long time. The resulting consequence is that it sometimes becomes difficult to distinguish the effects of the funded project from those of the larger researcher theme. Conversely, “negative” results in a specific funded VINNOVA-project may not seem to be an important result in isolation, but may still have important effects for the larger research theme.
With these limitations in mind, this report has sought to analyze the effects of the two food-related VINNOVA programs so far by utilizing primarily two different sources; the evaluation assessments in projects’ final reports and interviews. For projects initiated 2000 or later, the final reports include assessments of the results made by the project leader according to result categories specified by VINNOVA. In many instances the project officer at VINNOVA has supplemented the assessments with comments on the projects results. The pre-specified result categories in these assessments are as follows: Scientific publications, Scientific conferences, Other
publications, Master theses, PhD positions and dissertations, New research network, New company network, New research group or program, New scientific method or theory, New practical method, Technology transfer, Prototype, Product development, Products, Patents, New technology or equipment, Seminars or demonstrations for practitioners, New firm or commercialization, and New center. Since some of the categories are partly overlapping, this report has used a condensed version of these categories to structure and analyze the results. For projects that lack these assessments, it is still possible to discern results from the final reports.
The third issue is whether one can assume that the public policy initiatives led to – or caused – the effects upon research and industry. The broad approach was to examine the developments, and place them in relation to the public policy initiatives studied here. Thereby, we also recognize and address that there are many external variables affecting these developments, and that those variables are outside the control of policy agencies. We chose to try to examine some dimensions (specifically, publications and patents) through statistical analysis, to at least show relationships.
This discussion of causality is related to the second issue, that many other sources of financing were available to the research groups. Although one can think of short-term project evaluations as a one-to-one causality from public policy initiative to output results, the multiple sources of funding need to be considered in discussion of long-term effects.
Therefore, we have chosen to develop an analytical framework and research design which enable us to point out how these complex relationships may be chain-linked together.