Measurement problems

315. Several innovation surveys have collected data on both the R&D and the non-R&D part of total innovation expenditure. It turned out that many firms had difficulty in reporting innovation expenditure. The non-R&D items, in particular, are not usually directly available from their accounting systems. The foremost problem, accordingly, is not “which data to collect” but “how to collect reliable data” on innovation expenditure other than R&D expenditure.

316. Questionnaire design is crucial for the quality of the data collected on innovation expenditure. Small changes in the definitions or explanations given in the part of the questionnaire dealing with TPP innovation expenditure, and changes in layout or in the sequence of questions or items, will all affect the information gathered.

317. To evaluate the reliability of answers, it may be useful to ask firms to indicate the degree of uncertainty by saying whether their figures are based on detailed accounts or are fairly accurate or rough estimates. Although this kind of question may well raise the share of participants who give rough estimates only, the response rate may be higher.

2.4.1 The borderline between R&D and non-R&D innovation expenditure

318. In recent innovation surveys some firms had problems in differentiating between total innovation expenditure and R&D expenditure, especially at the borderline between R&D and non-R&D expenditure. Recent experience has shown that R&D expenditure measured as a share of total innovation

expenditure and R&D expenditure measured in a separate question (a Frascati-type question) do not match, even when both types of questions are asked in the same questionnaire. This reflects different methods of R&D accounting inside firms; sometimes they do not fully conform to the Frascati definition of R&D expenditure, and include some non-R&D activities. Detailed explanations, and the questionnaire layout, will help enterprises give consistent answers on R&D. This problem is especially acute for industries whose innovation consists to a large extent of design activities (e.g. automobile manufacturing). 319. Care must be taken to exclude activities which are part of the innovation process but rarely involve any R&D (e.g. patent work, licensing, market research, manufacturing start-up, process re-engineering, tooling up). At the same time, some activities are at least partly counted as R&D (e.g. pilot plants, prototypes, industrial design, process development).

320. The basic criterion for distinguishing R&D activities from non-R&D innovation activities “is the presence in R&D of an appreciable element of novelty and the resolution of scientific and/or technological uncertainty” (see Frascati Manual, para. 79). This criterion implies “that a particular project may be R&D if undertaken for one reason, but if carried out for another reason, will not be considered R&D” (Frascati Manual, para. 80).

321. The Frascati Manual (para. 112) suggests using the rule originally laid down by the US National Science Foundation as a rough guideline for distinguishing between R&D and non-R&D activities:

“If the primary objective is to make further technical improvements on the product or process, then the work comes within the definition of R&D. If, on the other hand, the product, process or approach is substantially set and the primary objective is to develop markets, to do pre- production planning, or to get a production or control system working smoothly, then the work is no longer R&D.”

322. It is recommended that the guidelines in the Frascati Manual, paragraphs 111-132, should be applied to innovation surveys. In a number of cases individual firms, especially in some industries, will continue to have problems in allocating some of their innovation activities to R&D and others to non- R&D.

2.4.2 Other difficulties

323. A breakdown of each of the above type-of-activity categories into intramural and extramural expenditure would supply desirable information. But this is not feasible for most enterprises, and so is not recommended here.

324. As a consequence, special care has to be taken when aggregating individual firm numbers to industry or national figures, because of double-counting. To make rough estimates of the amount of double-counting, it seems useful to know whether or not expenditure for external services is included or not.

325. TPP innovation expenditure in a given year can sometimes be misleading. Small firms in particular do not perform TPP innovation activities all the time. Collecting TPP innovation expenditure for a multi-year period thus provides useful additional information on TPP innovation activities. But restricted availability of data within firms is a serious obstacle to the multi-year approach.

326. One way of dealing with this would be to ask, in addition, whether innovation expenditure in previous years was well above, or well below, the reported amount for the year in question. Furthermore,

to facilitate estimates of the most recent trends in innovation expenditure it seems worthwhile to ask whether innovation expenditure is scheduled to grow, fall or stay at the same level. This seems to be especially desirable from an innovation policy viewpoint.