Performance Metrics

Before examining the performance of research commercialisation through our current TTO-based structural arrangements, it is important to define how success in research commercialisation is understood from the perspective of various stakeholder groups. Defining success in research commercialisation is a function of defining what outcomes are desired, then tracking and measuring performance in light of those desired outcomes.

From a government perspective, successful research commercialisation often means new jobs, growth in GDP and improved productivity in the economy. For example, a

report published by the Prime Minister’s Science, Engineering and Innovation Council in 2001 illustrates this view:

Commercialisation of public sector research is on the rise in Australia. . . If we can grow 200–250 more Australian research-based companies like five of those shown in this report over the next 5 years, the prize would be around $20 billion added to our annual export earnings. Australia would be well on the way to reducing the national debt and the cost of servicing it (Independent working groups for PMSEIC, 2001, p. 3).

From an industry point of view, research commercialisation is about transforming IP that is generated from public sector research into something marketable. It is only after a product or service is launched, revenue is received and cash flow becomes positive, that a commercial success is achieved. The consensus within the Australian biotechnology industry in regard to commercialisation performance is that there is a lack of local success stories (Vitale, 2004). For example, this view was expressed by an interviewee:

. . . from a therapeutic point of view there is no shining star, there is no Genetech or CAT or Biogen – no such company exists in Australia, no-one has a product that is on the market that we can point to and say well this was the company. I guess we do have Biota, but Relenza is not considered a success. When you have one company that manages to somehow stay in Australia, that is a success story.

Further evidence of industry’s perspective on research commercialisation performance comes from a study conducted by Hine and Vieira (Vieira & Hine, 2005). Based on a quantitative analysis of 85 listed Australian biotech companies, this study shows that losses are common in the industry (Figure 6.1). In 2004, only 13% of the companies researched were profitable, while 87% of the companies had negative revenues. Out of

>-10 -5 t o -10 -1 t o -3 0 to - 3 0 to - 1 0 to + 1 +1 t o +3 +3 t o +5 +5 t o +10 >+10 0 5 10 15 Negative revenue Positive revenue Revenue ($ million) N u m b e r o f b io te ch c o m p a n ie s

Figure 6.1 Profitability of publicly listed biotechnology companies in 2004. Source: data for the plot are obtained from Vieira and Hine (2005).

the companies which were not profitable, 45% of the companies lost between $5 and $10 million, and 14% lost more than $10 million.

From an academic perspective, most universities and public research organisations are ambiguous about the way they perceive research commercialisation success. On the one hand, they want to maximise financial returns by exploiting research results in the form of IP sold to industry; while on the other hand, they want to maximise the amount of technology that is transferred to the private sector. Some research institutions have difficulty in deciding the outcomes they wish to pursue in research commercialisation. For example, a university professor described in the interview how his university kept re-inventing itself:

They have a hard time making up their mind. It changes as a function I guess — the views of senior administration, so when I first came there wasn’t much emphasis put on it so — early to mid 1990’s. Then xxx was kind of re-invented to be a much more formal commercialisation arm in the University. And then it was dis-established, three years ago probably.

It is important to note that there are at least three different perspectives of research commercialisation success: government, industry and academia. Performance is often assessed based on the objective one has in mind. Indicators developed for measuring one objective usually ignore the other. Similarly, indicators applying to one stakeholder group are unlikely to be relevant to others. Ambiguities and mismatch between goal formulation and output measures may create difficulties in evaluating performance.

Current standard performance measures for research commercialisation, also known as metrics, are shaped by the prevailing IP-based view discussed in chapter 5. The metrics are developed to measure tangible outputs being achieved by research institutions through identifying, packaging, protecting and exploiting IP. These narrowly-defined metrics are neither designed to evaluate non-IP based transfer, nor are they intended to assess commercialisation success from the perspectives of industry and government.

The performance metrics which are used in Australia are heavily influenced by foreign models. The US Association of University Technology Managers (AUTM) Licensing Survey methodology is the most widely accepted standard for evaluating research commercialisation performance. The survey provides a yearly profile of US universities, hospitals, and research institutes and Canadian institutes. It focuses on indicators related to the generation of tangible commercial outputs. Typical indicators include:

• Number of full-time equivalent staff employed in TTOs

• Research expenditures per institutions

• Legal expenditure and reimbursement

• Patent related and start-up activities

• Licences, options, and assignments (LOAs)

The AUTM has collected annual survey data since 1991. Its survey methodology has become the basis for the Australian equivalent, the National Survey of Research Commercialisation, which has been conducted periodically by the ARC, the NHMRC and DEST since 2000.

The National Survey of Research Commercialisation is expensive to conduct, report, and respond to. The 2001 and 2002 survey spent $400,000 to cover the costs in designing and conducting the survey, compiling and analysing the data and publishing the results (CCST, 2005). Due to the high cost and the amount of time required to conduct the survey, there is usually a considerable delay before the results are published. For example, the National Survey of Research Commercialisation 2005– 2007 is the latest survey available in Australia, and it is already more than two years behind (DIISR, 2009).

There was a great deal of concern amongst the respondents of this study that defining research commercialisation success based on the current performance metrics is limited. Some technology transfer managers interviewed mentioned that the National Survey of Research Commercialisation only tracked narrow IP-based outputs, and therefore did not capture the net economic and social return accruing to the wider economy. Some argued that research commercialisation needs to be measured at an aggregate level. In order to evaluate the broader impacts of research commercialisation on society, new performance indictors have to be developed which require clearly defined objectives.