Best practice in implementation Many institutions either had no IT strategy or

one that was developed and guarded by the IT department. It was often suggested that re- search management systems were perceived to be difficult (both to specify and implement) and that as a result research was given a low priority for investment. In comparison systems to support finance, human resources and students, almost always secured investment easily. Explanations for this included the constantly changing external landscape but also questions of ownership. Responsibility for research systems is less focussed within institutions; a range of stakeholders feel the need to be involved whereas in other areas of the administration there are always clear systems owners. As a result most institutions felt there had been low levels of investment in research management systems.

It is apparent that not all institutions have clear and transparent frameworks for decid- ing infrastructure investments. In some cases this has led to proposals being discussed in many different committees (just to gain momentum), almost invariably without a coherent business case.

Many research system projects were judged by interviewees as unrealistic in terms of scope, timescale, budget and resources, reflecting again the need for properly developed project initiation documents and project controls. It was often suggested that stakeholders wanted to specify perfect systems from the outset rather than focussing core functionality first. This often led to daunting, poorly defined system projects which subsequently encountered difficulties with functionality and data cleansing and migration. Few institutions had attempted to segment their project delivery in to phases to make projects more manageable. Similarly, few described how they had managed to anticipate data quality issues and set aside adequate resources to tackle this up front.

Most felt that IT projects tended to be over-reg- ulated and that they were driven predominantly by IT departments. In many cases this inhibited wider involvement throughout projects, to the perceived detriment of project outcomes. Academic sponsorship and involvement was deemed to be essential to the success of re- search systems projects and, in particular, the involvement of academics early in the require- ments gathering process was deemed critical. A key ingredient of successful implementations was uniformly felt to be strong leadership at the very top of the institution – leaders with credibility with the academic community and sufficient seniority and empowerment to de- fend the project from interference by different stakeholders en route. In tandem with strong leadership and focus was the need for clear and authoritative decision-making. Too often, because of the consultative operating style within many institutions, decisions were taken with systems projects in an effort to please all stakeholders. Success was strongly linked to those institutions with project leaders capable of making tough decisions.

30

Do Not

Focus purely on functionality; the user experience of the system is also important Let the needs of a small group of academics drive the project

Address only corporate needs; also deliver visible benefits to academics

Spend too much time on requirements – allows less time to build the system

It was often the case that, when considering new research systems, institutions talked informally to those in the sector with whom they had links or those who they understood to have been involved in similar develop- ments. However, there was no rigorous or formal process to harness experiences within the sector or to build a bank of experience (with the unsurprising consequence of lack of commonality of systems and implementation methods). It is disappointing that many of the issues identified in this study reflect very similar issues identified in the MAC initiative underlining that the sector appears slow and unwilling to learn from its past experiences. While there are understandable reasons for the lack of shared experience across the sector, it was alarming that many institutions admitted that they seldom learned from their own implementation experiences. Invariably research system projects included a lessons learned element in the project process, but it rarely carried weight within institutions. Ironi- cally, it is perhaps these experiences from which institutions could learn most.

This study has identified many areas where there is duplication of effort, inefficiency in process, system implementation, and waste of resources – and found little evidence of lessons being learned. It would be salutary to carry out a study to quantify the costs to the sector overall, but that in itself may be a poor use of valuable resource given the strength of evidence.

Academic and user engagement Do

Include a range of stakeholder representatives on the project board

Involve users in early-stage design and requirements gathering

Break down barriers with users through advisory groups and inclusion in testing Secure academic sponsorship of the project

31 1. Institutions within the sector should work

more collaboratively with each other to harmo- nise their approach to processes and thereby minimise wasteful duplication of investment in systems across the sector. Consideration should be given to the establishment of a network or body charged with facilitating institutional links and mapping core processes in detail.

2. Institutions and funders should work more collaboratively to identify commonality in systems and processes so that they may share data in more cost effective and less resource intensive ways. Consideration should be given to the establishment of a framework or network to achieve this, with clearly identified terms and deliverables, and with engagement at decision-making level within stakeholder organisations.

3. Institutions should develop stronger relation- ships with suppliers and work with them to define needs more clearly. Significant progress could be made by exchanging expertise between those involved in actually managing research and suppliers whose expertise is in translating business processes into effective tools. This may require institutions to invest time and resources in understanding the holistic research management environment. 4. A national framework for data standards – encompassing both data and data defini- tions – should be developed across the sector, thereby enabling institutions to specify generic systems for reporting metrics, aligned to the tools for managing their individual institution. 5. Suppliers should look to participate in the development of data standards with the sector in an effort to drive consistency in research systems. Consideration should be given to the benefits this would bring to the sector through easier integration of systems and efficiencies in data exchange.

6. Institutions, supported by funding organi- sations, should be encouraged to develop long-term system strategies focussed upon core research management processes and in- formation needs. The vision for future systems should include mechanisms for addressing the instability of research databases, incompat- ibility of data between systems and concerns regarding data quality.

7. Notwithstanding the differences that exist in the appetite for performance management within the sector, each institution should establish a clear framework to address: • Responsibilities within an institution for

reviewing and acting upon information • The level at which performance manage-

ment data is to be carried out, but with access to underpinning detail to ensure confidence in data exists

• Succinct and consistent performance measures which avoid information overload and confusion

8. Work should be undertaken jointly by funders and the sector to harmonise external benchmarking data and to ensure it is acces- sible to institutions in a timely and consistent manner. The benefits for doing so are mutu- ally beneficial: institutions better understand their strengths and weakness, funders foster efficient competition and research excellence. 9. The sector should develop a culture that moves away from reactive information towards using information to anticipate change. Suppli- ers should be encouraged to equip institutions with user-friendly modelling tools to allow them to do this.

10. Institutions should have transparent methodologies for developing their IT strate- gies, involving all key stakeholders across the business. They should have clear and transparent frameworks for deciding IT infra- structure investments supported by properly

11. Recommendations

32

constructed business cases that articulate both cost and expected benefit. They must develop a more robust and business-like approach to prioritising IT investment decisions.

11. The composition of project teams to manage the development and implementation of research management systems must be balanced to include representatives of the business (including academics and managers) as well as IT staff. Key academic stakeholders must be involved early in the project to scope requirements effectively.

12. Projects should always involve champi- ons who have credibility with the academic community and sufficient gravitas within the institution to provide unchallenged leadership. 13. Resources for data cleansing, migration and conversion must be properly identified in a pro- ject and anticipated from the outset. This was one of the most consistent areas of difficulty in system implementation across the sector. 14. At an individual institutional level, ways in which lessons can be learned from past imple- mentations should be developed to establish a corporate memory.

15. The sector should develop a framework within which they can build a knowledge base of experience across the sector and share lessons learned of specifying, developing and implementing research management systems. This needs supporting and facilitating nationally. 16. A programme for taking these recom- mendations forward should be agreed by all stakeholders (JISC, funders, suppliers, institu- tions) with lead partners responsible for taking forward specific recommendations.

33

12. References

Evidence, UK Higher Education Research Yearbook 2009 (Thomson Reuters, 2009). John Green and David Langley, Professionalising Research Management (2009). Available at www.researchdatatools.com

John Green, Scott Rutherford and Thomas Turner, 'Best practice in using business intelligence in determining research strategy', Perspectives: Policy and Practice in Higher Education, 13:2 (2009)

Janette Hillicks, ‘Development Partnerships between HE and Vendors: Marriage made in Heaven or Recipe for Disaster?’, JISC InfoNet (May 2002). Available at http://www.jiscinfonet. ac.uk/Resources/external-resources/develop- ment-partnerships

34