Page 1
PLANNING AND INFORMATION OFFICE
Office of the Provost and DVC
Key Performance Indicators (KPIs)
The Strategic Information and Business Intelligence (SIBI) reporting tool is an ideal mechanism through
which to publish performance metrics for the University.
Performance reporting is an evolving space. The current KPI dashboard (May 2014) is the first formal
publication of KPIs through SIBI and represents the initial step in establishing an agreed set of institutional
metrics; it is not a definitive or final list. The existing suite of KPIs is expected to evolve as University
leaders develop a shared understanding around which metrics are key to measuring performance and as
more data becomes available for inclusion.
The current module contains a diverse set of KPIs and aims to encourage discussion to guide and inform
future KPI development. This incremental approach is preferable as it acknowledges that KPIs are flexible
and will change in response to institutional requirements and reporting maturity. Feedback on current
performance metrics, or suggestions for potential metrics, can be submitted to the Provost and SIBI
Program Board by emailing
pio.sibi@sydney.edu.au
.
To date the focus has been on institutional/faculty KPI development. Indicators for PSUs will be developed
at a later stage.
Background documents
Appendix 1: KPIs – current status and candidate metrics
During the initial KPI development phase SEG considered a draft list of potential KPIs for measuring
institutional, faculty and school performance across a range of areas aligned to the goals of the University’s
Strategic Plan 2011-2015. Some candidate KPIs were non-controversial, others were identified as requiring
further investigation and a third group lacked a robust data source. Some were abandoned following the
SEG review.
Appendix 1 outlines the current status of these candidate KPIs. The indicators already published have been
approved by both the Provost and the Head of the relevant portfolio.
Appendix 1 outlines:
the initial set of KPIs already published through SIBI
KPIs under development for release soon via SIBI
indicators broadly supported by SEG that are candidates for future SIBI development
indicators which require further work to determine the optimal methodology
indicators which may have value but need scoping and costing for data acquisition.
Appendix 2: The KPI framework
The underlying framework was endorsed by SEG in February 2013. Since that time, the development and
release of metrics has shifted to an incremental or phased approach.
Director Planning and Information May 2014
Appendix 1
Planning and Information Office Page 2
1.1 Initial set of KPIs available in SIBI
Area Indicator Metric Drill to…
Research
Research income Total HERDC research income by year
($m) Faculty
Research outputs Total HERDC weighted research
publication points Faculty
Student Diversity
Indigenous Access, Participation, Attainment, Success Faculty, UG / PG(Cw) / PG(Res)
Low SES Access, Participation, Attainment, Success Faculty, UG / PG(Cw) / PG(Res)
Gender % split Female / Male, Access, Participation, Attainment, Success
Faculty, UG / PG(Cw) / PG(Res)
Disability Access, Participation, Attainment, Success Faculty, UG / PG(Cw) / PG(Res) Regional / Remote Access, Participation, Attainment, Success Faculty, UG / PG(Cw) /
PG(Res)
Student Outcomes UG/PG Success UG/PG(Cw) success rates Faculty, F-1, F-2, Dom/Int
PG research completions HDR completions Faculty, Dom/Int
Financial Sustainability
Course enrolments Variance to budget Faculty, Dom/Int, UG/PG(Cw)
Load Variance to budget Faculty, F-1, F-2, Dom/Int,
UG/PG(Cw)
Student revenue Variance to budget Faculty, F-1, F-2, Dom/Int, UG/PG(Cw)
Internal cohort mix
International students (%) Faculty
Domestic students (%) Faculty
Undergraduate students (%) Faculty Postgraduate coursework students (%) Faculty Postgraduate research students (%) Faculty
Brand Overall brand strength Net Promoter score Faculty
Staff Satisfaction
Staff engagement The index value for Passion / Engagement as calculated by Voice Project.
Progress The index value for Progress as calculated by Voice Project.
Wellbeing An index value for Peace as calculated by Voice Project.
1.2 KPIs under development for release soon via SIBI
Area Indicator Metric Drill to…
Student Outcomes UG/PG Retention UG/PG(Cw) retention rates Faculty, Dom/Int
Student Experience *
Overall course satisfaction
(SCEQ) Agreement and broad agreement (%) Faculty, Dom/Int, PG/UG Good teaching (SCEQ) Agreement and broad agreement (%) Faculty, Dom/Int, PG/UG Unit of study satisfaction UoS with ≥ 80% satisfaction (% of total) PG/UG
Unit of study satisfaction UoS with ≥ 20% dissatisfaction (% of total) PG/UG HDR: Overall satisfaction
(SREQ) Agreement and broad agreement (%) Faculty, Dom/Int, PhD/M HDR: Supervision (SREQ) Agreement and broad agreement (%) Faculty, Dom/Int, PhD/M HDR: Generic Skills (SREQ) Agreement and broad agreement (%) Faculty, Dom/Int, PhD/M HDR: Student enrolments in
UoS Number of UoS enrolments Faculty, Dom/Int, PhD/M
Development Fundraising achievement Funds raised (% growth and dollars) Faculty, F-1
Donors (% growth and number) Faculty, F-1, donor type
Alumni Under discussion Faculty
Appendix 1
Planning and Information Office Page 3
1.3 KPIs with broad support from SEG and candidates for inclusion in SIBI development.
Yet to be finalised.
Area Indicator Metric Drill to…
Curriculum eLearning UoS with eLearning component (%) Faculty, F-1
Staff Diversity
Indigenous staff Indigenous staff (%) Faculty, F-1
Gender profile Gender indicator (by type and level) Faculty, F-1 Age profile Age indicator (by type and level) Faculty, F-1 Research student
satisfaction Research supervision PREQ quality of supervision Faculty, F-1
Research outcomes
ERA ranking Disciplines rated world class of above (%) FoR
Competitive research income
Category 1: Aust Competitive Grants ($m) Faculty, F-1 Category 2: Other public sector ($m) Faculty, F-1 Category 3: Industry and other ($m) Faculty, F-1
Category 4: CRC ($m) Faculty, F-1
Outputs
Books weighted (A1) Faculty, F-1
Books chapters (B1) Faculty, F-1
Journal articles (C1) Faculty, F-1
Conference papers (E1) Faculty, F-1
Creative works Faculty, F-1
Cross disciplinarity Publications with more than one ERA code Faculty
Community and International Engagement
Linkage grants Number of application partners Faculty, F-1
ARC Funding ($m) Faculty, F-1
Contract research income
Category 3: Australian contract income ($m) Faculty, F-1 Category 3: Other international income ($m) Faculty, F-1 Consultancy income ($m) Faculty, F-1
Contract income ($m) Faculty, F-1
Sustainability
Cost of administrative services Revenue allocated to admin services (%) Faculty, F-1 Directly controllable margin Variance to budget (%) Faculty, F-1 Change on previous year (%) Faculty, F-1
Operating margin Variance to budget (%) Faculty, F-1
Appendix 1
Planning and Information Office Page 4
1.4 KPIs that potentially provide performance insight but need further investigation, definition
and/or methodology review. Some can be implemented relatively easily.
Area Indicator Metric Comments
Risk management
WHS Serious incidents/near misses (per 100 FTE)
These have been proposed by Audit and Risk Management Office. These risk indicators need further discussion. They are likely to form a separate dashboard for restricted FIC and FAC access.
Lost time due to injury
Teaching Student/staff ratios
ATAR cut offs
Source country diversity Students from major market (%)
Research Research grants (movement)
Ethics and/or misconduct breaches Finance Load outcomes against targets
Salaries (% of revenue)
HR Staff turnover rates
Business continuity
UoS/courses available in other than face to face mode (in the event of a major shutdown or pandemic)
Demand
Quality
ATAR data – cut offs (prior to bonus schemes), median, range
Demand measures need further consideration. Some data needs to be defined and sourced. Gaukau scores, and other IFee measures
Volume Change in eligible applications (%) UG and PG Market share Market share (%) UG and PG
Conversion rates
Application to offer to enrolment conversion – UG/PG, Domestic/International,
research/coursework. By course. Curriculum
Curriculum review Percentage internal/external review Measure TBC Quality Assurance Performance + QA
Interdisciplinary courses Measure TBC
Graduate learning
outcomes Employer satisfaction Net Promoter Score (Employer)
Data available late 2013 from Brand Tracking Research student
outcomes PG research completions Rate (%) [methodology tbc] Yes
Research
Commercialisation Contracts providing commercial return (%) Measure TBC
Quality of publications
Citation rates
For further investigation Impact factor Nature/Science pubs? Community and International Engagement CELT
Students reporting CELT learning
experiences (SCEQ) (%) Measures TBC
High quality community relationships (#) Co-authored publications with
overseas academics
Rate of co-authored publications (%)
HERDC Measure TBC
Students as global citizens Measure TBC
Sustainability
Academic staff promotions Number, rate, success Yes Productivity Staff achieving objectives in alignment with SP through APD process (%) Environmental
sustainability Carbon footprint (USM) USM metric to be defined
PIO development required
Appendix 1
Planning and Information Office Page 5
1.5 KPIs that are potentially valuable, but need scoping for data acquisition and associated budget
estimate.
All require significant investigation
Area Indicator Metric Comments
Graduate outcomes
Quality outcomes More discussion required
All need further investigation Student satisfaction Graduate survey 1, 3, 5 years
Employer satisfaction Employer survey
UoS with appropriate QA of assessment (%) Research student
outcomes Graduate destinations (5 yr out) TBC
Community and International Engagement
Non research consultancy and contract
Number of non-research consultancies and contracts
Positions on external committees
rate per FTE (or % staff). External interests policy as start point?
Honours and awards rate per FTE (or % staff). Use ERA definitions? International course
accreditation Internationally accredited courses (#)
Staff
Support for staff APD completions, academic (%) GPD completions, professional (%) Honorary/Visiting academics
Doctoral qualifications Staff with doctoral qualification (%) (??) Other qualifications Staff engaged in continuing professional dev (%)
PLANNING AND INFORMATION OFFICE
Planning and Information Office Office of the Provost and DVC Level 6, Quadrangle T +61 2 9036 6490 M +61 434 568 744 F +61 2 9351 7301 E sandra.harrison@sydney.edu.au sydney.edu.au ABN 15 211 513 464 CRICOS 00026A
SUBMISSION TO SEG
28 February 2013
INSTITUTIONAL KEY PERFORMANCE INDICATORS (KPIs)
Authorisation: Stephen Garton (Chair SIBI Board)
2013/02
SEG members will be aware of the current business intelligence project under development, the Strategic
Information and Business Intelligence Program (SIBI) – a collaborative project between Planning and
Information Office and ICT. See:
http://sydney.edu.au/staff/planning/sibi/index.php
One of the important deliverables of SIBI is to establish coherent reporting against agreed institutional and
faculty objectives through desktop, ‘one touch’ enquiry.
We are now at the stage in SIBI development where discussion of potential institutional Key Performance
Indicators (KPIs) will help drive SIBI progress, focus business readiness requirements and prioritise
technical development.
In order to stimulate this discussion a proposed KPI Framework for the University is now submitted to SEG
for review. This draft Framework was reviewed and endorsed by the SIBI Board, chaired by the Provost, in
November 2012 and includes feedback from that discussion as well as feedback from other stakeholder
groups within the University.
The KPI framework will be an evolving model, that will necessarily be adapted over time to reflect changing
University priorities. At this point, SEG members are asked to review this draft and provide feedback on the
proposed set of KPIs as measures of institutional and faculty performance that reflect the University’s
current reporting requirements, noting that these KPIs can (and will likely) be amended over time in
response to internal and external drivers. For example, TEQSA Standards (expected at the end of 2013)
may necessitate change to the set of measures for learning and teaching outcomes (eg peer review of our
graduates). The framework needs to be sufficiently flexible to be able to respond to such changes in order
to guide data gathering requirements and future SIBI technical developments.
To date the focus has been on institutional/faculty KPI development – KPIs for PSUs will follow. There are
still a number of areas in this framework that need to be finalised, therefore SEG input at this stage of
development would help provide direction. For those measures that are relatively straightforward the SIBI
team can commence development of reports while discussion continues around the less straightforward
measures.
Recommendation
That SEG review the proposed set of KPIs as the next step in ongoing delivery of performance reporting
through SIBI, and provide feedback for finalisation of the overall KPI Framework.
S Harrison
Director Planning and Information 15 February 2013
Planning and Information Office
Office of the Provost and DVC
Planning and Information Office Office of the Provost and DVC
T +61 2 9036 6490
M +61 434 568 744
ABN 15 211 513 464 CRICOS 00026A
Reporting Performance: An Institutional KPI Framework
Consultation Papers
The establishment of a framework for monitoring institutional performance is a companion development to SIBI progress and implementation. We are now at the stage where discussion of a draft set of Key Performance Indicators (KPIs) will help drive SIBI progress, focus business readiness requirements and prioritise technical development. The breadth of performance reporting needs to be relevant and manageable. At the same time, however, we need to maximise the SIBI return on investment – so that some extended data sets will be included in the SIBI reporting capabilities.
Developing the Key Performance Indicator Framework
There are multiple sources and drivers for KPIs, providing a list of hundreds of candidate indicators. The sources include:
The University’s Strategic Plan (2011-2015)
The current suite of institutional KPIs (approved by Senate in 2010) Workslate projects
The University’s Risk Framework
The original 27 faculty KPIs (implemented in 2007) The previous Balanced Score Card project
Indicators developed for each PSU/Portfolio (HR, Marketing, DVC(E) for instance) TEQSA regulatory requirements
Individual faculty requirements
The draft KPI Framework (Attachments 1 and 2) has been prepared following initial consultation with relevant portfolios and PSUs and following criteria agreed by the SIBI Board earlier in 2012 (Attachment 4). The detailed draft list of KPIs and measures (Attachment 3) is seen as a starting point for broad based discussion and consultation.
Rationale
The final institutional KPI framework must include indicators that are evidence-based measures of the University’s performance, in the context of the Strategic Plan, not a series of measures that simply count outputs. It is also essential that the framework incorporates measures over which the University has performance control, not those where outcomes are substantially governed by external environments, and that they truly add value to monitoring institutiona/faculty performance and to informing decision making.
The aim is to provide some level of flexibility, so that Indicators can be shifted between Level 1 and Level 2 in response to changing University priorities.
Availability of data
KPIs are proposed in this Framework on the basis of meeting the University’s needs, rather than on the current availability of data. Once the indicators are agreed as an institution, we will assess the requirements for data capture, review the feasibility and cost of such data capture and make recommendations on if/how to proceed on a priority basis. As an institution we are at a stage of maturity where we need to move from simply using ‘what is available’ to determining ‘what is required’.
Targets and benchmarking
Once the framework and indicators are agreed and the relevant SIBI technical development is complete, the key next step is to develop targets for most indicators (there could be some indicators where setting targets is not sensible). This is in itself quite a major undertaking and likely best approached once there is a level of institutional maturity in
understanding the data and in conjunction with external benchmarking. S Harrison
15 February 2013
Planning and Information Office
Office of the Provost and DVC
Attachment 1
S Harrison, December 2012 Attachment 1, Page 1
Reporting Performance: An Institutional KPI Framework
Three levels of KPIs are proposed.
CORE KPIs
(at two levels) track performance at each of University, faculty and school levels in the following categories: Brand Mutual accountability Engaged enquiry – Education – Research– Community and international engagement – Alumni and development
Sustainability – Human resources – Financial
– Environmental
Risk factors
LEVEL 1 Core KPIs provide aggregated data as a snapshot of University, faculty and school performance.
LEVEL 2 Core KPIs provide granular and complementary information to supplement Level 1 KPIs, also at University, faculty and school levels.
Indicators can be shifted between Level 1 and Level 2 in response to changing University priorities. Both Level 1 and Level 2 KPIs measure University, faculty and school performance. The distinguishing feature between the two is the level of information granularity (for instance student cohort or research grant type).
ENABLING KPIs
reflect the strategic goals of each of the PSUs, and track the performance of each unit in achieving those goals and supporting the University’s core business.At a glance…
Indicator
Type
Characteristic Applies to
Aggregated Granular University Faculty School PSU
Core (Level 1)
Core (Level 2)
Enabling
Targets and Benchmarking
Where it is sensible to do so, targets will be considered for each of the KPIs – once the indicators are agreed across this institution. Internal benchmarking and Go8 benchmarking from a discipline perspective will be included where data are readily available.
S Harrison, December 2012 Attachment 1, Page 2
KPI characteristics
CO
RE
KPI
s
Le
vel
1
Core (Level 1) indicators reflect faculty performance, at an aggregated level, and the contribution to institutional performance.
These KPIs measure performance across our core business of Learning &Teaching and Research, and include some sustainability measures (eg those aspects finance and HR for which deans are responsible).
There is drill down from Faculty to School and, conversely, data can be aggregated to University level.
On the whole, they each meet the candidate KPI criteria (see Attachment 3). These KPIs will be the most visible on SIBI.
Leve
l 2
Core (Level 2) indicators are a more granular measure of faculty performance, complement the aggregated KPIs of Level 1 and, again, show the contribution to overall institutional
performance.
These indicators also measure performance across our core business of Learning &Teaching and Research, but at a more granular level.
They include some sustainability measures (eg those aspects of finance and HR for which deans are responsible).
There is drill down from Faculty to School and, conversely, data can be aggregated to University level.
There is also drill down to more granular cohorts or categories. For example: Level 1 indicator Level 2 indicator
Total research income Category 1 income
Category 2 income
and so on
Variance to commencing enrolment target (%) (all students)
Variance to commencing target (%) (CSP)
Variance to commencing target (%) (Domestic Fee) Variance to commencing target (%) (International)
and so on
Includes any other KPIs that we consider valuable, but not sufficiently critical to be in Level 1. Includes any faculty specific KPIs.
There is flexibility between Level 1 and Level 2 – KPIs can be promoted/demoted between the two levels, depending on the University’s changing strategic priorities.
EN
AB
LING
KPIs
Enabling KPIs reflect the performance of Professional Support Units (PSUs) in supporting
achievement of institutional objectives embedded in Core KPIs
These indicators are likely to be developed later in 2013 (although some PSUs are well on the way to developing these). SIBI will concentrate on reporting against Core KPIs as priority.
HR, Finance and Development/Alumni will have Enabling KPIs that measure their own professional performance (for example, metrics around recruitment performance, or central financial
management). However there will also be KPIs related to these areas that measure faculty performance (included in Core Levels 1 and 2) (for instance revenue measures and staffing measures).
ICT, CIS, Library will have only have Enabling KPIs, as there is no delivery on this by faculties. Some work was commenced on similar KPIs a few years ago, and may still be relevant.
PLANNING AND INFORMATION OFFICE
2
Appendix 2
Attachment 4
Planning and Information Office Office of the Provost and DVC
T +61 2 9036 6490
M +61 434 568 744
ABN 15 211 513 464 CRICOS 00026A
Institutional KPI Framework: Criteria for indicator selection
Planning and Information Office
swFinal September 2012
1 Relevance Each KPI needs to be aligned with the University’s Strategic Plan (SP), so that it is
transparently relevant to institutional goals (explicit and implicit). Explicit goals derive directly from the SP. Implicit goals include those embedded in Work Slate projects, the Risk Framework, TEQSA and other legislative requirements. Key to satisfying this criterion is the need to ensure the KPI is measuring strategic performance rather than operational goals.
2 Depth KPIs need to drill down through the organisation so that divisions/faculties/schools can view how they contribute to the whole of institution performance.
3 Driver to impact real change
3.1 Controllable
The University needs to be able to control the performance being measured and be able to impact performance, ie the KPI measures outcomes that are managed or controllable by the University and are not principally determined by outside factors.
3.2 Targets and benchmarking
The target is agreed by all stakeholders and external benchmarking (sector or international) is used to guide target setting.
3.3 Timely
Where possible the KPI is a lead indicator (although true lead indicators are rare) and performance measurement is possible in a time frame that will allow changes in process to impact results.
4 Supports legislative requirements
The extent to which the KPI supports TEQSA review/audit requirements and/or legislative requirements.
5 Accountable Clear accountabilities for each performance KPI outcome can be established – one owner clearly identified for each.
6 Clarity The KPI does not require specialised knowledge and makes sense without detailed explanation.
