Looking at apples and oranges. Learning from university rankings and other composite indicators

(1)

1 12 April 2013

Looking at apples and oranges.

Learning from university rankings

and other composite indicators

Andrea Saltelli

Unit Econometrics and Applied Statistics European Commission, Joint Research Centre

JRC – UNIVERSITIES: HOW TO INCREASE THE CONTRIBUTION OF

UNIVERSITIES TO SCIENCE AND INNOVATION

11 APRIL 2013, Berlaymont Building, Rue de la Loi 200, Schuman Room

(2)

Saisana, M., d’Hombres, B.,

Saltelli, A.,

Rickety numbers:

Volatility of university rankings

and policy implications

, 2011,

Research Policy, 40 165–177.

Paruolo, P., Saisana, A., Saltelli,

A., 2013,

Ratings and rankings:

Voodoo or Science?

Journal

Royal Statistical Society A, 176

(2), ---.

(3)

‘Invasive’ sensitivity analysis: shaking the assumptions

shows that rankings are volatile

Legend:

Frequency lower 15%

Frequency between 15 and 30% Frequency between 30 and 50%

Frequency greater than 50%

Note: Frequencies lower than 4% are not shown

1-5 6-1 0 11 -1 5 16 -2 0 21 -2 5 26 -3 0 31 -3 5 36 -4 0 41 -4 5 46 -5 0 51 -5 5 56 -6 0 61 -6 5 66 -7 0 71 -7 5 76 -8 0 81 -8 5 86 -9 0 91 -9 5 96 -1 00 _Original rank

Harvard Univ 100 1USA

Stanford Univ 89 11 2USA

Univ California - Berkeley 97 3USA

Univ Cambridge 90 10 4UK

Massachusetts Inst Tech (MIT) 74 26 5USA

California Inst Tech 27 53 19 6USA

Columbia Univ 23 77 7USA

Princeton Univ 71 9 11 7 8USA

Univ Chicago 51 34 13 9USA

Univ Oxford 99 10UK

Yale Univ 47 53 11USA

Cornell Univ 27 73 12USA

Univ California - Los Angeles 9 84 7 13USA

Univ California - San Diego 41 46 9 14USA

Univ Pennsylvania 6 71 23 15USA

Univ Washington - Seattle 7 71 21 16USA

Univ Wisconsin - Madison 27 70 17USA

Univ California - San Francisco 14 9 14 11 7 10 6 6 18USA

Tokyo Univ 16 16 49 20 19Japan

Johns Hopkins Univ 7 54 21 17 20USA

Simulated rank range - SJTU 2008

Volatility of university ranking, example

ARWU (Shanghai) 2008 data

(4)

Internal coherence of ranking systems

Non-invasive sensitivity analysis. Comparing the

internal coherence of ARWU versus THES by testing

the weights declared by developers with ‘effective’

importance measures suggests that ARWU is more

coherent than THES.

(5)

5 12 April 2013

A cooperation

between JRC and RTD

Research

Excellence

Indicators

(6)

Research Excellence indicators (JRC -RTD)

Code and Name

Definition

Sources

HICIT

(Highly cited

publications)

Field-normalized count of the 10%

most highly cited publications / GDP

Science Metrix

(Scopus)

PCTPAT

(Patent

Cooperation

Treaty

applications)

Patent applications filed under PCT by

inventors’ country of residence

(fractional counting) in all IPC classes /

GDP

OECD

TOPINST

(World class

universities)

Country scores on global top 250

universities (based on the

Leiden

Ranking

) / GDP

CWTS (Web of

Science)

ERC

(ERC grants

received)

Value of ERC grants received by

country of host organization, equally

spread over project duration / GDP

ERC, DG-RTD

CORDIS

(7)

Research Excellence – 41 country “global”

comparison

7 8 9 .8 7 4 .8 7 0 .7 6 0 .2 5 8 .2 5 7 .6 4 5 .4 4 3 .2 3 9 .5 3 7 .1 3 6 .4 3 1 .9 2 9 .2 2 7 .5 2 7 .4 2 7 .1 2 6 .1 2 4 .4 2 0 .2 1 8 .6 1 6 .7 1 4 .5 1 2 .8 1 1 .9 1 1 .6 1 0 .5 1 0 .2 9 .2 8 .5 7 .9 7 .4 7 .3 5 .6 5 .4 5 .3 4 .8 4 .6 3 .8 3 .5 ₃.5 2 .4 0 10 20 30 40 50 60 70 80 90 100 CH SE DK NL IL FI UK BE AT DE NO US IS EU 2 7 IE FR KR JP SI EE IT ES PT GR CY HU CZ CN LU MT HR LT SK TR PL LV BG RO BR IN RU 2008 2005

Composite based on 3 indicators:

• Highly cited Publications

• Top universities

• PCT Patent applications

(8)

Research Excellence - ERA countries

9 2 .4 7 7 .4 6 8 .6 6 8 .2 6 6 .2 6 3 .3 4 7 .4 4 5 .6 4 3 .6 4 1 .6 3 3 .6 3 2 .9 2 9 .1 2 9 .1 2 8 .7 2 5 .3 1 8 .6 1 7 .8 1 7 .1 1 6 .9 1 6 .4 1 4 .6 1 3 .4 1 0 .4 6 .6 6 .4 5 .9 5 .7 5 .5 ₅.4 4 .2 4 .2 3 .6 3 .3 0 10 20 30 40 50 60 70 80 90 100 CH SE IL FI DK NL UK IS BE AT DE _NO EU 2 7 _FR _CY _IE _IT _ES _SI HU EE GR PT CZ BG LU _MT PL HR LT TR SK LV RO 2008 2007

Composite based on 4 indicators:

• Highly cited Publications

• Top universities

• PCT Patent applications

• ERC grants received

(9)

Research Excellence Index

versus one of its variables:

Top Universities

AT BE BG CZCY DE DK EE GRES FI FR HU IE IT LTLU LV MT NL PL PT RO SE SI SK UK EU27 HR TR CH IS NO IL BR RUIN CN JPKR US 0 20 40 60 80 1 0 0 T O PI N ST p G D P-0 8 0 20 40 60 80 100 Index_WLD_08 Countries with no universities in global top 250 T op Un iv e rs it ie s in d ic a tor

Research Excellence Composite

(10)

Using ARWU (Shanghai) ranking, EU versus US, 2008-2012 .

Top one hundred in ranking, per capita, US=100

2008

(11)

11 12 April 2013

Using ARWU (Shanghai) ranking, EU versus US, 2008-2012 .

Top fifty in ranking, per capita, US=100

2008

(12)

Using ARWU (Shanghai) ranking, EU versus US, 2008-2012 .

Top ten in ranking, per capita, US=100

2008

(13)

13 12 April 2013

Top 250

universities/GDP and PCT patents/PCT

Looking at invividual US states and

Chinese region a-la-Aghion.

(Leiden ranking)

(14)

Using of the EUMIDA dataset

•

25 EU member states

•

Census of all HEIs from 25 EU member states (2,400 HEIs)

•

2008

Vertesy, D., Annoni, P., Nardo, M., University systems: beyond league tables. Engines of growth growth or ivory towers?, Proceedings of the IREG 2012 conference, Taiwan.

University

rankings

at the regional scale

(JRC with DG REGIO)

(15)

CODE VARIABLE NAME

HEI density Higher education density _{(Nr. of HEIs / pop. aged 18-30)}

SI5 intensity ISCED5 student intensity _{(Nr. of ISCED5 Students / pop. aged 18-26)}

IS5 mean Regional average of international students share (ISCED5) per HEI

SI6 intensity

Doctoral student (ISCED6) intensity

(Nr. of ISCED6 Students / pop. aged 22-30)

IS6 mean

Regional average of international doctoral student share (ISCED6) per HEI

RAC Ratio of HEIs defined as research active DDA intensity

Intensity of Doctoral Degrees Awarded (DDA per region)/(pop. age 22-30) * 1000

SSR mean Regional average of student to staff ratio per HEI

University

(16)

222 regional

scores

25 countries

(17)

Berkshire,

Buckinghamshire & Oxfordshire (100) E Scotland (96) NE Scotland (92)

Capital & City

Regions in Top20: Bremen (88) Vienna (85) Gr. London (81) Gr. Manchester (78) Prague (78) Gr. Brussels (75) Stockholm (75)

University

(18)

Country_Average Regional_Score

0

10

20

30

40

50

60

70

80

90

100

FI

IE

IT

SI

LT

RSU I

ndex

Country average

Regional scores

University

regional scale

(19)

RCI-Labour market performance

index (*) University System Research Index

University

regional scale

(*) This is one pillar out of 11 pillars of RCI, which includes variables on employment, short- and long-term unemployment, employment gender gap.

(20)

0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 RCI L abo r M ar ke t E ff ici ency Ind ex

University System Research Performance index

Not efficient labour market - stronger research performance Spanish/Greek regions (ES41, ES61; GR11, GR13, GR23)

University

regional scale

(21)

Summary innovation index 2010

(22)

Intergenerational Mobility in

Accessing Tertiary Education; JRC + World Bank

 86 countries

 Micro data for more than

1 million adults, including information on their

parents’ educational background

d'Hombres, B., and Nguyen-Hoang (2011) International and Time

Comparisons of Inequality in Tertiary Education. Background report to the World Bank's global study on “Equity of access and success in tertiary

education”.

Wide disparities across Europe

(23)

23 12 April 2013

How about getting

the environment

right?

(24)

-6.00 -4.00 -2.00 0.00 2.00 4.00 6.00 8.00 10.00 2003 2004 2005 2006 2007 2008 2009 2010 2011 GDP GEE GGE/GDP GEE/GGE

Year on year change of General Government expenditure on education and GDP at constant prices, of GG expenditure/GDP and of GG expenditure on education / Total GG expenditure

(EU27)  EU Governments rather than “investing” in education,

have constantly reduced education in the total budget during the crisis (Data from Eurostat).

(25)

25 12 April 2013 486 487 488 489 490 491 492 493 494 PISA 2006 PISA 2009 U n w e ig h te d A v e ra g e Cycle

Reading domain

EU countries OECD countries 491 492 493 494 495 496 497 498 499 500 PISA 2006 PISA 2009 U n w e ig h te d A v e ra g e Cycle

Mathematics domain

EU countries OECD countries

PISA data: EU versus the rest of OECD; boys and girls inn their 15’s

(26)

495 496 497 498 499 500 501 502 503 504 505 PISA 2006 PISA 2009 U n w e ig h te d A v e ra g e Cycle

Sciences domain

EU countries OECD countries

Education

(27)

6th dimension of the Rule of Law Index

(World Justice Project)

• 83 survey questions • 97 countries (20 EU)

(28)

7th_{dimension of the Rule of Law}

Index

(World Justice Project)

• 56 survey questions

(29)

29 12 April 2013 Corruption Perceptions Index (Transparency International) • 13 sources • 176 countries (27 EU)

(30)

0 20 40 60 80 100 120 140 160 180 200 S in g ap o re U n it e d S ta te s D en m a rk N o rw a y U n it e d K in g d o m A u st ra li a F in la n d M a la y si a S w ed en Ic e la n d Ir e la n d C a n a d a G er m a n y E st o n ia S a u d i A ra b ia Ja p a n L atv ia L it h u a n ia S w it ze rl an d A u str ia P o rt u g a l N et h er la n d s B e lg iu m F ra n ce S lo v en ia C y p ru s C h il e P e ru S p a in C o lo m b ia S lo v ak R e p u b li c M e x ic o S t. L u c ia H u n g ar y P o la n d L u x e m b o u rg S a m o a G h a n a C z e ch R e p u b li c B u lg ar ia A ze rb a ij an T u rk ey R o m an ia It al y S e y c h e ll es B a h a m a s, T h e G re e ce C ro a ti a Ja m a ic a C h in a M a lt a P a ra g u ay C e n tr al A fr ic an R e p u b li c

Ease of Doing Business Rank Paying Taxes

Ease of Doing Business Rank

Starting a Business

Dealing with Construction Permits Getting Electricity

Registering Property Getting Credit Protecting Inv estors

Paying Taxes

Trading Across Borders Enf orcing Contracts Resolv ing Insolv ency

World Bank Index ‘Ease of Doing Business’ + one of its sub-indices: ‘Paying taxes’; blue bars = EU countries

(31)

31 12 April 2013

(32)

Acknowledgements and thanks

Beatrice d’Hombres,

Patricia Dinis Mota Da Costa,

Luca Pappalardo,

Fiammetta Rossetti,

Michaela Saisana,

Daniel Vertesy

Unit Econometrics and Applied

Statistics, European Commission

Joint Research Centre