This section uses students’ average scores to summarise the performance of countries and economies in problem solving, both relative to each other and to the OECD mean. Since problem solving is a new domain in PISA 2012, the OECD average performance was set at 500 score points, and the standard deviation across OECD countries at 100 score points. This establishes the benchmark against which each country’s problem-solving performance in PISA 2012 is compared.
•
Figure V.2.3•
comparing countries’ and economies’ performance in problem solving Statistically significantly above the OECD average
Not statistically significantly different from the OECD average Statistically significantly below the OECD average
mean
score comparison country/economy countries and economies whose mean score is not statistically significantly different from the comparison country’s/economy’s score
562 Singapore Korea
561 Korea Singapore, Japan
552 Japan Korea
540 Macao-China Hong Kong-China, Shanghai-China
540 Hong Kong-China Macao-China, Shanghai-China, Chinese Taipei
536 Shanghai-China Macao-China, Hong Kong-China, Chinese Taipei
534 Chinese Taipei Hong Kong-China, Shanghai-China
526 Canada Australia, Finland, England (UK) 523 Australia Canada, Finland, England (UK) 523 Finland Canada, Australia, England (UK)
517 England (UK) Canada, Australia, Finland, Estonia, France, Netherlands, Italy, Czech Republic, Germany, United States, Belgium, Austria 515 Estonia England (UK), France, Netherlands, Italy, Czech Republic, Germany, United States
511 France England (UK), Estonia, Netherlands, Italy, Czech Republic, Germany, United States, Belgium, Austria, Norway 511 Netherlands England (UK), Estonia, France, Italy, Czech Republic, Germany, United States, Belgium, Austria, Norway 510 Italy England (UK), Estonia, France, Netherlands, Czech Republic, Germany, United States, Belgium, Austria, Norway 509 Czech Republic England (UK), Estonia, France, Netherlands, Italy, Germany, United States, Belgium, Austria, Norway 509 Germany England (UK), Estonia, France, Netherlands, Italy, Czech Republic, United States, Belgium, Austria, Norway 508 United States England (UK), Estonia, France, Netherlands, Italy, Czech Republic, Germany, Belgium, Austria, Norway, Ireland 508 Belgium England (UK), France, Netherlands, Italy, Czech Republic, Germany, United States, Austria, Norway 506 Austria England (UK), France, Netherlands, Italy, Czech Republic, Germany, United States, Belgium, Norway, Ireland 503 Norway France, Netherlands, Italy, Czech Republic, Germany, United States, Belgium, Austria, Ireland, Denmark, Portugal 498 Ireland United States, Austria, Norway, Denmark, Portugal, Sweden
497 Denmark Norway, Ireland, Portugal, Sweden, Russian Federation 494 Portugal Norway, Ireland, Denmark, Sweden, Russian Federation
491 Sweden Ireland, Denmark, Portugal, Russian Federation, Slovak Republic, Poland
489 Russian Federation Denmark, Portugal, Sweden, Slovak Republic, Poland
483 Slovak Republic Sweden, Russian Federation, Poland, Spain, Slovenia
481 Poland Sweden, Russian Federation, Slovak Republic, Spain, Slovenia, Serbia 477 Spain Slovak Republic, Poland, Slovenia, Serbia, Croatia
476 Slovenia Slovak Republic, Poland, Spain, Serbia
473 Serbia Poland, Spain, Slovenia, Croatia
466 Croatia Spain, Serbia, Hungary, Israel
459 Hungary Croatia, Turkey, Israel 454 Turkey Hungary, Israel, Chile
454 Israel Croatia, Hungary, Turkey, Chile, Cyprus1, 2 448 Chile Turkey, Israel, Cyprus1, 2
445 Cyprus1, 2 Israel, Chile
428 Brazil Malaysia
422 Malaysia Brazil
411 United Arab Emirates Montenegro, Uruguay, Bulgaria
407 Montenegro United Arab Emirates, Uruguay, Bulgaria
403 Uruguay United Arab Emirates, Montenegro, Bulgaria, Colombia
402 Bulgaria United Arab Emirates, Montenegro, Uruguay, Colombia
399 Colombia Uruguay, Bulgaria
1. Footnote by Turkey: The information in this document with reference to “Cyprus” relates to the southern part of the Island. There is no single authority representing both Turkish and Greek Cypriot people on the Island. Turkey recognises the Turkish Republic of Northern Cyprus (TRNC). Until a lasting and equitable solution is found within the context of the United Nations, Turkey shall preserve its position concerning the “Cyprus issue”.
2. Footnote by all the European Union Member States of the OECD and the European Union: The Republic of Cyprus is recognised by all members of the United Nations with the exception of Turkey. The information in this document relates to the area under the effective control of the Government of the Republic of Cyprus.
Source: OECD, PISA 2012 Database.
When interpreting mean performance, only those differences among countries and economies that are statistically significant should be taken into account (Box V.2.2). Figure V.2.3 shows each country’s/economy’s mean score, and allows readers to see for which pairs of countries/economies the differences between the means shown are statistically similar. The data on which Figure V.2.3 is based are presented in Annex B. For each country/economy shown in the middle column, the countries/economies listed in the column on the right are those whose mean scores are not
sufficiently different to be distinguished with confidence.3 For all other cases, Country A scores higher than Country B if
Country A is above Country B in the list in the middle column, and scores lower if Country A is shown below Country B. For example, while Finland clearly ranks above the United States, the performance of England (United Kingdom) cannot be distinguished with confidence from either Finland or the United States.
Box V.2.2. What is a statistically significant difference?
A difference is called statistically significant if it is very unlikely that such a difference could be observed in the estimates based on samples, when in fact no true difference exists in the populations.
The results of the PISA assessments for countries and economies are estimates because they are obtained from samples of students, rather than a census of all students, and they are obtained using a limited set of assessment tasks, not the universe of all possible assessment tasks. When the sampling of students and assessment tasks are done with scientific rigour, it is possible to determine the magnitude of the uncertainty associated with the estimate. This uncertainty needs to be taken into account when making comparisons so that differences that could reasonably arise simply due to the sampling of students and tasks are not interpreted as differences that actually hold for the populations.
Figure V.2.3 lists each participating country and economy in descending order of its mean problem-solving score (left column). The values range from a high of 562 points for the partner country Singapore to a low of 399 points for the partner country Colombia. Countries and economies are also divided into three broad groups: those whose mean scores are statistically around the OECD mean (highlighted in dark blue), those whose mean scores are above the OECD mean (highlighted in pale blue), and those whose mean scores are below the OECD mean (highlighted in medium blue). Box V.2.3 provides guidance to gauge the magnitude of score differences.
Because the figures are derived from samples, it is not possible to determine a country’s precise rank among the participating countries. However, it is possible to determine, with confidence, a range of ranks in which the country’s performance lies (Figure V.2.4).
Singapore and Korea are the highest-performing countries in problem solving, with mean scores of 562 points and 561 points, respectively. Fifteen-year-olds in these two countries perform about a full proficiency level above the level of students in other OECD countries, on average. Japan ranks third among all participating countries, and second among OECD countries, with a mean score of 552 points. Four more East Asian partner economies score between 530 and 540 points on the PISA problem-solving scale: Macao-China (with a mean score of 540 points), Hong Kong-China (540 points), Shanghai-China (536 points) and Chinese Taipei (534 points). Twelve OECD countries perform above the OECD average, but below the former group of countries: Canada (526 points), Australia (523 points), Finland (523 points), England (United Kingdom) (517 points), Estonia (515 points), France (511 points), the Netherlands (511 points), Italy (510 points), the Czech Republic (509 points), Germany (509 points), the United States (508 points) and Belgium (508 points).
Five countries, Austria, Norway, Ireland, Denmark and Portugal, score around the OECD mean.
There are clear and substantial differences in mean country performance on the problem-solving assessment. Box V.2.3 illustrates how the differences in mean performance compare to differences in problem-solving proficiency within countries/economies. Among OECD countries, the lowest-performing country, Chile, has an average score of 448. This means that the gap between the highest- and lowest-performing OECD country is 113 score points – well above one standard deviation. About 90% of students from Korea perform above Chile’s mean score; conversely, only about 10% of students from Chile perform above Korea’s mean score (Table V.2.2). Overall, more than two proficiency levels (163 score points) separate the highest-performing (Singapore) and lowest-performing (Colombia) countries in problem solving. Only about one in 20 students in the four best-performing countries and economies performs at or below the mean of the lowest-performing country.
•
Figure V.2.4 [Part 1/2]•
problem-solving performance among participating countries/economies
Problem-solving scale
mean score S.E.
range of ranks
oEcd countries all countries/economies upper rank lower rank upper rank lower rank
Singapore 562 (1.2) 1 2 Korea 561 (4.3) 1 1 1 2 Japan 552 (3.1) 2 2 3 3 Macao-China 540 (1.0) 4 6 Hong Kong-China 540 (3.9) 4 7 Shanghai-China 536 (3.3) 4 7 Chinese Taipei 534 (2.9) 5 7
North West (Italy) 533 (8.6)
Western Australia (Australia) 528 (4.0)
North East (Italy) 527 (6.4)
Canada 526 (2.4) 3 5 8 10
Australian Capital Territory (Australia) 526 (3.7)
New South Wales (Australia) 525 (3.5)
Flemish Community (Belgium) 525 (3.3)
Victoria (Australia) 523 (4.1)
Australia 523 (1.9) 3 6 8 11
Finland 523 (2.3) 3 6 8 11
Queensland (Australia) 522 (3.4)
German-speaking Community (Belgium) 520 (2.6)
South Australia (Australia) 520 (4.1)
England (United Kingdom) 517 (4.2) 4 11 9 16
Estonia 515 (2.5) 6 10 11 15
Centre (Italy) 514 (10.8)
Northern Territory (Australia) 513 (7.9)
France 511 (3.4) 6 14 11 19 Netherlands 511 (4.4) 6 16 11 21 Italy 510 (4.0) 7 16 12 21 Czech Republic 509 (3.1) 7 15 12 20 Germany 509 (3.6) 7 16 12 21 United States 508 (3.9) 7 16 12 21 Belgium 508 (2.5) 9 16 14 21 Madrid (Spain) 507 (13.0) Austria 506 (3.6) 8 17 13 22 Alentejo (Portugal) 506 (13.4) Norway 503 (3.3) 11 18 16 23 Ireland 498 (3.2) 15 19 20 24 Denmark 497 (2.9) 16 20 21 25
Basque Country (Spain) 496 (3.9)
Portugal 494 (3.6) 17 20 22 26
Sweden 491 (2.9) 18 21 23 27
Tasmania (Australia) 490 (4.0)
Russian Federation 489 (3.4) 23 27
Catalonia (Spain) 488 (8.4)
South Islands (Italy) 486 (8.5)
French Community (Belgium) 485 (4.4)
Slovak Republic 483 (3.6) 20 23 25 29
Poland 481 (4.4) 21 24 26 31
Spain 477 (4.1) 21 24 27 31
Slovenia 476 (1.5) 22 24 28 31
Notes: OECD countries are shown in bold black. Partner countries and economies are shown in bold blue. Regions are shown in black italics (OECD countries) or blue italics (partner countries).
Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), North East (Bolzano, Emilia Romagna,
Friuli Venezia Giulia, Trento, Veneto), North West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands
(Basilicata, Calabria, Sardegna, Sicilia).
Brazilian states are grouped into larger geographical units: Central-West Region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), Northeast Region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), North Region (Acre, Amapá, Amazonas, Pará, Rondônia,
Roraima, Tocantins), Southeast Region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South Region (Paraná, Rio Grande do Sul, Santa Catarina).
1. Footnote by Turkey: The information in this document with reference to “Cyprus” relates to the southern part of the Island. There is no single authority representing both Turkish and Greek Cypriot people on the Island. Turkey recognises the Turkish Republic of Northern Cyprus (TRNC). Until a lasting and equitable solution is found within the context of the United Nations, Turkey shall preserve its position concerning the “Cyprus issue”.
2. Footnote by all the European Union Member States of the OECD and the European Union: The Republic of Cyprus is recognised by all members of the United Nations with the exception of Turkey. The information in this document relates to the area under the effective control of the Government of the Republic of Cyprus.
Countries, economies and subnational entities are ranked in descending order of mean problem-solving performance.
Source: OECD, PISA 2012 Database.
•
Figure V.2.4 [Part 2/2]•
problem-solving performance among participating countries/economies
Problem-solving scale
mean score S.E.
range of ranks
oEcd countries all countries/economies upper rank lower rank upper rank lower rank
South (Italy) 474 (8.4)
Serbia 473 (3.1) 29 32
Croatia 466 (3.9) 31 33
Hungary 459 (4.0) 25 27 32 35
Dubai (United Arab Emirates) 457 (1.3)
Turkey 454 (4.0) 25 28 33 36
Israel 454 (5.5) 25 28 33 37
Chile 448 (3.7) 26 28 34 37
Southeast Region (Brazil) 447 (6.3)
Cyprus1, 2 445 (1.4) 36 37
Central-West Region (Brazil) 441 (11.9)
South Region (Brazil) 435 (7.8)
Brazil 428 (4.7) 38 39
Medellín (Colombia) 424 (7.6)
Manizales (Colombia) 423 (5.3)
Malaysia 422 (3.5) 38 39
Sharjah (United Arab Emirates) 416 (8.6)
United Arab Emirates 411 (2.8) 40 41
Bogotá (Colombia) 411 (5.7) Montenegro 407 (1.2) 40 42 Uruguay 403 (3.5) 41 44 Bulgaria 402 (5.1) 41 44 Colombia 399 (3.5) 42 44 Cali (Colombia) 398 (9.0)
Fujairah (United Arab Emirates) 395 (4.0)
Northeast Region (Brazil) 393 (11.0)
Abu Dhabi (United Arab Emirates) 391 (5.3)
North Region (Brazil) 383 (10.9)
Ajman (United Arab Emirates) 375 (8.0)
Ras al-Khaimah (United Arab Emirates) 373 (11.9)
Umm al-Quwain (United Arab Emirates) 372 (3.5)
Notes: OECD countries are shown in bold black. Partner countries and economies are shown in bold blue. Regions are shown in black italics (OECD countries) or blue italics (partner countries).
Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), North East (Bolzano, Emilia Romagna,
Friuli Venezia Giulia, Trento, Veneto), North West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands
(Basilicata, Calabria, Sardegna, Sicilia).
Brazilian states are grouped into larger geographical units: Central-West Region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), Northeast Region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), North Region (Acre, Amapá, Amazonas, Pará, Rondônia,
Roraima, Tocantins), Southeast Region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South Region (Paraná, Rio Grande do Sul, Santa Catarina).
1. Footnote by Turkey: The information in this document with reference to “Cyprus” relates to the southern part of the Island. There is no single authority representing both Turkish and Greek Cypriot people on the Island. Turkey recognises the Turkish Republic of Northern Cyprus (TRNC). Until a lasting and equitable solution is found within the context of the United Nations, Turkey shall preserve its position concerning the “Cyprus issue”.
2. Footnote by all the European Union Member States of the OECD and the European Union: The Republic of Cyprus is recognised by all members of the United Nations with the exception of Turkey. The information in this document relates to the area under the effective control of the Government of the Republic of Cyprus.
Countries, economies and subnational entities are ranked in descending order of mean problem-solving performance.
Source: OECD, PISA 2012 Database.
12 http://dx.doi.org/10.1787/888933003573
Box V.2.3. interpreting differences in piSa problem-solving scores: how large a gap?
In PISA 2012, student performance in problem solving is described through six levels of proficiency, each of which represents 65 score points. Thus, a difference in performance of one proficiency level represents a comparatively large disparity in performance. For example, students proficient at Level 2 on the problem-solving scale are only starting to demonstrate problem-solving competence. They engage with unfamiliar problem situations, but need extensive guidance in order to progress towards a solution. They can perform only one task at a time, and can only test a simple hypothesis that is given to them. Meanwhile, students proficient at Level 3 are more self-directed in their problem solving. They can devise hypotheses to test themselves, and can handle multiple constraints by planning a few steps ahead, provided that the constraints can be addressed sequentially.