has shown that two major dimensions along which performances of countries/economies differ are related to whether interaction with the problem situation is needed in order to uncover relevant information, and depending on whether the task primarily corresponds to knowledge-acquisition or to knowledge-utilisation processes.
Together, the differences in performance according to the nature of the problem situation and the major problem-solving process targeted identify several groups of countries/economies (Figure V.3.10). Interestingly, these groups often overlap with historical and geographical groupings.
•
Figure V.3.10•
Joint analysis of strengths and weaknesses, by nature of the problem and by process
Note: This figure plots the odds ratios for success on interactive items, compared to static items, on the vertical axis, and the odds ratios for success on knowledge-acquisition tasks (“exploring and understanding” or “representing and formulating”), compared to knowledge-utilisation tasks (“planning and executing”), on the horizontal axis. Both axes are in logarithmic scale.
Source: OECD, PISA 2012 Database, Tables V.3.1 and V.3.6.
BETTER PERFORMANCE ON INTERACTIVE TASKS, RELATIVE TO STATIC TASKS
Stronger-than-expected performance on interactive items, weaker-than-expected performance on knowledge-acquisition tasks
Stronger-than-expected performance on interactive items and on knowledge-acquisition tasks
Weaker-than-expected performance on interactive items and on knowledge-acquisition tasks
Weaker-than-expected performance on interactive items, stronger-than-expected
performance on knowledge-acquisition tasks
OECD average
OECD average
BETTER PERFORMANCE ON KNOWLEDGE-ACQUISITION TASKS, RELATIVE TO KNOWLEDGE-UTILISATION TASKS Norway Austria Italy Croatia Serbia Turkey United States Singapore Czech Republic Sweden Australia England (UK) Macao-China Shanghai-China Chinese Taipei Hong Kong-China Uruguay Japan Poland Portugal Canada Slovenia Germany Belgium France Spain Hungary Colombia
United Arab Emirates
Malaysia Brazil Montenegro Bulgaria Denmark Finland Ireland Russian Federation Chile Israel Estonia Slovak Republic Netherlands Korea 1 2http://dx.doi.org/10.1787/888933003592
Six East Asian countries and economies, namely Korea, Singapore, Hong Kong-China, Macao-China, Chinese Taipei and Shanghai-China, stand out for their very high success rates on knowledge-acquisition tasks, compared to their success rates on planning and executing tasks. Within this group, however, there are relatively stark differences in their performance on interactive problems. Students in Korea and Singapore are significantly more at ease with these
problems than students in Shanghai-China, Chinese Taipei and Macao-China. Students from Hong Kong-China are in a middle position.
While all of these countries and economies rank in the top positions for overall performance, this analysis suggests that in Shanghai-China, Chinese Taipei and Macao-China, a focus on students’ skills at dealing with interactive problem situations is required in order to improve further and close the performance gap with Korea and Singapore. In reviewing their curricula, teachers and curriculum developers may want to introduce more opportunities for students to develop and exercise the traits that are linked to success on interactive items, such as curiosity, perseverance and creativity. They may find inspiration in the curricula and teaching practices of their regional neighbours.
Among lower-performing countries and economies in problem solving, the low performance of Latin American countries (Brazil, Colombia, Chile and Uruguay) appears to be mainly due to a large performance gap on knowledge-acquisition tasks. These countries have no particular difficulty with interactive tasks – and Brazil even shows a relative strength on such tasks.
In these countries, efforts to raise problem-solving competency should concentrate mainly on improving students’ performance on “exploring and understanding” and on “representing and formulating” tasks. These tasks require students to build mental representations of the problem situation from the pieces of information with which they are presented. Moving from the concrete problem scenario to an abstract representation and understanding of it often demands inductive or deductive reasoning skills. Teachers and curriculum experts may question whether current curricula include sufficient opportunities to model these abstract reasoning skills and whether these opportunities are offered in the classroom.
In contrast, several countries in Southern and Eastern Europe, namely Bulgaria, Montenegro, Slovenia, Croatia and Serbia, show relatively weak performance both on knowledge-acquisition tasks and on interactive tasks, compared to their performance on “planning and executing” and on static tasks. In these countries, students seem to find it particularly difficult to understand, elaborate on, and integrate information that is not explicitly given to them (in a verbal or visual format), but has to be inferred from experimental manipulation of the environment and careful observation of the effects of that manipulation. Students in these countries may benefit from greater opportunities to learn from hands-on experience.
The performance gap between OECD countries in Europe and North America and the top-performing countries in problem solving mainly originates from differences in students’ performance on knowledge-acquisition tasks. In general, the PISA problem-solving assessment shows that there is significant room for improving students’ ability to turn information into useful knowledge, as measured by performance differences on the dimensions of “exploring and understanding” and “representing and formulating” problem situations.
Within this group, Ireland and the United States stand out for their strong performance on interactive items, compared, for instance, to the Nordic countries (Sweden, Finland, Norway and Denmark), the Netherlands, and some countries in Central Europe (in particular, Poland, Hungary and the Slovak Republic). Therefore, the analysis also identifies a strong potential for the Nordic and Central European countries to improve on their students’ ability to cope with interactive problem situations. To do so, educators may need to foster such dispositions as being open to novelty, tolerating doubt and uncertainty, and daring to use intuition to initiate a solution.
Finally, several countries, while performing at different levels, show a similar balance of skill when compared to each other, and one that is close to the OECD average pattern of performance. Italy and Australia, for instance, have a very similar pattern of performance to that observed in Japan, although in terms of overall performance, Japan ranks significantly above Australia, which, in turn, performs better than Italy. These three countries all perform close to their expected level on interactive items (based on the OECD average pattern of performance), and slightly above their expected level on knowledge-acquisition tasks (although the example of Korea and Singapore shows that significant gains are still possible for them). In other countries, such as Spain, England (United Kingdom) and Germany, performance across tasks reflects the balance observed across OECD countries, on average.
For students in this group of countries, as a whole, there are no clear indications as to which aspects of problem-solving competence deserve particular attention. Nevertheless, the profile of performance may differ across particular groups of students. Such differences across groups of students will be analysed in the next chapter.
Two notes of caution: First, throughout this chapter, patterns of performance within countries and economies have been compared to the OECD average patterns in order to identify comparative strengths and weaknesses. Implications drawn from this analysis tacitly assume that this international benchmark corresponds to a desirable balance between the various aspects of problem-solving competence. The OECD average was selected for pragmatic reasons only. Therefore, the normative interpretation of the benchmark can be challenged, and alternative comparisons (for instance, to the pattern observed in the top-performing country) are equally possible.
Second, although this analysis can provide interesting indications, any conclusion that is drawn from subsets of the PISA problem-solving test must be carefully checked against evidence collected independently in each system on the strengths of the respective curriculum and teaching practices. Lacking supporting evidence, the conclusions should be interpreted with caution. Indeed, the PISA problem-solving assessment comprises a total of 42 items. When success is analysed on subsets of items that share common characteristics, the number of items inevitably drops. While the 42 items together reflect a consensus view of what problem-solving competence is, when this item set is split into smaller sets to analyse the individual components of problem-solving competence, the resulting picture is necessarily
less sharp.3 The results of analyses based on small sets of items may sometimes be driven by idiosyncratic features of one
or two items in the pool rather than by their common traits.
Notes
1. A complementary analysis that can diagnose more detailed strengths and weaknesses will be made possible by the availability of behavioural sequences recorded by the computer interface (process data). After having identified the elementary task demands of each assessment item, the data recording students’ interactions with items can be used, for instance, to identify patterns in terms of frequent stumbling blocks that hinder students from reaching the solution.
2. Fisher’s exact test of independence of rows and columns was performed. The null hypothesis of independence of rows and columns for the contingency tables pairing the cognitive processes with the nature of the problem situation cannot be rejected (p-value: 0.69). 3. This is a problem of external validity that is not reflected in the standard errors provided with the statistical analysis in this chapter. While the inference about strengths and weaknesses is internally valid for the particular test of problem solving analysed, the question of external validity is whether a different test, constructed according to the same definition and framework, would give exactly the same results: i.e. to what extent one can generalise from performance on a dozen items to competence on the unobserved construct underlying these items.
References
OECD (2013), PISA 2012 Assessment and Analytical Framework: Mathematics, Reading, Science, Problem Solving and Financial Literacy, OECD Publishing.
http://dx.doi.org/10.1787/9789264190511-en