the problem situation
Exploring and understanding (10 items) representing and formulating (9 items) Planning and executing (16 items) monitoring and reflecting (7 items) Static (15 items) 5 2 6 2 Interactive (27 items) 5 7 10 5
Source: OECD, PISA 2012 Database.
In addition to these two aspects, each assessment unit is also characterised, on a more superficial level, by the particular context in which the problem situation occurs. The framework distinguishes problems with a social focus from problems with a personal focus, as well as problems cast in a technological setting from problems cast in a non-technological setting.
Items in the problem-solving test can also be classified according to their response format. A major distinction is between selected-response formats, which ask respondents to choose one or more answers from a closed list of possible responses, and constructed-response formats, where students produce a self-constructed response.
Nature of the problem situation
How a problem is presented has important consequences for how it can be solved. Of crucial importance is whether the information about the problem disclosed at the outset is complete. These problem situations are considered static. Question 3 in the problem-solving unit TRAFFIC, described in the sample tasks section at the end of Chapter 1, is an example of a static unit: students are given all information about travel times and have to determine the best location for a meeting.
By contrast, problem situations may be interactive, meaning that students can explore the situation to uncover additional relevant information. Real-time navigation using a GPS system, where traffic congestion may be reported in response to a query, is an example of such a situation.
Interactive problem situations
Interactive problem situations often arise when encountering technological devices, such as ticket-vending machines, air-conditioning systems or mobile telephones for the first time, especially if the instructions for using them are not clear or are not available. Individuals often confront these types of problems in daily life. In these situations, some relevant information is often not apparent at the outset. For example, the effect of performing an operation (say, pushing a button on a remote control) may not be known and cannot be deduced, but rather must be inferred by actually performing the operation (pushing the button) and forming a hypothesis about its function based on the outcome. In general, some exploration or experimentation is needed to acquire the knowledge necessary to control the device. Another common scenario is when a person must troubleshoot a fault or malfunction in a device. Here a certain amount of strategic experimentation – generating and testing hypotheses – must take place in order to collect data on the circumstances under which the device fails.
Interactive problem situations can be simulated in a test setting by a computer. Including interactive problem situations in the computer-based PISA 2012 problem-solving assessment allows for a wider range of authentic, real-life scenarios to be presented than would otherwise be possible using pen-and-paper tests. Problems where the student explores and controls a simulated environment are a distinctive feature of the assessment.
Static problem situations
In static problems all relevant information is disclosed at the outset and the problem situation is not dynamic, i.e. it does not change during the course of solving the problem.
Examples of static problems are traditional logic puzzles, such as the Tower of Hanoi and the water jars problems (“How would you use three jars with the indicated capacities to measure out the desired amount of water?”); decision-making problems, where the student is required to understand a situation involving a number of well-defined alternatives and constraints so as to make a decision that satisfies the constraints (e.g. choosing the right pain killer given sufficient details about the patient, the complaint and the available pain killers); and scheduling problems for projects, such as building a house or generating a flight schedule for an airline, where a list of tasks with durations and relationships between tasks is given.
Figure V.3.2 illustrates how the nature of the problem situation varies across the PISA 2012 problem-solving items that were made public. While all of the interactive units shown in Figure V.3.2 are set in technology contexts, the assessment also included interactive problems in non-technology contexts; for instance, some items ask students to orient themselves in a maze. Overall, a majority of items – 27 of 42 – are interactive.
•
Figure V.3.2•
examples of problem-solving tasks, by nature of the problem nature of the problem situation Sample questions
Interactive MP3 PLAYER – Items 1, 2, 3 and 4 (field trial) CLIMATE CONTROL – Items 1 and 2 TICKETS – Items 1, 2 and 3
Static TRAFFIC – Items 1, 2 and 3
ROBOT CLEANER – Items 1, 2 and 3 Source: OECD, PISA 2012 Database.
What success on interactive tasks implies for education policy and practice
The static or interactive nature of the problem situation is related to how information is presented. Static problems, where all relevant information is disclosed at the outset, are the typical textbook problems encountered in schools, whereas in most contexts outside of schools, the relevant information to solve the problem has to be obtained by interacting with the environment. Static problems can be regarded as a special case of interactive problems. This highlights the fact that the set of skills that are required to solve static tasks is a subset of the skills required for interactive tasks.
To excel in interactive tasks, it is not sufficient to hold the problem-solving skills required by static, analytical problems; students must also be open to novelty, tolerate doubt and uncertainty, and dare to use intuitions (“hunches and feelings”)
to initiate a solution. A relatively weak performance on interactive items, compared to performance on static items, may indicate that students may benefit from greater opportunities to develop and exercise these traits, which are related to curiosity, perseverance and creativity.
Success on interactive and static tasks
Figure V.3.3 plots average success rates for interactive items against average success rates for static items. The figure immediately reveals that, in general, country rankings are similar across the two types of items. Performance on interactive items is strongly related to performance on static items. However, as Figure V.3.3 shows, performance is not always perfectly aligned. Countries that share similar levels of success on static items do not necessarily share the same performance on interactive items. Often, when considering two countries with similar performance on static items, one country is significantly stronger on interactive items than the other.
•
Figure V.3.3•
differences in countries’/economies’ success on problem-solving tasks, by nature of the problem
1 2http://dx.doi.org/10.1787/888933003592
Average percentage of full-credit responses
for
interactive items
0 10 20 30 40 50 60 70
Note: Ireland and Sweden share similar levels of performance overall, but illustrate different patterns of performance across interactive and static items; this example is discussed in the text.
Source: OECD, PISA 2012 Database, Table V.3.1.
Average percentage of full-credit responses for static items 70 60 50 40 30 20 10 0
Interactive and static items
Ireland
Sweden
In Ireland, for instance, the percentage of full-credit answers was, on average, 44.6% across all items. This resulted from a 44.4% success rate on static items and a 44.6% success rate on interactive items. Because interactive items were found to be slightly harder than static items, on average across OECD countries, it can be deduced that performance on interactive items was stronger than expected in Ireland. In comparison, the success rate of students in Sweden (43.8%) was similar to that of students in Ireland overall, but this resulted from a higher success rate on static items (47.7%) and a lower success rate on interactive items (41.6%). While the former is in line with the OECD average, the latter is significantly below the OECD average (Figure V.3.3 and Table V.3.1).
Figure V.3.4 ranks countries and economies according to whether their students had greater success on interactive or on static tasks, after accounting for overall differences in performance. This analysis accounts for the relative difficulty of static and interactive tasks by comparing relative success in each country/economy to the average relative success across OECD countries. It also adjusts for country/economy-specific response format effects (Figure V.3.9). To continue with the same example used above, the measure of relative success on interactive items is 1.16 in Ireland – and thus significantly above 1, indicating stronger-than-expected performance on interactive items. Relative success is only 0.91 in Sweden (significantly below par), indicating weaker-than-expected performance on interactive items (Table V.3.1).
Compared with students in other OECD countries, students in Ireland, Korea, Brazil, the United States, Portugal, Singapore, Canada and Japan were more successful on interactive tasks than expected, given their overall performance. In contrast, students in Bulgaria, Montenegro, Slovenia, Sweden, Denmark, Shanghai-China, Chinese Taipei, Finland, the Slovak Republic, Austria, the Netherlands, Croatia and Serbia had more facility with static tasks than with interactive tasks, as compared to the relative success of students in other OECD countries. This may indicate a difficulty related to the specific skills used uniquely to solve interactive tasks.