Introduction: From measuring ICT diffusion to measuring impact, a bumpy road

118. After four years in effect, there is little doubt that Plan Avanza has left a mark on Spain’s information society: there is greater public and political consciousness about the importance a strong knowledge economy for the country’s future, stakeholders are increasingly active in IS policy, and a growing number of sub-national governments and NGOs are participating more ambitiously in the Plan. While these accomplishments have established an impressive momentum in furthering the IS agenda in Spain, the challenge for policy makers remains in establishing what benefits have been achieved for targeted beneficiaries. This chapter contributes to this goal, taking a closer look at how the Plan’s initiatives have affected citizens, businesses and the public administration. As the backbone to these achievements, progress in the deployment of key ICT infrastructures are also examined.

119. But, what constitutes a successful IS strategy? How is impact on society defined in the context of IS policy? One way of examining the results of IS strategies is by utilising the S-curve framework,44 which establishes three phases of IS development: e-readiness, e-intensity and e-impact.

Figure 4.1 Assessing Information Society Development: the S-curve model45

Source: OECD elaboration, based on model by Industry Canada.

 E-Readiness: E-readiness refers to the extent to which the technological, commercial and social

infrastructures necessary to support a competitive information society have been established. The definition also extends to the quality of these infrastructures. Because ICT infrastructures serve as platforms for the development of ICT goods and ICT-supported services, e-readiness indicators allow policy makers to construct a statistical picture of whether an information society is capable of supporting the development and use of these additional applications.

 E-Intensity: E-intensity indicators, on the other hand, assess the extent to which ICTs and ICT-

supported services are exploited by users. Indicators on ICT-usage, the frequency and purpose of their use, as well as the quality of services provided can help establish a benchmark for whether the ICT infrastructure developed in the e-readiness stage has the potential to provide added worth to society, the next stage of the S-curve.

 E-Impact: The ultimate goal of IS policies is to maximise the potential of ICTs to contribute to

social and economic outcomes. In this context, E-impact refers to the value that is created by the usage of ICT platforms and ICT-supported services. Measuring the impact of IS policies, then,

75

involves determining whether- and to what extent- ICTs have contributed to increasing efficiency and cost-effectiveness, produced savings, created new sources of wealth, or generated welfare for citizens.

120. What are some indications that a “tipping-point” has been reached between one phase of the S- curve and another? In addition to monitoring data representing the progression of ICT access, usage and/or value-creation, an additional signal that suggest policy makers should begin to consider shifting priorities is the decreasing cost-effectiveness of current policies. Indeed, broad-based policies aimed at all the whole of society or economy become less effective over time due to diminishing returns. For example initiatives to increase broadband adoption in rural households will (eventually, with progress) mean that adoption rates slow, and investments/efforts made in communications, subsidies, etc. to extend will be less cost-effective with time. This requires that, in transitional phases of the S-curve, policies become more targeted and precise: seeking to impact specific socio-economic or demographic groups or specific sectors. It is through reaching the specific groups that are farther behind in IS development that, at an aggregate level, greater progress can be made.

121. Relative to measuring states of readiness or intensity, measuring the impact of IS policies can be more problematic. Indeed, measuring ICT-supported value creation in terms of such factors as the amount of revenues generated from e-commerce, growth of ICT sector, trust in government, satisfaction with public services, or perceived quality of life- and linking these to the investments and efforts of IS policies is a challenge for policy makers due to three issues:

 The magnitude of the impact of ICTs on socio-economic outcomes varies and can be difficult to quantify. In some instances, underlying causality mechanisms remain the subject of ongoing research. Although a general consensus exists among policy makers and researchers about the

positive role that ICTs can play in pursuing socio-economic objectives, research is ongoing to measure the scale of these effects.

 A multitude of externalities can affect IS targets. IS strategy objectives are broad, and such

outcomes depend on a host of externalities and macroeconomic conditions. Isolating the influence of ICTs from that of other factors can complicate policy makers’ assessments.

 The development of outcome-based- or impact- indicators for IS strategies is ongoing: until

recently, the evaluation of IS policy has focused on (i) access/coverage of ICT goods and services, and (ii) take-up/usage of services supported by ICTs or the Internet. Efforts are underway to complement such indicators with those that assess the outcomes, and international stakeholders are co-operating to reach a consensus on standardised definitions and methodologies for comparison.

Box 4.2 The challenges of measuring e-impact: the productivity paradox in education

In the sector of Education, the productivity paradox presents one example of some of the aforementioned challenges of measuring e-impact. For instance, what is expected to follow the adoption of a new technology and justify the corresponding investment is the expectation of an increase in productivity. But, as the productivity paradox states, what actually happens is that if a new technology is adopted in a context in which processes are not changed, technology may be found to be useless, if not obtrusive, and in many cases may even lead to a decrease in productivity.

In education, technology is a tool that can be used for a variety of purposes. Whether the adoption of technology leads to improvements in educational performance will depend on whether its introduction contributes to changes in the learning process or in the educational system which will produce measurably better outcomes. Without such changes, and when technology is used merely to maintain an existing pedagogical process or structure, the positive results may be limited, especially over the longer term. Additionally, improvements associated with changes in methodology which require appropriate technical and pedagogical support.

The right question, therefore, is not which new technology leads to increased productivity, but which new technology-supported methodologies improve student performance over traditional ones and what other factors intervene. The almost infinite array of methodological possibilities makes this kind of investigation extremely difficult, but it is not impossible provided that sufficient effort is devoted to the accumulation and dissemination of the resulting knowledge base.

Some of the pieces already available of such a knowledge base are well known, but the resulting landscape is still fragmentary. For instance, two examples of existing research-based knowledge are that the adoption of technology in the classroom increases the motivation and level of engagement of students in the classroom, and, as the most recent OECD report on technology use and educational performance using PISA data shows, there is sustained evidence that home use of technology by 15-year-olds is linked to educational performance. Yet, such a link cannot easily be made in respect to school use of technology, partly because of the productivity paradox (using technology to teach the same way), and partly because of the threshold effect (insignificant levels of technology use in the classroom that can hardly be compared with the engagement of today’s learners with technology at home). Source: OECD (2010), Are the New Millennium Learners Making the Grade? Technology Use and Educational Performance in PISA, OECD Publishing, Paris.