Conclusions

Monitoring and measurement are critical in combating marginalization. They should be an integral part of strategies aimed at identifying social groups and regions that are being left behind, raising their visibility, and identifying what works in terms of policy intervention. Effective monitoring and disaggregated data are needed for assessing progress towards equity-based targets. Too often, national statistical surveys fail to adequately capture the circumstances and conditions of those being left behind, reinforcing their marginalization. Timely data for monitor- ing equity gaps in learning are even harder to come by.342

This volume began with a question: Can the available research on the assessment of

learning (and in learning to read, in particular) contribute to a more effective way to improve educational outcomes in developing countries? It has been widely assumed

that the answer is “yes.” But, it has not been clear which types of assessments ought to be used for which purposes. Up until fairly recently, most national and interna- tional policy makers relied on highly technical international comparative studies to determine how things are going. In contrast, local leaders, school headmasters, and teachers often relied on school-leaving national examinations in order to determine how “their” students were learning. As described herein, both of these approaches misses the child as a learner. Rather than deal with how a child is learning to read, most policy makers have known only the summative individual or average score, with very little information on how that score came to be what it is, and what should be done in terms of improved instructional design. With the advent of SQC hybrid assessments, it is now possible to pinpoint the nature of children’s difficulties and to do so with a precision and timing that will allow for potential intervention before it is too late.

This change from the macro-view to the micro-view is not trivial. It is also not complete. Much work still needs to be done to anchor new SQC approaches into future education decision-making.

Some Issues that Remain in Developing New Assessments

Choices of, and results from, learning assessments likely will be debated for years to come, and these discussions will make the field of educational quality richer. As the knowledge base on assessment continues to grow, the next steps will likely take the form of expanding and deepening the use of indicators as one essential avenue for improving learning, teaching, and schooling worldwide, and in particular for those most in need in developing countries.

This review has raised many issues. Some of these seem to have clear paths to improvement based on reasonably solid research. For example, hybrid assess- ments can address policy issues around basic learning skills in efficacious ways that are more focused, time-sensitive, and generally less expensive. They can also be tailored according to linguistic and orthographic variation with both validity and reliability. Furthermore, they can be designed to effectively address the interests of an expanded set of stakeholders.

At the same time, hybrid SQC assessments are only beginning to be understood conceptually, empirically and in practice, in part because knowledge about their use is still at an early stage relative to LSEAs. To refine and extend the use of hybrid assessments, a number of key questions require further research based on field trials in PSEs in low-income countries. These include the following. a. Reading comprehension. Component skills have been the main focus of hy-

brid studies to date. More needs to be known about the consequential relation- ship between these skills and reading comprehension.

b. Longitudinal studies. To improve the predictive power of hybrid studies, it will be crucial to follow students from first grade through the end of primary school, or through overlapping short-term longitudinal studies. This will be especially important for intervention studies (see next point).

c. Intervention studies. Research has recently begun to determine how reading component skills can be introduced into the curriculum (see Chapter 5, on the Liberia Plus project). More interventions that look at a wider range of variables will be required to better understand the variety of interventions that can be effective across different contexts and languages.

d. Instructional design. The ultimate goal of research on student learning is to de- sign better ways of instruction, where instruction involves all the different fac- tors that relate to learning (such as teacher preparation, materials development, pre-school skills, and parental support). Hybrid assessments need to be able to inform and improve instructional design, without overburdening teachers. e. Timed testing. ORF rates can be benchmarked in terms of words per minute

and various rates have been suggested as necessary to read with comprehension. Nonetheless, some questions have been raised about the pedagogical consequenc- es of timed tests in PSEs. Is the intensive timing approach a problem? Are there effective alternatives to strict time pressure? These questions need further research.

f. Individual versus group testing. To date, nearly all EGRA and similar testing has been done on an individual basis. There are good reasons for this, especially with very young children. Even so, given the cost in time and resources to engage each child in individualized testing, further research should consider ways of obtaining similar high-quality data from group-style testing, where appropriate.

g. L1 and L2 reading. Most of the knowledge base on first and second language literacy is derived from studies in OECD countries, particularly with the L2 be- ing English. The variety of languages and orthographies in use in LDCs should provide opportunities to better understand first and second language reading acquisition, and ways to assure smoother and more effective transitions in both oral and written modes.

h. Child and adult reading acquisition. Most research to date has generally treated adults as different from children when it comes to learning to read. A life-span understanding of reading acquisition is needed, followed by assess- ments built upon this model. Instructional reading programs for children and adults could benefit from a closer understanding and interaction.

i. Reading, math, and other cognitive skills. Hybrid reading assessments have

made progress in recent years, and some work has begun on math as well.343

Future hybrid assessment methods, conducted during the primary school years, could provide similar value to that of hybrid reading assessments.

j. System-wide EGRA. To what extent should education systems collect data on early literacy skills? At present, large-scale systemic assessment using EGRA or EGRA-like tools is being done to date only in India (through Pratham), while most other countries participating in EGRA have focused on smaller samples in limited regions. It will be important to better understand the pros and cons

of using hybrid assessments for larger systemic education purposes.344

k. Individualized diagnostics. Schools in OECD countries usually have a read- ing specialist who can assist children who exhibit problems in early reading. Such specialists are rare in PSEs in developing countries. Hybrid assessments may be one way for teachers or assistants to give children special help with learning to read that they would not otherwise receive. Research will need to clarify the possibilities of individualized interventions.

l. Psychometrics of SQC assessments. What are the statistical characteristics of using SQC assessments? How large do assessments need to be in item sampling and population sampling to be statistically reliable, and how much depends on the particular context and population sample in which the assessment is undertaken? More work needs to be done on these and related empirical and psychometric issues as the use of hybrid assessments expands.

343. See the work of the Research Triangle Institute on EGMA (Early Grade Math Assessment). See https://www.ed- dataglobal.org/documents/index.cfm?fuseaction=showdir&ruid=5&statusID=3. (accessed March 14, 2010). 344. Thanks to M. Jukes for this helpful observation.

Use of Information and Communications Technologies

Given the many complexities of developing appropriate assessments, especially in very poor contexts, it may seem a step too far to consider adding technology to the mix of issues to consider. Yet, technology is rapidly changing all lives in our era of increasing globalization, and must be considered where possible in this discussion as well. Information and communications technologies (ICTs) can and will provide support for the kinds of recommendations made in this review: whether in multi- lingual instructional support, data collection in the field, or use of communications

for more real-time implementation of interventions.345

It is variously estimated that only a tiny fraction (less than 5 percent) of ICT investments globally have been made that focus on poor and low-literate

populations.346 _{Many of the current ICT for education (ICT4E) efforts, even if}

deemed to have been successful in terms of overall impact, have not included a sufficiently pro-poor perspective. For example, the vast majority of software/web content (mainly in major languages such as English, Chinese, French, Spanish) is of little use to the many millions of marginalized people for reasons of literacy, language or culture. It is increasingly clear that user-friendly and multi-lingual ICT-based products can satisfy the needs of the poor to a much greater extent than heretofore believed. Providing such tools and developing the human resources ca- pacity to support the local development and distribution of relevant content is one important way to help initiate a positive spiral of sustainable development.

How can SQC assessments help this situation? First, there are new op- portunities to provide ICT-based instructional environments that build on what we are learning from assessments of reading. Can we, for example, provide indi- vidualized instruction in multiple languages that build on childrens’ skill levels in each language? Data coming from recent work in India and South Africa is

gives some reason to be optimistic.347 _{Second, ICTs can also be used to collect}

data using hybrid assessment instruments. When implemented properly, such tools (based most likely on mobile phone platforms) will be able to provide not only more reliable data at the point of collection, but also far greater possibilities reducing the time needed for data transfer and analysis, a major priority for SQC

approaches to assessment.348

345. For a broad review of the domain of monitoring and evaluation using ICTs in education, see Wagner, 2005; and in support of literacy work, see Wagner & Kozma, 2005.

346. Wagner & Kozma, 2005.

347. In a project undertaken in Andhra Pradesh (India), Wagner (2009b) and Wagner et al. (2010) found positive results from a short-term intervention using local language (Telugu-based) multimedia to support literacy learning among primary school children and out-of-school youth, using SQC-type assessment tools. This study also showed the power of ICTs in supporting multilingual learning environments among very poor populations who had little or no previous experience with computers. Moving ahead, important gains in SQC assessments will likely use ICTs (mobile devices especially) to provide more timely and credible data collection.

348. The International Literacy Institute has recently developed just such a tool, based on an Android operating system for collecting EGRA data in the field. Application of this tool has yet to take place.

Moving Forward

The effective use of educational assessments to improve learning is fundamental. However, effective use does not only refer to the technical parameters of, say, reliability and validity. What is different today is putting a greater priority on near-term, stakeholder

diverse, culturally sensitive, and high-in-local-impact assessments. Learning assessments—

whether large-scale, household surveys, or hybrid (smaller, quicker, cheaper)—are only as good as the uses that are made of them. Most are being constantly improved and refined. Globalization, and efforts by the wealthier countries to compete for the latest set of “global skills” will continue, and these countries will no doubt benefit as a function of such investments. But the globalization of assessments, if narrowly de- fined around the fulcrum of skills used in industrialized nations, will necessarily keep the poorest children at the floor of any global scale, making it difficult to understand the factors that lead poor children to stay inadequately served. Current efforts to broaden the ways that assessments are undertaken in developing countries will grow as an important part of solutions that aim at quality improvement in education.

Learning assessments can break new ground in educational accountabil- ity, largely by having as a policy goal the provision of information that matters to specific groups in a timely manner, such that change becomes an expectation. Current efforts to broaden the ways that assessments are undertaken in developing countries will enhance accountability, and it is only through improved accountabil- ity that real and lasting change is possible. Finally, and crucially, there is a need to sustain a significant policy and assessment focus on poor and marginalized popula- tions—those who are the main target of the MDG and EFA goals. They are poor children in poor contexts. They are those at the bottom of the learning ladder.

Aminata’s Story: An Update

It’s two years later, and Aminata is now 11 years old, and is still in school. Her instructor, Monsieur Mamadou, was able to participate in a training about how to make his school better, and how to help his learners read better—a first for the school. Instead of being left out of the teaching and learning, Aminata has found herself called upon in class, as have all the children, even in the back rows. There is now a community spirit, ever since Monsieur Mamadou said that if all the children learn how to read, the school will win a prize. Aminata didn’t think much of this until the whole class received new primers, one for each child. Each primer was written in her home language, with the later part of the primer in French. There are colored drawings inside the book, and lots of fun exercises on how to pronounce letters and syllables and words. Aminata practices these outside of class with her cousin, and has finally figured out how to break the code, so that she can now read lots of words. She can also help her mother with her medicine prescriptions, as well as her sister who has just entered first grade in school.

Could this updated story happen? For many who work in the field of international education, such a revisionist story seems unlikely. Furthermore, it would seem dif- ficult to include such a story in a review of the technical aspects of learning assess- ments. Nonetheless, it is likely to be the only way this story will come to fruition. Aminata’s story will not be revised because of increased willpower, friendlier teach- ers, benefactors from abroad, more textbooks, better lighting and more latrines— though all such factors can and do play a role in learning and schooling. Only through a revision of expectations and concomitant accountability with multiple stakeholders will Aminata’s updated story become a reality.