Quality of testing and transparency

SLC and HSEB examination data are not analyzed to establish test validity and reliability. In small-scale studies, item analysis, validity, and reliability used to be analyzed post- examination, not preexamination. As stated earlier in this report, there were various shortcomings and limitations in the post-examination analysis. ERO has undertaken item analysis and pretesting to test validity and reliability, which other public assessment institutions should also practice.

OCE uses different sets of test papers in the six compulsory subjects for five different regions in Nepal for SLC students who appear at the same time for the same examination. These sets of test papers are supposed to be parallel or equivalent, and OCE claims that these test papers as parallel as well. Again, the claims for the equivalency of test papers are

38 _{Office of Controller of Examinations. 2013. Piloting of Achievement Test Based on Objective Test Items as per the}

based on perception and judgment, not established through item analysis. There is also the issue of comparability of test scores from year to year—differences might be due to the level of difficulty of the test.

Items are developed as a set in the form of test paper or group of items by units or chapters and kept in the bank for later use. As stated earlier in this section, developed items are not calibrated, analyzed, or revised for the final form in OCE and HSEB. A process for item calibration, analysis, and revision has been adopted by ERO in the NASA (footnote 16). In the NASA, a large number of items are prepared, pretested, and after item analysis banked for later use. Moreover, ERO has also adopted item response modeling in test development, item analysis, and result interpretation. This can be a lesson for others. The NASA and other public examinations are limited to students who are enrolled in school. Because children and youths not in school are not covered in the test, the proportion of out-of-school children and youth and their learning levels have not been determined. However, Department of Education annual data provide net enrollment, gross enrollment, level transition rate, dropout rate, and such other information. Promotion rate or pass percentage of students at different grades or levels are reported by the Department of Education in its publications such as flash reports, status reports, and consolidated reports; by OCE in its annual publication, School Leaving Certificate Examination Statistics; and by the MOE in its publications, Nepal Education in Figures: At-a- Glance.39 _{The level of disaggregation of promotion data varies in these documents (gender,}

region, school, etc.); otherwise, reporting is limited mainly to promotion status. SLC and HSEB examination results are not analyzed with a view to support classroom teaching and learning or curriculum improvement.

Examinees have a right to question the scoring of their answer sheet. This right is

guaranteed through appeal. In Nepalese education, currently only re-totaling is done, and reevaluation of an answer sheet is not entertained. Provision of reevaluation would increase transparency and would also be valuable in improving the technical aspects of the test.

4. Good Practices and opportunities

Some of the good practices and further improvements required in the present practices are shown in Table A2.5.

Weaknesses in the present assessment system that need to be improved include the following:

(i) Comparability—sets of test papers; year-to-year and subject-to-subject variations need to be explained and comparability maintained (item response theory can be helpful).

(ii) Result analysis—analysis of test results needs to be done with the view to provide suggestions to improve curriculum as well as teaching and learning practices.

39 _{Ministry of Education. 2012. Nepal Education in Figures: At-a-Glance 2012. Kathmandu; Ministry of Education.}

Appendix 2

143

(iii) Noncognitive skills—need to be emphasized in the curriculum as well as properly assessed.

(iv) Certification practices—move from arbitrary 101 scale marking to single subject certification and letter grading.

The endeavors to improve the assessment system in Nepalese education should also reflect upon the possible barriers. The main probable barriers are as follows:

(i) Delays in passing of amended Education Act has delayed some of the initiatives and created confusion.

(ii) An examination-oriented education system and public aspirations negate some of the quality improvement endeavors such as the continuous assessment system. Without a pass/fail, it is not perceived as an assessment.

(iii) Frequent transfer of staff and loss of institutional memory is problematic. For example, technical units were established by different projects in OCE and HSEB, but these do not exist now.

(iv) Although the technical aspects of test development and marking are used, steps (such as in scrutiny, marking scheme, etc.) are not followed properly.

(v) Conceptual clarity is lacking in a number of aspects (such as grace mark, cut-off score, composite pass, pass/fail practice, etc.).

(vi) “Assessment of learning” is still the dominant concept.

table A2.5: nepal—Good Practices and opportunities for improvement in Assessment of Student Learning outcomes

initiated Further improvements required

Emphasis on school-based assessment and continuous assessment system (assessment of learning)

Linking with students’ learning support Test development based on specification grid

(mainly external) Teacher and school also base test development on curriculum and grid Attempt to include higher-level items (mainly

external exam) Classroom questions, school, and resource center level tests also need to pay attention to this. External exams can also be improved. Noncognitive skills also to be assessed. Attempts to increase transparency (re-totaling,

examining answer sheet) Reevaluation should be entertained on the appeal of the examinee Item banking (sets or items by units) Items to be calibrated, analyzed, and revised

and then final form kept in the bank Access to the teacher and school for the sample items

Test administration—established procedure More rigorous efforts and procedures to address problem of cheating

Test result publication—minimized time,

multiple sources for publication Error minimizationTest result analysis, reporting, and utilization

These barriers need to be addressed properly to improve the quality of education in general and of assessment practices in particular.

There are also a number of opportunities to improve assessment practices and the education system as a whole, such as the following:

(i) Recent reform initiatives such as teacher professional development training, the continuous assessment system, Child Friendly School National Framework, and the Information and Communication Technology Master Plan have emphasized improvement of quality of education.

(ii) Along with other aspects of education, the National Curriculum Framework and the SSRP have provided guidelines on curriculum and assessment. Some of them have already been implemented.

(iii) Quality of education and quality of student learning have been major concerns by the public, the media, and the government. Major aspects of this discussion are fairness and quality of examination.

(iv) Work is under way to establish a national examination board.

d. recommendations and Future directions