Competency-based training “Helping Mothers Survive: Bleeding after Birth” for providers from central and remote facilities in three countries

(1)

EDUCATION AND TRAINING

Competency-based training

“Helping Mothers Survive: Bleeding after Birth”

for providers from central and remote facilities in three countries

Cherrie L. Evans

a,

⁎

, Peter Johnson

b

, Eva Bazant

c

, Neeta Bhatnagar

d

, Jane Zgambo

e

, Asma R. Khamis

f

a

Jhpiego– An afﬁliate of Johns Hopkins University, Maternal and Newborn Health Unit, Baltimore, USA

b_Jhpiego_{– An afﬁliate of Johns Hopkins University, Global Learning Ofﬁce, Baltimore, USA} c

Jhpiego– An afﬁliate of Johns Hopkins University, Monitoring and Evaluation Unit, Baltimore, USA

d

Jhpiego– An afﬁliate of Johns Hopkins University, Clinical Services and Training Ofﬁce, Dehradun, India

e

Jhpiego– An afﬁliate of Johns Hopkins University, Family Planning and Reproductive Health Unit, Lilongwe, Malawi

f

Jhpiego– An afﬁliate of Johns Hopkins University, MNCH Department, Unguja, Zanzibar, Tanzania

a b s t r a c t

a r t i c l e i n f o

Article history: Received 1 October 2013

Received in revised form 19 February 2014 Accepted 12 April 2014

Keywords: Birth attendant Competency maintenance Continuing professional development Maternal mortality

Postpartum hemorrhage Simulation

Skilled birth attendant Training

Objective: To validate a new training module for skilled and semiskilled birth attendants authorized to provide care at birth—Helping Mothers Survive: Bleeding After Birth (HMS:BAB)—aimed at reducing postpartum hemor-rhage, the leading cause of maternal mortality worldwide. BAB training involves single-day, facility-based training that emphasizes simulation of scenarios related to prevention, detection, and management of postpar-tum hemorrhage. Methods: A total of 155 skilled and semiskilled birth attendants participated in training in India, Malawi, and Zanzibar, Tanzania. Knowledge and confidence were assessed before and after training. Skills and acceptability were assessed after training. Results: Knowledge and confidence scores improved significantly from pre- to post-training among all cadres in all three countries. The proportion of providers with passing knowledge scores increased significantly from pre- to post-training among all cadres except for those already high at baseline. On three post-training skills tests the overall proportion of individuals with a passing score ranged from 83% to 89%. Conclusion: BAB training in prevention and management of postpartum hemorrhage increased knowledge and confidence among skilled and semiskilled birth attendants. Further studies are needed to determine the impact of this training on skills retention and clinical outcomes following postpartum hemor-rhage, after broader implementation of the training program.

1. Introduction

Postpartum hemorrhage is the leading cause of maternal mortality worldwide, accounting for one-quarter of maternal deaths[1,2]. The global incidence of postpartum hemorrhage has been reported at 10.8%, with a range from 7.2% in Oceania to 25.7% in Africa[3]. Globally, more than 13 million of the 136 million women giving birth each year suffer from postpartum hemorrhage, primarily caused by uterine atony[4].

Active management of the third stage of labor (AMTSL) is critical for reducing the risk of postpartum hemorrhage and has been shown to de-crease postpartum hemorrhage by as much as two-thirds[5,6]. AMTSL consists of three interventions: administration of a uterotonic within one minute of birth, controlled cord traction during contractions to de-liver the placenta, and uterine massage once the placenta has dede-livered. Uterotonic use within one minute of birth is considered the key compo-nent in reducing postpartum hemorrhage[7].

Training of birth attendants in AMTSL is critical to ensure that all women receive high-quality care at birth[8]. Providers need to be trained to recognize and treat postpartum hemorrhage; and need a mechanism to maintain their competencies since the actual occurrence of life-threatening hemorrhage can be relatively rare for any single pro-vider. Simulation-based training uses a model for practice and improves acquisition of a variety of psychomotor, problem-solving, and commu-nication skills[9]. Low-dose, high-frequency simulated practice has been associated with greater skills retention and transfer to perfor-mance, and training that occurs in close proximity to the work site has been shown to result in greater retention of skills and changes in perfor-mance[10–12]. More recently, governments are requesting on-site training or training that does not take providers away from the work-place for an extended time. Additionally, it has been acknowledged that current approaches do not always reach the cadres managing the majority of births[13].

In conjunction with Laerdal Global Health, Jhpiego has designed a training series entitled Helping Mothers Survive (HMS). Theﬁrst training module in this series is Bleeding after Birth (BAB), which is pat-terned after the Helping Babies Breathe training in newborn resuscita-tion[14]. The BAB module had not previously been validated in the ⁎ Corresponding author at: Jhpiego, 1615 Thames Street, Baltimore, MD, 21231 USA.

Tel.: +1 410 537 1986; fax: +1 410 537 1476. E-mail address:[email protected](C.L. Evans).

http://dx.doi.org/10.1016/j.ijgo.2014.02.021

Contents lists available atScienceDirect

International Journal of Gynecology and Obstetrics

j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / i j g o

Open access under CC BY-NC-ND license.

(2)

ﬁeld, and its effects on participants knowledge and skills had not been characterized. The objectives of the present study were to validate the BAB training module, including methods, materials, and integration with simulators for ability to transfer knowledge and skills, and accept-ability to learners and facilitators, and to evaluate the psychometric properties of the learner assessment tools (construct validity using item difﬁculty and discrimination statistics from each of the knowledge and skills assessments).

2. Materials and methods

With guidance from Jhpiego country staff, 144 skilled and semi-skilled birth attendants in India, Malawi, and Zanzibar, Tanzania were purposively invited to participate in the study. The intervention coun-tries were selected based on their high burden of maternal mortality. The participants were clinical staff currently involved in bedside clinical practice in the delivery ward at peripheral and higher-level public facilities. Two guidelines were used to recruit providers: (1) providers who had attended births in the last 90 days; and (2) preference for a mix of providers from both remote and central levels of the health sys-tem. Although all participants were authorized to provide immediate postpartum care to women, two of the sites included providers who did not fully meet the international deﬁnition of a skilled birth atten-dant (SBA)[15].

The study design was a pre- and post-assessment of participants in BAB training in Haldwani and Rudrapur, India, in February 2012; in Lilongwe district in Malawi, in April 2012; and in Stone Town, Zanzibar, Tanzania in July 2012. The BAB module was designed to teach birth attendants to prevent postpartum hemorrhage using international standards and a simplified protocol for management of postpartum hemorrhage. The training consists of a single-day, facility-based training emphasizing simulation of scenarios relevant to prevention, detection, and management of postpartum hemorrhage. Materials include a graphicflipbook for training, job aid for supporting clinical decisions, and handbook for facilitation and learning. An economical apron-style simulator, consisting of an abdominal“skin” containing a uterus holding a fetal mannequin, postpartum size uterus, and blood tank, was used to simulate birth, normal bleeding, and postpartum hemorrhage. Twelve experienced maternal and newborn health trainers from the interven-tion countries were oriented to the BAB module, simulators, skills assessment standardization, and study procedures in a two-day session conducted by the principal investigator (CLE). In India, three groups were trained with 16, 16, and 21 participants on each day. In Malawi, 22 were trained on thefirst day and 23 on the second. In Zanzibar there were 27 participants on thefirst day and 21 on the second. During all sessions there were no more than six learners to each trainer during hands on practice.

The materials, methods, and assessment tools of BAB were reviewed by the International Federation of Gynecology and Obstetrics (FIGO), the International Confederation of Midwives, the United Nations Population Fund, the World Health Organization, the Maternal and Child Health Integrated Program, the American College of Obstetricians and Gynecologists, the American College of Nurse-Midwives, and the American Academy of Pediatrics. The training module and accompa-nying knowledge and skills assessment were_{ﬁnalized and approved} on October 21, 2011, at a meeting in Washington, D.C., that involved representatives from the respective organizations.

The present study focused on two levels of the Kirkpatrick model of evaluation of training: participant reaction and participant learning

[16]. We measured change in four aspects of learning: BAB relevant knowledge, self-reported confidence on five clinical skills, actual clinical skills post-training, and acceptability of training materials and methods. Trainers were both participants and data collectors. Their participant role was to provide feedback on the module from a trainer perspective. Knowledge and confidence were assessed using two pre- and post-training oral questionnaires: 26 items for knowledge andfive items

for conﬁdence. These were administered orally by trainers in the local language to eliminate bias due to variations in literacy among providers. Participants rated their conﬁdence in providing AMTSL, treating post-partum hemorrhage, accessing advanced care, assessing the placenta, and bimanual uterine compression.

Skills were assessed using three Objective Structured Clinical Exam-inations (OSCEs). The three OSCEs assessed prevention of postpartum hemorrhage, management of retained placenta, and management of severe hemorrhage from uterine atony. OSCEs were conducted using simulators immediately after training. A criterion reference pass score was developed for all assessments using the Angoff procedure, with input from global organizations that reviewed the module[17].

Acceptability of materials, methods, and speciﬁc aspects of BAB was assessed by participant evaluation questionnaires using aﬁve-point Likert type scale (1 =“disagree” to 5 = “agree”).

The Johns Hopkins Bloomberg School of Public Health Institutional Review Board (IRB) approved the protocol for this study. The study was approved by the Ministries of Health of India, Malawi, and Zanzibar and was considered exempt from review by their respective IRBs. Verbal informed consent was obtained from all participants at the start of the training event.

We based our sample on the plan to invite on 48 participants per country. Therefore, the sample size of 144 gave 85% power to detect an improvement from 35% to at least 50% in the proportion of partici-pants who could perform AMTSL and identify signs and manage post-partum hemorrhage[8]. Continuous data were described with means and standard deviations, and categorical data were described as per-centages. For comparison of pre- and post-training knowledge, skills, and conﬁdence, paired t tests were used for continuous variables and the McNemar test was used for binary outcomes.

To validate the instrument, each of the knowledge assessment items was subsequently analyzed for item difficulty and discrimination. An item difficulty index is a ratio of the number of individuals correctly answering an item to the total number attempting to answer the item. Results range from 0.00, when all individuals answer incorrectly, to 1.00, when all individuals answer correctly. Typically, items with a dif fi-culty index of less than 0.50 are discarded. An item discrimination index is a point biserial correlation comparing performance on a given item to overall performance on the assessment[18]. Discrimination indexes range from−1.00 to +1.00[18]. The underlying assumption is that if performance on item response is not correlated with overall perfor-mance on the assessment, the item is defective and should be revised or discarded. All analyses were conducted using Stata 12 (StataCorp, College Station, TX, USA) and SAS version 9.1.3 (SAS Institute, Cary, NC, USA) with a type 2 error of 0.05.

3. Results

We invited 144 individuals to participate; however, 11 additional providers participated in the training for a total of 155. Characteristics of the study participants are shown inTable 1. In Malawi, cadres consisted of more skilled birth attendants, compared with India and Zanzibar. The proportion of participants who performed AMTSL was rel-atively higher in Malawi and India compared with Zanzibar. Overall, a majority (57%) of trainees had delivered between 1 and 44 deliveries in the last 90 days and a sizable proportion (40%) had attended 45 or more deliveries. Although BAB was meant for birth attendants in active practice, overall 13% had not conducted a birth in the last 90 days.

The mean percentage of correct answers on the knowledge assess-ment increased significantly from pre- to post-training among all cadres and in all three countries (Table 2). The proportion of participants who passed the knowledge test after training compared to before increased significantly overall, with the largest increase in the mean score and passing rate among auxiliary nurse midwives. For the 26 knowl-edge assessment items, difficulties ranged from 0.57 to 0.98. Thirteen items were categorized as low difficulty (0.8 to 1.0) and 13 items

(3)

were categorized as moderate difﬁculty (0.6 to 0.8). No items were deleted owing to inappropriate difﬁculty (b0.50). Eighteen items demonstrated desirable levels of discrimination (≥0.30). No items were deleted owing to negative discrimination.

The overall self-reported confidence scores increased significantly from before to after training among all cadres, in all countries, and re-gardless of the amount of AMTSL experience or number of deliveries in the previous 90 days (Table 3). Among thefive confidence items assessed, confidence on bimanual uterine compression increased the most from pre- to post-training, from 2.8 to 3.8; Pb 0.001 (not shown). Pass rates on OSCEs were similar for AMTSL, retained placenta, and postpartum hemorrhage (Table 4). Due to implementation challenges, the simulators did not arrive in Zanzibar in time for ap-propriate use during preparation for training. As a result, facilitators were not standardized in providing OSCEs, therefore the OSCEs were not conducted in a standardized manner for comparison with the other countries. Malawi trainees performed comparably to India trainees on AMTSL and retained placenta. For severe postpar-tum hemorrhage, trainees in India scored higher than trainees in Malawi. In India, AMTSL scores were lowest for auxiliary nurse mid-wives, higher for nurses, and highest for doctors. However, auxiliary nurse midwives outperformed nurses on retained placenta and for severe postpartum hemorrhage. Comparing nurses across countries,

performance was similar on AMTSL and severe postpartum hemor-rhage. However, nurses in Malawi had higher scores than those in India on retained placenta.

The acceptability of training materials and methods, including time allotted for training of one day, was high among all cadres and countries (4.0 or above out ofﬁve items). Ratings were highest for having enough trainers, use of the simulator as a teaching tool, and training with differ-ent provider types combined. Auxiliary nurse midwives gave the lowest ranking to acceptability of materials in English, at 3.1 compared with the group mean of 4.1.

4. Discussion

The present study showed that one-day simulation training in post-partum hemorrhage prevention and management increased knowledge and conﬁdence among skilled and semiskilled birth attendants in three countries. After training, over 90% of participants passed the knowledge test, and 83%–89% passed the skills assessments on AMTSL, retained pla-centa, and severe postpartum hemorrhage. The comparative perfor-mance of skilled and semiskilled workers was of particular interest. The ability of semiskilled workers to demonstrate skills comparable to their skilled colleagues post training suggests that greater access to this focused, simulation-based training for these cadres could give Table 1

Participant characteristics, overall and by country.

Variable Total Malawi India Zanzibar

No. (%) No. (%) No. (%) No. (%)

Cadre Health orderlies 22 (14) -- -- -- -- 22 (40)

Auxiliary nurse midwives 20 (13) -- -- 20 (38) --

--Nurse/nurse midwives 81 (52) 40 (85) 18 (34) 23 (42)

Clinical ofﬁcers, medical assistants, physicians 21 (14) 5 (11) 13 (20) 3 (5)

Cadre unrecorded 9 (6) 2 (4) -- (0) 7 (13)

Student 2 (1) -- -- 2 (4) --

--Total 155 (100) 47 (100) 53 (100) 48 (100)

AMTSL performed in last 90 days (No.) 0 40 (28) 10 (23) 9 (17) 21 (44)

1–10 42 (29) 13 (29) 17 (32) 12 (25)

≥11 63 (43) 21 (48) 27 (51) 15 (31)

Total 145a

(100) 44 (100) 53 (100) 48 (100)

Deliveries made personally in last 90 days (No.) 0 19 (13) 10 (22) 6 (12) 3 (6)

1–44 67 (57) 17 (38) 24 (46) 26 (55)

≥45 58 (40) 18 (40) 22 (42) 18 (39)

Total 144b

(100) 45 (100) 52 (100) 47 (100)

a

AMTSL experience was unrecorded for 10 participants.

b

Number of deliveries performed was unrecorded for 11 participants.

Table 2

Pre-test and post-test knowledge, by cadre and county.

Percentage of items responded correctlya _{Percentage of trainees who passed}b

No. trained

No. with testsc

Pre Post P value Pre Post P value

Mean ± SD Mean ± SD % (SD) % (SD)

Cadre

Health orderlies (Zanzibar only) 22 22 69 ± 11 77 ± 12 b0.001 41 (50) 68 (47) 0.07

Auxiliary nurse midwives ANM (India only) 20 20 62 ± 11 81 ± 11 b0.001 15 (36) 80 (41) b0.001

Nurses/nurse midwives (3 countries) 81 74 78 ± 9 91 ± 6 b0.001 84 (37) 99 (12) 0.003

Clinical ofﬁcers, medical assistants, and doctors (3 countries) 21 17 87 ± 6 96 ± 5 b0.001 100 (0) 100 (0) n/a

Cadre unrecorded (Zanzibar, Malawi) 9 4 82 ± 16 98 ± 2 b0.001 75 (50) 100 (0) 1.0

Students (India only) 2 2 75 ± 3 98 ± 3 b0.001 100 (0) 100 (0) n/ad

Total 155 139 75 ± 12 88 ± 10 b0.001 70 (46) 91 (28) b0.001 Country Malawi 53 42 83 ± 6 93 ± 5 b0.001 100 (0) 100 (0) n/a India 47 47 70 ± 12 87 ± 11 b0.001 45 (50) 89 (31) b0.001 Zanzibar 55 50 74 ± 12 85 ± 11 b0.001 67 (47) 87 (35) 0.006 a

Knowledge test had 26 items. P value from paired t test.

b

Pass threshold was 70%. P value from McNemar test for paired data on binary outcomes.

c

Of 155 trained, 16 were missing pre-test, post-test, or both.

d

(4)

them the competency needed to effectively prevent and manage post-partum hemorrhage.

BAB training resulted in significant gains in confidence among all birth attendants. Given the effect that provider self-confidence has on mediating the application of knowledge and skills, this may be the most important outcome of the training and should remain an impor-tant component of future BAB evaluations. This is particularly true for complex skills performed in emergency situations relatively infre-quently and under the greatest degree of stress. Using simulation and scenario practice, BAB training focused on building confidence to per-form bimanual compression of the uterus during severe postpartum hemorrhage. At the start of training, confidence levels were low for each provider cadre, but increased significantly after the training.

The participants found the BAB training methods highly acceptable, including length of the training, use of the simulators, length of time for practice, building conﬁdence in skills, and use of mixed cadres during training, as well as overall understandability and ease of use. However, auxiliary nurse midwives in India indicated that they would have pre-ferred the materials in the local language. This is not surprising, given their level of education and English literacy.

The training needs of those who have educational and administra-tive experience but who are less acadministra-tively engaged in clinical practice should also be considered, since 13% of participants reported not having delivered a baby in the last 90 days.

The study strengths include implementation in three countries (two in Sub-Saharan Africa and one in South Asia) and application to a wide array of birth attendants. BAB training relies heavily on graphical materials and is not primarily language dependent. Although training materials were in English, the training was done in Hindi, a mix of Chichewa and English, and Swahili.

The study has several limitations. The sample was purposively se-lected and does not necessarily represent all the providers in India, Malawi, or Zanzibar; however, the sample did include various cadres of providers from remote facilities as well as centrally located areas. Similarly, the results cannot necessarily be generalized to other coun-tries. In Malawi, both training days occurred at health facilities and sev-eral participants missed the pre-test or post-test because of labor ward responsibilities. The lack of OSCE skills assessment using a simulator be-fore training prevents us from understanding skills gains due to BAB training. Participants’ self-reported confidence data may have been in-fluenced by desire to please the trainers. This was the first field valida-tion of the BAB training, and this study did not have comparison groups. Historically, training in emergency obstetric care has been carried out in longer, centralized workshops aimed at improving a large num-ber of competencies at once in a single cadre of provider. BAB training is a departure from the large centralized workshops and is consistent with the need to reach a wider array of frontline health workers who are called upon to attend births and are not always reached by current strategies[13]. Validation of BAB was needed in order tofinalize the ma-terials and recommendations for implementation and to ensure that ef-fective and appropriate measures will be available for monitoring and evaluation when BAB training is scaled up in low-resource countries.

As BAB training is more widely implemented, it will be important to evaluate participant performance in the clinical setting and change in health outcomes (i.e. levels 3 and 4 of the Kirkpatrick model). Further work will be needed to monitor whether knowledge, skills, and con ﬁ-dence are sustained over time in relationship to the frequency and pat-tern of low-dose, high-frequency clinical skills practice[19]. To optimize this training, we advocate evaluation of the most effective schedule of this short, frequent practice following initial introduction of BAB and Table 3

Self-reported conﬁdence score, pre-training and post-training.a,b

Variable Pre-training Post-training P value

Cadre Health orderlies (Zanzibar only) 10.3 ± 4.3 18.7 ± 2.5 b0.001

Auxiliary nurse midwife (India only) 13.8 ± 2.8 19.4 ± 2.3 b0.001

Nurses/nurse midwives (all 3 countries)

17.5 ± 3.3 20.7 ± 2.3 b0.001

Clinical ofﬁcers, medical assistants, and doctors (all 3 countries)

18.0 ± 3.5 21.3 ± 2.5 b0.001

Country Malawi 17.8 ± 3.2 21.3 ± 2.4 b0.001

India 15.8 ± 3.8 19.3 ± 2.5 b0.001

Zanzibar 13.9 ± 5.3 19.9 ± 2.6 b0.001

Performed AMTSL in last 90 days (No.) 0 12.7 ± 4.6 19.4 ± 2.6 b0.001

1− 10 15.3 ± 4.2 20.3 ± 2.5 b0.001

≥11 18.2 ± 3.0 20.7 ± 2.5 b0.001

Deliveries made personally in last 90 days (No.) 1− 15 14.3 ± 3.8 19.6 ± 2.5 b0.001

16− 45 14.1 ± 5.0 19.7 ± 2.9 b0.001

≥46 18.2 ± 3.7 21.1 ± 2.2 b0.001

a

Values are given as mean ± SD unless otherwise indicated.

b _{An overall conﬁdence score was created from summing ﬁve questions about AMTSL, postpartum hemorrhage, accessing advanced care, assessing completeness of placenta, and}

bimanual uterine compression. Each question had aﬁve-point Likert-type response scale from 1 to 5. Total possible score is 25.

Table 4

Post-training objective structured clinical examination, by topic, country, and cadre.a

Two Countries Malawi India

Objective Structured Clinical Examination (pass threshold) Total Nurses and nurse midwives

Clinical ofﬁcers

Total Auxiliary nurse midwives

Nurses and nurse midwives

Doctors

n = 92 n = 44b _{n = 36} _{n = 4} _{n = 48}c _{n = 20} _{n = 17} _{n = 11}

AMTSL (7 out of 11 items in India; 9 out of 12 in Malawi 83 (38) 83 (37) 83 (38) 75 (50) 82 (39) 75 (44) 82 (39) 100 (0) Retained placenta (5 out of 7 items in both countries) 89 (31) 90 (28) 92 (28) 75 (50) 88 (33) 90 (31) 76 (43) 100 (0) Severe postpartum hemorrhage (8 out of 11 items in India;

8 out of 12 items in Malawi)

84 (36) 79 (42) 78 (42) 100 (0) 90 (30) 95 (22) 76 (43) 100 (0)

a

Values are given as percentages.

b

In Malawi, n = 47 were trained; n = 3 are missing on OSCE and/or cadre was unrecorded.

c

(5)

also challenges to implementation. Additionally, effectiveness or impact evaluations will be needed comparing competency, performance, and outcomes of providers who receive this training approach with providers who receive traditional training. Cost-effectiveness studies will be needed to show whether the same or higher levels of compe-tency and performance can be achieved with fewer_{ﬁnancial and} human resources, which would facilitate scale-up. Additional training modules in the Helping Mothers Survive training series are being planned to address additional causes of maternal mortality and these will require evaluation.

Acknowledgments

Laerdal Foundation provided funding for this validation study. Laerdal Foundation played no role in the conduct of the study, data analysis, or preparation of the manuscript.

Conﬂict of interest

The authors have no conﬂicts of interest. References

[1]Oyelese Y, Ananth CV. Postpartum hemorrhage: epidemiology, risk factors, and causes. Clin Obstet Gynecol 2010;53(1):147–56.

[2]Khan KS, Wojdyla D, Gulmezoglu M, Van Look P. WHO analysis of causes of maternal death: a systematic review. Lancet 2006;367(9516):1066–74.

[3]Calvert C, Thomas SL, Ronsmans C, Wagner DS, Adler AJ, Filippi V. Identifying regional variation in the prevalence of postpartum haemorrhage: a systematic review and meta-analysis. PLoS One 2012;7(7):e41114.

[4] World Health Organization, UNICEF, UNFPA, The World Bank. Trends in ma-ternal mortality: 1990 to 2008.http://whqlibdoc.who.int/publications/2010/ 9789241500265_eng.pdf. Published 2010.

[5]Leduc D, Senikas V, Lalonde AB, Ballerman C, Biringer A, Delaney M, et al. Active management of the third stage of labour: prevention and treatment of postpartum hemorrhage. J Obstet Gynaecol Can 2009;31(10):980–93.

[6]Begley CM, Gyte GM, Devane D, McGuire W, Weeks A. Active versus expectant management for women in the third stage of labour. Cochrane Database Syst Rev 2011;11:CD007412.

[7] World Health Organization. WHO recommendations for the prevention and treatment of postpartum haemorrhage.http://apps.who.int/iris/bitstream/10665/ 75411/1/9789241548502_eng.pdf. Published 2012.

[8]Althabe F, Buekens P, Bergel E, Belizán JM, Campbell MK, Moss N, et al. A behavioral intervention to improve obstetrical care. N Engl J Med 2008;358(18):1929–40.

[9]Robertson B, Schumacher L, Gosman G, Kanfer R, Kelley M, DeVita M. Simulation-based crisis team training for multidisciplinary obstetric providers. Simul Healthc 2009;4(2):77–83.

[10]Sturm LP, Windsor JA, Cosman PH, Cregan P, Hewett PJ, Maddern GJ. A systematic review of skills transfer after surgical simulation training. Ann Surg 2008;248(2): 166–79.

[11]Harder BN. Use of simulation in teaching and learning in health sciences: a system-atic review. J Nurs Educ 2010;49(1):23–8.

[12]Coomarasamy A, Khan KS. What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review. BMJ 2004;329(473):1017.

[13]Stanton C, Armbruster D, Knight R, Ariawan I, Gbangbade S, Getachew A, et al. Use of active management of the third stage of labour in seven developing countries. Bull World Health Organ 2009;87(3):207–15.

[14]Goudar SS, Somanavar MS, Clark R, Lockyer JM, Revankar AP, Fidler HM, et al. Still-birth and newborn mortality in India after helping babies breathe training. Pediatrics 2013;131(2):e344–52.

[15] World Health Organization, Department of Reproductive Health and Research. Making pregnancy safer: the critical role of the skilled attendant: a joint statement by WHO, ICM and FIGO.http://whqlibdoc.who.int/publications/2004/9241591692. pdf. Published 2004.

[16]Smidt A, Balandin S, Sigafoos J, Reed VA. The Kirkpatrick model: a useful tool for evaluating training outcomes. J Intellect Dev Disabil 2009;34(3):266–74.

[17]Howley LD. Performance assessment in medical education: where we’ve been and where we’re going. Eval Health Prof 2004;27(3):285–303.

[18]Case SM, Swanson DB. Constructing Written Test Questions: For the Basic and Clinical Sciences. Philadelphia: National Board of Medical Examiners; 2001.

[19]Ersdal HL, Vossius C, Bayo E, Mduma E, Perlman J, Lippert A, et al. A one-day "Helping Babies Breathe" course improves simulated performance but not clinical management of neonates. Resuscitation 2013;84(10):1422–7.