Surveys were administered to all members, faculty and fellows, of the division of

(1)

Supplemental Material Needs Assessment

Surveys were administered to all members, faculty and fellows, of the division of Pediatric Critical Care Medicine (PCCM). Survey questions asked participants their opinions regarding the ability of the current assessment tool, based on the Pediatric ACGME-C&M, to assess a fellow’s medical knowledge, performance of clinical and administrative duties, decision making, and attitudes. Questions also explored the tool’s clarity of language, length, and

educational impact. Survey results were used to inform subsequent focus group and individual discussions held separately. These sessions served to further elucidate current practices and processes for completing the ACGME C&M-based assessment tool, barriers to timely

completion of the tool, and perceptions of the impact of delays in completion. In addition, questions pertaining to personal interpretations or understanding of the ACGME C&M and the general meaningfulness or usefulness of fellow assessments were posed.

The results of the surveys and focus groups identified several challenges and

weaknesses to address with the new assessment tool. Faculty felt that the assessment tool’s length was a key deterrent to timely completion. While the survey results demonstrated positive perceptions of faculty’s and fellow’s understanding of the current response scales, the focus group discussions with faculty identified the ACGME-C&M language in the current

assessment tool as an obstacle. The verbiage led faculty to feel they had to read each milestone anchor every time they assessed a fellow to ensure they knew what each question was

assessing. In addition, each anchor compressed multiple abilities with which a fellow might have variable competency leaving faculty confused about which milestone level to choose.

Downloaded from http://meridian.allenpress.com/jgme/article-supplement/421139/pdf/jgmed18010061 by guest on 22 August 2021

(2)

Within the focus groups, faculty identified that these language problems contributed to a delay in completion and decreased faculty’s willingness to spend time on narrative comments. Also, there was a strong sense that the needs of the regulatory bodies and the needs of the fellows to ensure learning or performance improvement were not shared and, at worst, maligned despite the neutral outcome on the surveys. Finally, fellows felt that the poor interrater reliability and lack of narrative comments explaining faculty’s ratings impaired accurate identification of plans for improvement which was consistent with the survey findings. Tool Development

The Clinical Competency Committee (CCC) met three times to identify the assessment tool goals, framework, and response scales. The CCC discussions led to the identification of three goals for the new assessment tool. First, the tool should be developed using educational theory, acknowledging and mitigating limitations whenever possible, and focusing on feasibility and acceptability as compliance was identified as the largest challenge to providing meaningful information. Second, the tool must assess important workplace-based decisions and

professional activities of PCCM within the unique context of the specialty and, specifically, the institution. Finally, the tool must allow for formative assessment, benchmarking, tracking, and development of individualized learning plans and goals if viewed separately by individual fellows or if viewed by the Program Director (PD) or Associated Program Director (APD). In total, ten EPAs have been identified for PCCM. The first five EPAs are for general pediatric training across the continuum of residency through fellowship with the next two being common to all pediatric subspecialties. The remaining three – “acute management of the critically ill patient, including those with underlying chronic disease”; “manage and coordinate

(3)

care in pediatric critical care units for optimal patient outcomes”; and “management of patients at the end of life” – serve as the PCCM-specific EPAs.

We conducted a 90-minute faculty development workshop to help faculty recognize common strategies for assessing fellows – global rating scales, competencies, milestones, and EPAs – and identify the strengths and weaknesses of each strategy. Within this session, faculty were introduced to the scale associated with EPAs and the plan to implement an EPA-based assessment using the PCCM EPA “acute management of the critically ill patient, including those with underlying chronic disease”. Initially, faculty were asked to comply with making an

entrustment decision based on the EPA as written. This request was met with great concern that the EPA was too broad and would not be useful or meaningful as an assessment tool. To address this concern, a list of sub-EPAs was developed by the CCC using the American Board of Pediatrics PCCM subspecialty examination content, faculty (>5 years of clinical experience) opinion, PCCM textbooks, and frequency of admission diagnosis in the medical-surgical

intensive care unit. Again, this list was introduced to the faculty where additions, subtractions, and modifications were made but there was still concern about the need for more granularity, mostly based on task acuity, before faculty would agree to use the tool. In the final iteration, Observable Practice Activities (OPAs), based on more granular patient management tasks within a sub-EPA, were developed. The sub-EPAs with their OPAs were reviewed by the faculty wherein minor additions, subtractions, and modifications were made. At this point, agreement was reached by all faculty to utilize the novel EPA-OPA assessment tool.

Defining the response scale for the novel assessment tool began with the CCC discussing the fellow behaviors that comprised each entrustment level. This created a culturally-specific

(4)

shared mental model of entrustment definitions. The response scale and definitions were brought to the faculty for discussion and modification. The end result was the response scale that follows:

1. Observation Only = lowest level of entrustment, faculty member makes all decisions with fellow only observing or enacting decisions directed by the faculty member

2. Direct Supervision = faculty and fellow discuss every decision prior to execution or makes only minor decisions (such as increasing/decreasing sedation infusions) and then immediately discusses decision with faculty member after execution

3. Indirect Supervision = faculty is present in the unit but fellow is making decisions and then discussing with faculty immediately after execution [faculty note: if you are standing with the fellow during patient care, you are allowing them to make all the decisions or conduct all patient care without your input, you are present only to observe and then debrief/discuss later] 4. Unsupervised = faculty is not present in the unit while fellow is making

decisions and discussion only occurs when faculty returns to the unit

5. Supervises Others = fellow functions at Level 4 and is allowed to direct others as a direct supervisor (described in Level 2)

Tool Implementation including Faculty and Fellow Development

After identifying the frameworks for tool development, a resources-needed evaluation was conducted and identified the following needs. A system by which the assessment tool could be disseminated, collected, and compiled for analysis was required. That resource would also need to be easily accessible in the clinical environment, provide shared responsibility for completion between the faculty and fellows, and not require a large time commitment from faculty to access and complete. Financial implications were identified including cost of the tool development, implementation, and dissemination as well as personnel and facility costs.

(5)

A password protected SurveyMonkey tool was created because of ease of use and familiarity with this program. SurveyMonkey can be accessed via a computer or cellular device, and a link to the divisionally provided fellow cell phones was added. An independent license ($1020/year) for which only the PD, APD, and program administrator had access was purchased to ensure security. Several faculty and fellows tested the links and time required to complete the assessment tool and found the SurveyMonkey tool very acceptable. No other personnel or facility costs were identified.

Two identical 90-minute faculty sessions, held during regularly scheduled meetings, introduced faculty to needs assessment data and final assessment tool layout. Faculty were allowed an opportunity for hands-on practice with the tools as well as another opportunity to ask questions, voice any concerns, and offer any suggestions. Two sessions were held in order to capture the majority of faculty. Fellows were provided similar information in a separate session during a mandatory meeting. Any fellows not present because of duty hour limitations were provided the information on a one-on-one basis. To supplement these sessions, emails were sent once a month for the first 4 months of implementation. These emails contained a brief recapitulation of the purpose and expectations regarding the new assessment tool as well as step-by-step directions for completion. As time went on, commonly identified

questions or mistakes were included. Each email also contained a word document attachment for later reference.