Learning Tool's Guide

The CDC (2013) has developed an evidence based E-Learning website to assist in development and implementation of an electronic program. The analysis of content is through instructional analysis, learning objectives, interactivity, interface and navigation, and general content. The evaluation process includes formative and summative evaluation, a common educational evaluation process. Formative evaluation is a method of evaluation which evaluates the product before it is launched and then trouble-shoots the product. Summative evaluation evaluates outcomes.

Instructional Analysis

Rationale for utilizing this method and the focus of this analysis included why, when, who, content, goals, and audience responsibilities. The program had a flexible design without

mandatory responsibilities for the audience. The why of this format was threefold: distances of the potential audience, time frame of the project with time input control for the administrator, and the testing if an electronic vehicle would be useful and used. All these conditions were met. The when for the project was capstone dependent and was met. The DNP student's goal for the project was to provide a learning framework with information, an arena for collaboration, and educational material to improve skills. The provisional goal was met. The lack of response, however, prevented effective evaluation of the site. The who for this project was purposefully professional. Invitations were extended to all members of the Massachusetts Pediatric Palliative Care network through their executive directors, primary care providers of Notre Dame Pedi Pal patients, Boston Children's Hospital PACT Team, UMASS Pediatric Oncology Team, Shelly's Place staff, and the Music Therapy Department at Anna Maria College and the Child Life Specialist Program Directors at Wheelock College. Only one member outside of the five in Notre Dame Pedi Pals successfully entered and two others successfully applied to PhConnect.org and were accepted as members, but did not fulfill the next step of requesting membership to the site forum even with additional invitations. One entrance point would have facilitated some increase in membership. Content focus was provided to assist different professional audiences by the DNP student. Lack of response to the evaluation tool prevented the DNP student from knowing its efficacy.

One of the initial caveats was to possibly provide CEU's for the ELNEC program. This, however, was not possible as one must provide this training in person to facilitate answering of questions. Motivation for attendance and usage of this site was purely at one's own impetus.

Interactive presentations were not possible to do on this site, and the DNP student had no personal skill in the usage or formation of Wiki's or platform building.

If this project was to continue, finding a more interactive site would be ideal.

Contracting an Information Technologist to assist with platform development which would allow for interactive participation in real-time would improve the site. Creating an easy to use format, would increase active participation resulting in engaged and interactive learning. If an existing learning platform, such as Blackboard was available, this would be implemented. The voluntary nature of the designed pilot also did not assist with participation. Mandatory participation in the forum would be required in a future, similar project as one study (Gafni & Geri, 2010) showed this increased engagement. Providing free professional rewards, such as CEU's or similar

professional credits would be used for future projects to augment audience size and participation. Although colleges were contacted, such as Anna Maria College and Wheelock College, in any future project, the DNP student would inquire about student learning project needs, invite student collaboration, and model the practice during the project.

Learning Objectives

Learning objectives were utilized following the SMART model which stands for S: specific, M: measurable, A: action-oriented, R: reasonable, and T: time-bound. As the responses in the pilot were limited to the discussion question section, this area was the only section which could be measured and required specific responses. Measurement of responses was discussed in the qualitative data section. Action orientation for the learning was through the discussion section, responses to the DNP student's surveys, and the Journal Club site which became inactive due to lack of response. A true evaluation of reasonableness was not possible

with such low response rates. Most interactivity on the site was passive: reading the stories, the articles, watching the videos, and accessing the pages. Real-time interaction was only possible through scheduling meetings which did not occur as planned through the journal club.

Site Evaluation

Interface and navigation was difficult on this site. There was no easy way to attach articles or other items with usage of e-mail. The site was set up to foster "Community of Practice" meetings, so my usage of this as a "Network of Learning" was a work-around. Acceptance of the site limitations therefore was necessary.

There were limitations to the site pages as well. Site content evaluation of health literacy was as follows: the font size was small, graphics were difficult to attach and use, synopsis of articles was at a level of grade 11.7 and discussion questions were at grade 9.1 from Microsoft 2007 Flesh-Kincaid Readability. Future recommendations for any pilot or project of this type would be a grade eight reading level for all content. The EBP content was partially interactive and this content or similar would be used. None of the videos on the site were available in another language which could be used by the clinicians to help families.

Sister Karen Hokanson SND, Ed.D. was invited and accepted the DNP student's request to review the educational format and content of the site. Some of Sister Hokanson's (2013) comments were as follows: "Excellent learning goals with clear outcomes….It might be good to group power points by topic and use bit.ly website to shorten the links." The second comment helped to improve the ease of viewing on the site.

Formative evaluation was attempted by improving the look of the site during the pilot as much of the content was added weekly by design. Pre-pilot, the DNP candidate tested the

program with a friend during the formation of the site in order to check ease of entrance and her opinion as to the set up. Frequent requests for comments regarding the site were initiated. There were no responses to these. Summative evaluation was done at the conclusion of the pilot. Please see Appendix I for the content and survey responses. The DNP student's answers were the only response and must therefore be viewed cautiously.

Project Model Theory Evaluation

Wenger, et al., (2011) used a social value model to evaluate networks of learning. Evaluation included these indicators judging learning and were:

1.) immediate value

2.) potential value through knowledge capital 3.) applied value such as practice change

4.) realized value: performance improvement 5.) value reframing and redefining success.

Wenger's immediate value was evident in initial interest in joining the site which increased by 42% over the nine week pilot. Figure 4 in this text showed views and responses visible during the pilot's course. While responses in general were limited, those who responded seemed to gain some immediate benefit. The DNP student certainly gained the immediate value of cultivating the site, adding to the site, and continuing to invite members to the site. The site's educational expert noted, "I learned a lot from the information shared" (Personal communication, Sr. Karen Hokanson SND, Ed.D., 3/15/2013).

Wenger's potential value from knowledge capital was not measurable due to low response rates for any of the surveys. There was no way to track views other than in the discussion area; utilization of the videos and other resources could not be measured. The DNP student learned that knowledge capital can be a personal goal and measures capturing this were not known or explored at the pilot initiation. Motivation to learn over time was a challenge for all.

Wenger's applied value could not be evaluated due to low response rate. The DNP student certainly became more knowledgeable regarding resources, the diseases, and the interdisciplinary teammates. Certainly the view regarding palliation throughout the care cycle was a new personal incorporation for the DNP student.

Reframing of expectations of involvement per Wenger's model was implemented at week five; stressing discussion involvement and participating in the Mito 101 course was emphasized to all members. Other than the DNP student, no one else responded to the challenge and

reframing of expectations was unsuccessful. Realized value was not measurable due to limited responses.

One of the measurements which the DNP student anticipated as a reframing measurement indicator was the post-pilot survey. As there was only one responding participant from Notre Dame Pedi Pals, this could not be used. Lack of response again influenced measurable data. For this model theory, lack of response influenced four out of the five areas of measurement and the pilot was either a failure, read and not responded to, read and some behavioral changes in practice were implemented, and read and new insights regarding collaboration were gained in

one's practice. This DNP student cannot know the results.

Post-Project Plans

A goal of pilot testing this learning network was to facilitate collaboration across the Massachusetts Pediatric Palliative Care Network. Unfortunately, the limited response rate suggested that the project should not continue at this time. Only one person from the Network accessed the site and did not participate in any of the discussions. The site itself was geared toward discussions and committee meetings. The DNP student adapted its general usage to do the preliminary work of the "Network of Learning" Phase in preparation for implementing a "Community of Practice" Phase to problem solve collaboratively if that was the group wish. No group formulated in the nine weeks. In areas other than the discussion sections, there was no tally of visits. As there was only one response from the Survey Monkey, there was no means of measuring the effectiveness and usefulness of the various pages developed.

Pediatric Palliative Care is a very specialized field and the practitioners in this field are passionate about what they do. Opportunities for collegial networking is limited, particularly in the community setting. As the site is free, the DNP student plans to continue the site for another six months adding further material and continuing to invite members of PhConnect.org to join.

Post-Pilot, a thank you letter with an evaluation will be sent to all members who joined and to the three who joined PhConnect.org, but never requested membership to the site. See Appendix K for details of this letter and the evaluation contents. As there was so little traffic present in the site, its cost effectiveness was not proved and cannot be promoted for usage by the Massachusetts Department of Public Health's Pediatric Palliative Care Network. Cost for the

pilot was as follows in Table 5. Table 5.

Updated cost of the pilot program

______________________________________________________________________________