Program Evaluation: Onboarding Program Kaitlin Gibbons
The University of Tampa EME 640
Request For Proposal
A mid-sized southern university is looking to develop and implement an onboarding program for their new staff members. The program would be implemented on a per person basis and would last for one year from the day they were hired. The purpose of the program is to help new employees get started and settled into their new roll on the university’s campus. The
program will include an online resource page for new hires on the campus learning management site Blackboard, as well as pamphlets, flyers, and mentoring program with a member in their office. The overall goal of the program is to welcome new hires onto the campus and acclimate them to the new environment and policies/procedures on campus. The program also moves the responsibility of acclimating the individuals away from the human resource department and towards the individual departments on campus. The desired outcome is for the new hires to feel comfortable in their position and become a productive member in their office, and ultimately lead to higher retention rates on the campus.
Overall evaluation plan
The human resources department on the campus and a graduate student assistant will conduct the overall evaluation plan for the onboarding program at the University. Three types of evaluation will take place during the development and implementation of the program: formative evaluation, summative evaluation, and confirmative evaluation. The total evaluation will take approximately 18 months to complete. This will allow for the development and testing of materials and program (formative evaluation phase), evaluation of the complete program (summative evaluation phase), and a follow up confirmation that the program was effective months after completion (confirmative evaluation phase). The evaluation will begin in the spring and last one full year, followed by a confirmative evaluation four months after.
The evaluation of the program is to ensure that the new onboarding program is successful and providing the right type of environment and information to new hires of staff at the
university. The outcomes of the evaluation will determine whether or not to expand the program to new faculty members, not just staff, and if the program should continue. The evaluation will be composed of different data collection tools and assessment, which are:
Subject Matter Expert (SME)- Formative evaluation • Interview with individual
One-to-One Evaluation-Formative evaluation • Attitude acceptance survey (20 questions) • Interview for additional information
Attitude acceptance survey –Summative evaluation • 20 question survey about program
Posttest survey-Summative evaluation
• 20 question posttest about policies, procedures and campus items. Attitude acceptance survey-Confirmative
Formative Evaluation Timeline for Formative Evaluation
April 1-15 Recruit SME for evaluation
April 1-15 Develop tools for SME evaluation
April 1-10 Recruit four to six individuals for one-to-one
evaluation
April 1-15 Develop tools for One-to-one Evaluation
April 15 Schedule SME Meeting for first week of May
April 15 Schedule One-to-one Volunteers for the first
two weeks of May
May 1-8 Conduct SME Interviews on the Evaluation
May 1-15 Conduct One-to-one evaluations
May 8-22 Analyze data collected from SME
May 15-30 Analyze data from One-to-one
May 21- June 4 Make and finalize changes to the program
Expert Review
There will be two expert reviews performed on the material for the onboarding process. The first expert will be an individual in the human resources department from another university of the same or similar size and hierarchy that have implemented an onboarding process for its new staff members. This individual will be able to provide insight into what segments of the process were successful for their institution as well as locate any that seem vague or not needed. An email correspondence will be sent to a number of universities with an onboarding program and solicit a volunteer to review the program. The email will include a background about the university as well as the goals and purpose of the onboarding program. Unfortunately due to budget needs, the individual will not be compensated for their time.
The program plan and materials developed will be sent via email to the individual for a total of two weeks. At the end of the two weeks, an interview meeting will be scheduled to discuss their thoughts on the materials. The meeting will be conducted over the phone or via Skype or Google Hangout. The interview will include closed and open-ended questions about the materials. The individual will also be asked to supply any additional information that they feel
would be appropriate for the program. A sample of the interview protocol can be seen in table 1 below.
For the second expert review, a department head with in the university will be asked to review the materials. The reason for choosing a department head is that an individual in their position will be responsible for the new hires and implementing the process with in their department. This individual has also been faced with acclimating a new individual to their department and their department policies and will be able to provide insight into how a
department is organized and how effective a program like this will be. An email will be sent out to all department heads on campus asking for a volunteer to evaluate the program and materials. The email will contain information about the program and what will be asked of them.
Similar to the first expert review, this individual will be sent the program plan and materials via email. After two weeks, an interview meeting will be set up. For this meeting face to face is preferred since they will be on the campus. However because scheduling conflicts can occur, phone or video calls through Google Hangouts or Skype are an option. The same
interview protocol will be used in the interview with this individual. Table 1
Sample of Interview Protocol 1. How comprehensive do you feel the materials
were? Answer:
2. What items on the supervisor checklist would
you change? Answer:
3. Were there any items listed in the procedure that
you did not feel were necessary? Answer:
4. Were their items not listed in the procedure that
you feel would be beneficial? Answer:
5. Would you add any information to Blackboard
site? Answer:
One-to-One Evaluation
The second type of formative evaluation to be used will be a series of one-to-one
at the university in the last six months to a year. These individuals would not have gone through an onboarding process and will have insight into what items in the plan would have been useful for them becoming familiar with the university and their various departments.
The human resources department will be utilized to provide the campus emails of those individuals hired in the last 6 months to a year, and an email asking for volunteers will be developed and sent out. A total of four to six individuals are needed for this portion of the evaluation. The individuals will need to be able to dedicate either one three-hour session or three one-hour sessions to view the materials (the former is preferred). Upon the day of the evaluation, the individual will be brought into the conference room located in the human resource
department. The room will be equipped with a laptop so the volunteer can evaluate the
Blackboard site for the onboarding program. The volunteers will be asked to take notes on the materials and on things that they believe will be beneficial to add. The designer will be available in the room to answer any questions or concerns that the volunteer may have about the material. After all the materials have been reviewed, a short attitude acceptance survey will be given to the individuals to complete. The survey will include both Likert style and open ended questions and contain 20 questions. A sample of the survey can be seen in Table 2. They will also be
interviewed by the designer and asked to share any additional thoughts about the program. Table 2
Survey Sample Questions 1. Would you have benefited from a program like
this when you were hired?
Strongly Agree, Agree, Disagree, Strongly Disagree
2. I would have felt more comfortable in my position with a program like this
Strongly Agree, Agree, Disagree, Strongly Disagree
3. I have participated in an onboarding program in
the past Strongly Agree, Agree, Disagree, Strongly Disagree
4. I found the Blackboard Site useful and informative
Strongly Agree, Agree, Disagree, Strongly Disagree
5. What would you like to see added or changed to the materials for the program?
The revision suggestions and additional information given by both the SME and the one-to-one evaluations along with the surveys from the one-one-to-one will be taken into account and added to the final program materials and procedures. The goal is to refine the materials and program to ensure that it is the best possible product for its audience.
Summative Evaluation
Timeline for Summative Evaluation-Year After Implementation
May 1-15 Develop attitude acceptance survey and
posttest
May 1-31 Recruit new staff member for evaluation
June 1-30 Send out surveys and posttest to individuals
who volunteered and were hired in June of the previous year
June 1-30 Collect surveys and posttest
June 1-30 Recruit new staff member for evaluation
July 1-31 Analyze data from survey and posttest from
June participants
July 1-31 Send out surveys and posttest to individuals
who volunteered and were hired in June of the previous year
July 1-31 Collect surveys and Posttest
July 1-31 Analyze data from survey and posttest from
July participants
August 1-4 Results from both months synthetized
August 1-15 Adjustments to the program are made from
survey results
Upon completing the onboarding program for the year, all new staff members who were hired in the months of June and July will be asked to participate in the evaluation of the program. Ideally there will be ten total participants for this group. One year after they were brought on as staff, the individuals will be sent an attitude acceptance survey about their time in the onboarding program, and a brief posttest about various policies, procedures, and location of things on
is, as stated earlier, to make the individuals feel welcomed and be knowledge about their new job and work environment.
The attitude and acceptance survey will show if the individuals feel a part of the
university community, their feelings toward the program, and if they felt it made a difference in their attitude toward their job. The survey will have similar questions as the ones that were asked in the one-to-one evaluation. Positive results are ideal for continuing this program, however if a trend develops in one area of negative result it could mean training is needed with supervisors in the onboarding program. Sample of the survey questions can be seen in Table 3.
Table 3
Sample of Survey Questions 1. The onboarding program made me feel
more comfortable in my job
Strongly Agree, Agree, Disagree, Strongly Disagree
2. The information on the Blackboard site was
easy to find Strongly Agree, Agree, Disagree, Strongly Disagree 3. I can locate all buildings on campus Strongly Agree, Agree, Disagree, Strongly
Disagree 4. I feel that I understand my position better
because of the program
Strongly Agree, Agree, Disagree, Strongly Disagree
5. What is one thing you would change about the program?
The posttest that will be distributed will be to see if they know the employee rules and regulations for the university and where things are located that they would need to know. This posttest will have no penalty for them for their participation; it will be used to evaluate the knowledge being shared with them. Low results from this posttest can show that aspects of the onboarding not dealing with acceptance are not being covered with the individual when they start their position. A sample of the posttest can be seen in Table 4.
Table 4
Sample of Posttest Questions 1. What are the two locations to find food on
campus?
Answer: 2. Where can I find my Payroll information Answer: 3. How much notice do I need to give for
Vacation? Answer:
4. What is our schedule in the summer? Answer: 5. What are the hours of operation during the
academic year?
Answer:
Confirmative Evaluation
The confirmative evaluation will take place four months after the onboarding program is completed. The evaluation will be conducted to ensure that the information provided during onboarding and process that the individual went through has helped them on their job now that they are no longer undergoing the process. The same people who participated in the summative evaluation will be contacted to participate in confirmative evaluation. They will be asked to fill out another attitude and acceptance survey with Likert style and open-ended question to see how their experience has been since completing the program, and if their attitudes have changed since the first survey. It will include questions similar to the formative and summative attitude and acceptance surveys but will also include questions on how effective they feel they are at their position. A sample of this survey can be viewed in Table 5. Results from this survey will be analyzed against the previous summative survey. It is from here that a decision will be made on whether to keep the program or discontinue it.
Table 5
Sample of Confirmative Survey Questions 1. I am more likely to continue in this job
because of the Onboarding Program?
Strongly Agree, Agree, Disagree, Strongly Disagree
2. I still feel confident on how to do my job correctly
Strongly Agree, Agree, Disagree, Strongly Disagree
3. I feel a part of my office community Strongly Agree, Agree, Disagree, Strongly Disagree
4. I feel comfortable to still approach my supervisor with questions
Strongly Agree, Agree, Disagree, Strongly Disagree
5. What would you change about the program after completing it six months ago?
