Part 4: Closing the Loop Initiatives
4.3 Capsim Simulation for Learning Goal: Functional Integration
Question: What is CAPSIM?
Answer: CAPSIM business simulations engage participants in a dynamic competition to turn struggling companies into successful, profitable businesses. Classes are divided into teams that work together to focus on strategy, finance, production, marking and all the key elements that interact to build their
business. Instructors focus on directing the learning experience towards defined learning goals. Delivered online, in the classroom, or a combination of both, delivered in condensed or expanded time frames, CAPSIM programs have the flexibility to adapt to any academic curriculum. The CAPSIM formula for participant achievement, minimal preparation and easy adaptability has proved successful at more than 500 business schools. CAPSIM consists of an in course simulation that culminates in an exam known as Comp-XM. Data from the simulation and Comp-XM are then used to measure specific learning goals as part of the college’s Assurance of Learning program.
Question: What does CAPSIM measure?
Answer: CAPSIM has 14 educational goals that they can be selected for measurement. These goals can be fine-tuned based on an educational institution’s specific Assurance of Learning program. CAPSIM’s goals were created based on input from professors using Comp-XM as part of their schools ongoing assessment process for accreditations purposes, and were designed in accordance with guidelines set forth by the Association to Advanced Collegiate Schools of Business (AACSB).
Question: How are faculty engaged throughout the process?
Answer: Faculty are engaged throughout the entire process, from identifying which learning goals to measure, customizing the simulations and the exams, as well as, customizing the reports that are produced upon completion of the simulations and Comp-XM.
Question: What are some B-Schools using AACSB for Accreditation?
Answer: Examples of participating schools include the following:
Kennesaw State University/ Brigham Young University/ Indiana University / Syracuse University/ University of Massachusetts / Woodbury University / University of
LaVerne/ University of Texas at Arlington/ University of Toledo/ Villanova University/ York College
Spring 2013 Pilot – Background
The idea of using Capsim originated after performing significant research regarding methods implemented by other universities to (1) improve student learning and (2) to use as an assessment tool to collect more accurate data. In particular, Kennesaw State University’s The Coles College of Business was highlighted in AACSB’s March 2012 Spotlight publication for its strong assessment program and use of simulations. Assisted by one of the Assessment leads from the college, Dr. Hope Baker, shared valuable insight with CBE staff into the success Kennesaw has experienced using Capsim simulations and testing. This resulted in a draft proposal to the CBE Dean for approval of a Capsim pilot in Spring 2013.
The proposal was approved and the Capsim simulation was implemented in every section of MGMT 4650, the undergraduate Capstone, a total of eight sections, and one MBA capstone course, MGMT 6800. Six faculty members and two HIRE Center staff members were involved in the implementation of the pilot. All fees were waived for all participating students, over 300 undergraduate students and approximately 20 graduate students. Costs of the simulation and exam were paid for by the college.
The simulation required a level of HIRE Center staff support to provide assistance to faculty and students. HIRE Center staff took the lead on monitoring and assessing the pilot for effectiveness in terms of use as an education tool and use as an assessment tool. This was assessed through meetings held with participating faculty, participating students, attending a Capsim annual seminar, and speaking with key Capsim
representatives at the Capsim headquarters in Chicago, Illinois.
The Spring 2013 simulation was determined a success in the two areas it was pilot testing: (1) use as an educational tool and (2) use as an assessment tool. Faculty found students improved in areas of functional integration, data-driven decision analysis, teamwork and communication. Assessment staff found the simulation provided more accurate and transparent data to measure student proficiency in key areas. As a result, the CBE Dean made the decision to continue supporting Capsim for Fall 2013 in all sections of the MGMT 4650 course, with the plan to continue in Winter and Spring 2014. A re-evaluation of the simulation program will follow at the end of the first full academic year. Fees will again be waived for all participating students and the costs will be covered by the college in Fall 2014. Lessons learned from feedback from participating faculty and students have provided a number of suggestions for improvement moving forward.
The following suggestions were implemented for the Fall 2013 implementation:
• Having a Capsim representative conduct a seminar for faculty in Summer 2013 to prepare faculty for the simulation as an Instructor.
• Developing an in-class introductory tutorial for students, to be administered by trained HIRE Center staff at the beginning of the quarter in each participating course section.
• Providing students with computer lab support throughout the quarter.
• Providing students with more in-person support throughout the quarter, to be provided by HIRE Center staff.
To better assist faculty to full integrate the simulation into the capstone course, the CBE invited a Capsim representative to come to campus and hold a two-day seminar for the faculty who are teaching, or who normally teach, the capstone course on September 16 & 17, 2013. The seminar was successful and helped faculty better integrate the simulation into their course. HIRE Center staff also attended the seminar to gain the knowledge and understanding in order to be a source of support for students throughout the academic year. The agenda of the Fall Capsim Seminar for Management Faculty can be found in Table 4.1 below.
Table 4.1
The following three sub-sections review the feedback from (1) faculty, (2) students, and (3) Capsim professionals in respect to the pilot program that was executed in Spring 2013.
Spring 2013 Pilot: Student Feedback
Issues Faced:
• Note enough time to learn simulation – early weeks in the quarter wasted learning user interface. • Students lost at the beginning.
• Capsim “Coach Tool” ineffective. • Student guide book ineffective.
• User manual is inappropriate for first time users, not user-friendly.
• Strategy development occurred later on in the quarter because time was spend learning how to use the simulation.
• Software can be intimidating.
• Mechanisms where hard to learn quickly and independently.
• Concord campus has more of a commuter environment than the Hayward campus and was difficult to meet with teammates.
Benefits Experienced:
• Simulation taught how to work as a team, learned about how important oversight is in running a company with multiple departments.
• Allowed students to look at different aspects of a company. • “Interesting”
• Learned how making decisions in one area affected another – learned how to make decisions in order and what kind of impact that has on the success of a company.
• Learned importance of having a specialty in business. • Received experience on how to run a real business.
• Learned how important skills such as teamwork and communication were in running a business. • Simulation was “dynamic” and was in “real-time.”
Advice for the Future:
• Youtube videos were found to be more helpful. • More help with learning about the balance scorecard.
• Students suggested better communication with students when Capsim has a technical issue (i.e. server is down).
• Shorten the learning curve.
• Allow for open/reserved lab sessions.
• Allow for feedback opportunities on strategies throughout the simulation from the instructor. • More time allotted between rounds.
• Better tutorials on how user interface works. • More video trainings.
• External websites were found to be more helpful.
• Structure where one person enters data for a group (minimizes confusion).
• Submitting online tickets for help was not useful, calling Capsim directly for help was found to be more useful.
• Allow in-class time to work in groups – beneficial to Concord campus, which is seen as a commuter campus.
• Offer ways to meet virtually.
• More help with devising strategies early on.
• Add a “Round Zero” that is not graded to help students get to know the software and ask questions. • Allow in-class time to reflect on and analyze decisions made in each round to help understand the
Spring 2013 Pilot: Faculty Feedback
Issues Faced:
• Guidebooks were not very helpful
• Students need support throughout simulation
• Students were lost at the beginning and faced a steep learning curve • Need better integration with existing syllabus
Benefits Experienced:
• Improved student quantitative skills
• Gave students exposure to real-world business conditions • Students learned about making decisions
Advice for the Future:
• More in-house support for students
• Scheduling 30-minute session in class with Capsim representative over the phone for students to openly ask questions
• Email system was not useful
• Create ‘CEOs’ for students to select group members (aka ‘Company Employees’)
o Install HR element that Capsim was strongly missing
o Allow students to practice creating resumes and conducting interviews
• Capsim should supplement existing teaching, not replace it.
• Need more faculty support at the front-end and student support throughout the simulation • Provide computer lab support
• Combine capsim better with existing class so that not teaching two separate classes • Faculty noticed positive effects of using simulation in the classroom as a teaching tool • Capsim representative was very helpful
• Faculty posed the question, ‘Maybe MGMT 4650 needs to focus more on forecasting’
Programmatic Changes Brainstormed:
• Make Capsim a stand-alone simulation course
o Making Capsim a stand-alone class and bridge with capstone
i.e. MGMT 4650-A: Strategy Class MGMT 4650-B: Capsim • Extend Capsim over the course of two quarters/one semester
The following pages consist of meeting and seminar debriefings. Meetings between HIRE Center staff and Capsim representatives were held to better understand how Capsim can help address the BSBA learning goal: functional integration (among additional learning objectives such as, communication, teamwork and data analysis). In addition, HIRE Center staff also attended an out-of-state seminar held by Capsim for those wanting to better understand the simulation program and how it can best be integrated into their business program.
Assessment Meeting Summary May 17 & 21, 2013 Meeting Attendees: Sandy Luong and Ron Duran (Capsim)
Prior to Meeting: Sandy Luong emailed Ron Duran documents regarding Program Learning Goals and Learning Objectives for BSBA.
Key Topics Discussed:
(1) How to use data gathered from CAPSIM simulation and Comp-XM exam for assessment.
Process for Translating Raw Simulation/Exam Data into Assessment Data:
1. Each learning goal is analyzed via the simulation and/or Comp-XM exam (see minutes Table 1) 2. Download raw scores into a csv file and export file
3. Identify random sample of students assessed (i.e. 4 students from each section of MGMT 4650) 4. From the selected students’ data, pull questions identified as part of group of questions measuring
particular objective (see minutes Table 1)
5. Calculate raw percentage scores and overall percentage scores
6. Translate calculations into four categories for assessment (0 = Not Meeting Expectations, 1 = Needs Improvement, 2 = Meets Expectations, 3 = Exceeds Expectations)
7. Set college benchmark for students, i.e. 3 based on assessment scoring, or 70% or higher on raw percentage score.
8. Discussed different ways of calculating raw data.
a. Walk through of various analyses conducted from exported raw data files. 9. Discussed different ways of presenting data.
Table 4.2
BSBA Learning Goal Capsim Module Simulation Comp-XM Questions
1A. Foundation Knowledge Effective Business Mgmt Total Balanced Scorecard
H-319, F-347, O-351, M-354, F279, A362, F-115, F-290, H-323, H-329, F-358, A-364, F-41, A-94, F- 301, H-324, P-258, F-260, A-286, M-308, A-320,
H-326
1B. Business Integration Critical Thinking Total Balanced Scorecard
A-362, H-319, M-334, M-346, F-347, M-349, O- 351, M-354, M-355, S-74, F-267, F-274, F-279, S-
73, S-363, A-389, S-75, F-115, F-290, P-309, O- 322, H-329, F-358, A-364, A-16, F-41, S-76, A-94,
F-301, O-322, H-324, P-328, P-258, F-260, F- 261, A-286, M-308, A-320, H-326, P-373 2A. Quantitative Methods & Tools Quantitative Learning Skills Process and Customer ServiceFinancial, Internal Business N/A
2B. Technology for Data Driven Decisions Innovation & Technology Total Balanced Scorecard M-349, O-351, M-355, S-74, S-363, F-115, P-309, H-323, H-329, F-358, A-364, A-16, S-76
3A. Oral Communication -- Stockholder Debriefing Presentations N/A
3B. Written Communication Course - Embedded Assignment N/A N/A
3C. Teamwork -- Peer Evaluations N/A
4A. Ethics Course - Embedded Assignment future embedded in simulation N/A (Available module in the
Capsim Simulation User Seminar Debriefing - Full June 6-7, 2013 – Chicago, Illinois
Sandy Luong, HIRE Center
The two-day Capsim seminar focused on two main objectives. Day One consisted of placing attendees through hands-on exercises, allowing them to understand the simulation through the eyes of the students. Day Two consisted of talks regarding how Capsim products can be used as a teaching tool and for
assessment purposes.
Day One of the seminar started with talks given by Capsim experts regarding an overview of the simulation, introductory topics include how the simulation developed, and how it transitioned from industry (for corporate training) into academia as an educational tool, particularly for undergraduate and graduate business students. Day One continued with hands-on exercises putting attendees through the simulation through the eyes of the students.
Subsequent topics covered on Day One include:
• Capsim from the view of the Professor • Available teaching resources
• Getting the student introduced to the simulation • Common student pitfalls
• Introduction to additional plugins (HR and TQM)
The hands-on walk through of the Capstone simulation consisted of:
• Attendees being placed in groups of four to five members and given a student capstone account to access.
o The simulation consists of a total of eight rounds, represented in the simulation as eight
years of a company’s life. The seminar exercise completed three complete rounds. In each round, a number of decisions were made in multiple areas of a company, such as R&D, Marketing, Production Operations and Finance. Decisions were also made by each team in areas of Human Resources and TQM.
• A discussion following each round of the data results for each group.
o During each round, teams experienced:
(1)The kinds of decisions students needed to make. (2)The common issues arise in each round.
(3)What kinds of technical/logistical problems students may face when it comes to working with the simulations user interface.
• An open discussion at the end of each round allowed for a deeper understanding of:
(1)The different approaches instructors can adopt to review the results with the students. (2)The different teaching styles instructors have adopted when reviewing data.
(3)The common things instructors have emphasized and highlighted when looking at the pages of data results.
(4)How to use the data reviews as a way to integrate conceptual teachings and find ways to insert more in-depth conceptual teaching.
• Day 1 concluded with talks around best practice suggestions for instructors (i.e. focus on a particular theme for each round), and common mistakes by instructors (i.e. overwhelming students with data up front, focus on individual pages at a time rather than presenting students with the full report up front).
Day Two began with a look at different ways students can conduct a ‘Shareholder Debriefing’ on their individual companies and the different approaches Instructors have used to facilitate the Debriefings. Other topics covered include: (1) how to embed the simulation in a course and (2) the different ways instructors have combined the simulation with their own course material and design.
The second half of Day 2 began with talks regarding the growing trend of using simulations as an assessment tool for accreditation purposes, particularly highlighted was for AACSB Accreditation in particular. Speaking as the resident expert on AACSB was the Comp-XM co-creator and product development consultant, Dr. Erich C. Dierdorff from DePaul University’s College of Business.
Topics covered on Day 2 include:
• A background on assessment for accreditation purposes, assurance of learning, and particular emphasis on AACSB expectations.
• Why Capsim’s products (Foundation/Capstone simulations and Comp-XM exam) can be used as an assessment tool, in addition to being used as an educational tool.
• How Capsim has moved into the direction of addressing more and more needs requested by Business schools to incorporate more and more assessment aspects to their products. • A background on the research studies conducted by Capsim and academic experts to:
(1)Identify the most commonly recognized learning goals according to between 50-60 business schools. Over 300 learning goals were aggregated, categorized and analyzed in one particular study conducted through collaboration between Capsim and DePaul University. Seven learning goals were identified in this study, called “The Big 7.” The Big 7 learning goals were cross-referenced with the learning goals of CBE. All learning goals currently held by CBE were listed as part of the Big 7.
(2)Ensure accurate knowledge content was used in measuring the identified learning goals in point # 1 above.
(3)Ensure appropriate “items” were targeted for when compiling data for assessment. • Jordan Novak, Capsim Client Engagement Consultant and the co-creator of Capsim’s assessment
support tools gave a more detailed talk regarding data aggregation. Topics include:
(1) How data is pulled from both the simulation and the exam to address identified learning goals.
(2) Capsim’s capabilities regarding what it can and cannot do for business schools to assist them in satisfying their assurance of learning needs. (Note: Capsim does not claim to be able to satisfy all assurance of learning needs, especially in relation to accreditation. What Capsim does state, is their abilities as a service provider to assist in accreditation efforts.
• Capsim concluded Day 2 of the seminar with a look into Capsim’s upcoming products currently in beta testing. These products have been particularly designed with assurance of learning needs, particularly for accreditation of business schools, in mind. Capsim presented two products in testing:
(1)A module to address the learning goal Teamwork in a comprehensive way that consists of strong data evidence.
(2)A module to address the learning goal Ethics in a way that would allow for situational analysis for assessing a student’s ability to make ethical decisions. This module is not ready to be used for assessment. However, after explaining CBE’s current efforts to revise assessment of CBE’s Ethics learning goal, Dr. Dierdorff discussed his currently on-going research surrounding assessment of the learning goal Ethics and offered to share his unpublished research with Sandy Luong for CBE faculty to consult when looking at the Ethics learning goal again in Fall 2013. Sandy Luong to follow up with an email regarding discussed research.
In a private meeting requested and scheduled prior to the start of the seminar, Sandy Luong was given the opportunity to discuss assessment specifically for the CBE. Meeting held was with co-creators of Capsim’s assessment support tools, Dr Dierdorff , Professor of Management at DePaul University, and Capsim resident expert on assurance of learning, and Jordan Novak, Capsim Client Engagement Consultant and