© 2007 WACRA®. All rights reserved ISSN 1554-7752
USING THE RESULTS FROM A COMPUTER SIMULATION TO
EVALUATE MBA PROGRAM OUTCOMES
Paul O’Neill, John Kerrigan, and Doug Schreder National-Louis University
CHICAGO, ILLINOIS, U.S.A.
Abstract
This paper will discuss how to use the results from a computer simulation to assess MBA program outcomes. It is part of an ongoing effort by the College of Management and Business at National-Louis University to assess the effectiveness of its MBA program. Specifically, this paper will discuss how we evaluated 130 students by evaluating their results within 10 major performance categories in the Capstone Business Simulation. Data from this study will be used to improve the MBA program. The goal of this paper is to provide ideas and methods and for other users of simulations and case studies to engage in similar evaluative and curriculum improvement efforts.
As part of this evaluation/ assessment process, the authors conducted an earlier study to determine the major concepts necessary for the effective performance in the Capstone Business Simulation. Information was gathered from students and course instructors. The results of this study were presented at the 2004 WACRA conference in Buenos Aires and the 2005 conference in Brno, Czech Republic. The assessment presented in this paper takes a broader look at the MBA program: scores of simulation teams are examined and compared to major performance categories as defined by the simulation developers.
KEY WORDS: Strategic management, MBA, evaluation, assessment, business simulation
INTRODUCTION
The College of Management and Business, National Louis University, developed an MBA program in 1999. The program’s design is based upon NLU’s experience in using adult learning principles that appeal to and meet the needs of adult students. Since its inception, the program has undergone considerable evaluation and revision as detailed in previous WACRA presentations including a report presented at the 2005 WACRA conference in Brno, Czech Republic [Weis and Schreder, 2005]. The program currently consists of twelve courses taught in a cohort model in both face-to-face and online versions and with available concentrations in areas of finance, marketing, and general management studies. One of the key focuses of the program is the hands-on application of learning. For this reason, a business simulation, Capstone (by Management Simulations of Northfield, Illinois), was selected for the program. The simulation is web-based and is designed to be an integrative experience, combining the application of key MBA program concepts with strategic and operational decision-making. Classroom teams compete by running rival businesses for five to six sessions during the last course of the program, Strategic Management. Students compete with other student teams as well as computer teams.
The use of the Capstone simulation has been found to be both beneficial and challenging to students. Student teams are required to apply previously learned business concepts from courses as disparate as
Economics, Finance, Organizational Behavior, and Human Resources. Consequently, student success in the simulation is considered to be a strong indicator of student success in the overall MBA program. To ensure that there is the best-fit possible between the MBA program and simulation content, the authors are leading an evaluative and developmental effort to systematically examine both the simulation and program content. The results of the last two years efforts have been presented at the last two WACRA conferences [Kerrigan and O’Neill, 2006] [O’Neill and Kerrigan, 2005].
Added to this effort has been a University and North Central Association initiative to better measure program outcomes. Our MBA program has always used indirect measures of program outcomes from end-of-program surveys of students. To strengthen our assessment, we decided to add a direct measure of student performance. This is a measure derived from actual student performance in vehicles such as exams, papers, or projects, directly observed, and evaluated by a set of specific criteria [Walvoord, 2003]. An excellent means of accomplishing direct program measurement is to “ask students to demonstrate their learning…in a senior capstone course where students will be expected to show they have mastered the tools of research appropriate to their discipline” [Bantra, 2005, p.37]. After consulting with Management Simulations, the authors felt there was an opportunity to conduct an assessment study using the scores of simulation student and computer teams as provided in the Capstone Analyst Report.
The simulation’s major performance categories (Appendix 1), that also reflect the MBA program goals, would be examined. Team performance would be viewed as reflective of overall program performance. The effectiveness of team performance would be determined by examining overall team performance in comparison to simulated computer team performance. From the analysis, recommendations would be developed for program and course improvement. A sample Analyst Report of one of the class competitions studied in this report is provided as an example and includes the of performance of three student teams (“Andrew”, “Baldwin”, And “Chester”) and one computer team (“Digby”).
The MBA Program Team, in its ongoing review and revision of the program, will continue to use the results of this and previous studies to improve the curriculum. The methodologies developed and used by the authors are explained in the next section. The research steps being used for this assessment are described below.
RESEARCH STEPS AND METHODOLOGY
The following steps are being followed in conducting the assessment of the Capstone Simulation and MBA program outcomes.
1. Collect team scores from most recent simulations
2. Compare and evaluate overall scores of student teams to computer teams 3. Compare performance category scores of student teams and computer teams 4. Develop conclusions on overall performance of student teams
5. Identify performance areas needing improvement
6. Make suggestions for improvement in teaching of performance areas.
RESULTS
The following is a brief discussion of results and actions taken based on the above six steps: 1. Collect team scores from most recent simulations
• Results were collected from 10 MBA class/groups comprised in total of 32 student teams and 21 computer teams from simulation competitions during the 2003-5 timeframe. 2. Compare and evaluate overall scores of student teams to computer teams
• As shown in Appendix 3, the student teams overall scored 77% as well as the computer teams. Two student teams beat the overall average score of computer teams.
3. Compare performance category scores of student teams and computer teams
• As shown in Appendix 3, the student team scores ranged from 60% to 88% of the average computer team in individual performance categories. At least one student team was able to beat the computer team in each of the 10 performance categories. The strongest student performances were in the areas of forecasting, avoidance of emergency loans, customer satisfaction, and wealth creation. The weakest student
performances were in the areas of working capital, market share, profits, and financial structure.
• As expected, given the preprogrammed aspect of Capstone computer team simulation decisions, the standard deviations or variances of student team scores for the various components as well as for overall points in the simulation was considerably more than for computer teams (e.g. more than double or 766 vs. 319 in the case of total points).
4. Develop conclusions on overall performance of student teams
• While the overall student scores were lower than computer teams by 23%, it is possible for student teams to beat the computer scores. In fact, the Sample Analyst Report (Appendix 2) shows two student teams (Andrews and Baldwin) in Round 1 (as well as cumulative scores after Round 6) of the simulation outscoring the computer team (Digby). • Considering that our students compiled these overall scores in a 6-week course,
compared to most school’s 10-14 week courses, we feel that overall, students demonstrated acceptable performance.
• Within the individual performance categories, there was substantial variation (60% to 88%) in the student scores among the 10 categories. This suggests that by targeting and improving the lower scoring categories, we could see substantial improvement in the overall student scores.
• The considerable variance of student teams in scoring compared to computer teams could indicate substantial differences in simulation preparation, MBA program learning, and/or inherent abilities of students in various student groups.
5. Identify performance areas needing improvement
• We believe there are two areas in which we can help students improve their Capstone scores. First, we can target improvement in the MBA courses covering material relating to categories where student team scores were weaker (e.g. working capital, market share, profits, and financial structure). Second, we can improve the structure and the instruction in our final Strategic Management course so that the students can better integrate their knowledge from previous courses and better apply that knowledge in the simulation. 6. Make suggestions for improvement in teaching of performance areas
• We have several suggestions for improving student performance and scoring in the simulation. First, we need to follow through on our plans to have a one-session Capstone workshop in advance of our final Strategic Management course. This workshop will get students more familiar with the logistics of the simulation so that they will be able to spend more of their time integrating and applying concepts from all their previous courses into the simulation. Second, we will advise all future Strategic Management/Capstone instructors to use the weekly Analyst Reports to give feedback and guidance to student teams. Third, we need to continually review Analyst Report results of future MBA Strategic Management courses using the Capstone Simulation so that instructors will be able to continue to strengthen the weaker areas as highlighted in this paper.
FUTURE RESEARCH
The authors suggest that the results of this study be combined with the information gathered in the previous studies. Additional methods for examining and improving direct student outcomes will also need to be explored. The analysis of these data can lead to a comprehensive action plan for improving the curriculum of the MBA program. Once these improvements are in place, the study described in this paper can be replicated in order to determine the extent of the improvement in the student outcomes.
REFERENCES
Bantra, T. “How Much Have We Learned”. BizEd (September/October 2005).
Kerrigan, J. and P. O’Neill. “Aligning an Integrative Business Simulation and an MBA Curriculum – An Ongoing Process.” International Journal of Case Method Research and Application,” (2005) XVII, 2.
O’Neill, P. and J. Kerrigan. “Improving the Alignment Between an Integrative Business Simulation and a changing MBA Curriculum – Part 2.” International Journal of Case Method Research and Application, (2005) XVII, 3.
Walvoord, B. E. “Assessment in Accelerated Learning Programs: A Practical Guide.” New Directions for Adult and Continuing Education, No. 97, (Spring 2003).
Weiss, E. and D. Schreder. “The National-Louis University Experience Revisited Again – Revision of the MBA Program.” WACRA conference presentation, Mendel University, Brno, Czech Republic, July, 2005.
APPENDIX 1
CAPSTONE PERFORMANCE CATEGORIES
Margin: Margin points are earned in three areas: Contribution Margin Percentage, Net Margin Percentage, and ROS or Return on Sales.
Profits: The Profit category examines the rate at which wealth is being created. Where margins look at percentages, this category examines the actual value of the profit.
Emergency Loans: When you run out of cash, you have "a liquidity crisis". Emergency loans are closely linked to your working capital policy and forecasting abilities.
Working Capital: The Working Capital category examines your reserves. Working Capital equals Current Assets minus Current Liabilities.
Market Share: This category examines the extent to which your team has exceeded the market share of the other teams.
Forecasting: The Forecasting category examines your ability to forecast demand, build adequate inventories to satisfy demand, and yet not accumulate excessive inventory.
Customer Satisfaction: The Customer Satisfaction category examines your performance from the customer's perspective. Each of your products can earn points if it meets three criteria: It must sell 50 thousand units during the year, it cannot stock out, and its Customer Survey score must be 30 or more. A product’s December Customer Survey Score is developed using measures based on marketing’s "4 P’s": Product, Price, Promotion, and Place.
Productivity: The Productivity category examines the productivity of your workforce through the course of the simulation, and includes Sales/Employee, Profit/Employee, and Turnover rate.
Financial Structure: The Financial Structure category examines the Financial Structure of your company — its relationship between Debt and Equity.
Wealth Creation: Wealth Creation examines three measures: Cumulative Profits, Cumulative Free Cash Flow, and Market Capitalization
APPENDIX 2
SAMPLE ANALYST REPORT Analyst Report 2011- SimID C10267, Part I: Point Summary
Select a previous round: 1
Go
LEGEND: Points = 0, = 10, = 20, = 30, = 40, = 50, = 60, = 70, = 80, = 90, = 100
Team Andrews Baldwin Chester Digby
Margins Profits Emergency Loans Working Capital Market Share Forecasting Cust Satisfaction Productivity Financial Structure Wealth Creation Total Points 594 521 354 463 Cumulative Points 4218 3785 2775 3121
APPENDIX 3
STUDENT TEAMS VERSUS COMPUTER TEAMS
# Courses/Classes 10
# Total students 130
# Total Student teams 32
# Students/team - average 4.06
# Total Computer teams 21
Performance Category: Margin Profits Emer.
Loans Working Cap. Mkt. Share For- casting Cust. Sat. Produc-tivity Finan. Struc. Wealth Creation Total Points
Average score - class teams 207 328 511 317 236 174 42 374 392 434 3016
Highest score - class Teams1 389 572 600 500 600 395 230 574 550 600 4284
Lowest score - class teams 52 8 250 100 0 20 0 183 100 71 905
Standard deviation - class teams 84 156 92 104 164 85 50 90 97 139 766
Average score - computer teams 272 481 598 469 397 199 62 447 507 504 3937
Highest score - computer teams1 441 598 600 600 575 299 189 547 600 600 4461
Lowest score - computer teams 128 300 550 300 100 60 0 322 350 386 3352
Standard deviation - computer teams 94 88 11 87 147 57 53 64 81 64 319
Student average to computer average
scores - percentile 76% 68% 85% 68% 60% 88% 68% 84% 77% 86% 77%
# Student teams that beat or equaled
average score of all computer teams 6 7 11 2 3 11 9 7 1 10 2
1
