Action-Oriented Methodology - A Program Evaluation: Examining the Effect of a Residential Busin

The primary goal of this study was to conduct a research and program evaluation of a residential, first-year business learning community to compare program alumni’s academic achievement and university engagement to that of non-learning community students. For the purposes of this study, program evaluation will be defined as “a study designed and conducted to assist some audience to assess an object’s merit and worth” (Stufflebeam, 2000, p. 35). The CIPP program evaluation was used to frame this study’s mixed-methods design. Within the CIPP Model’s framework, the researcher focused on two evaluation types: process and product. When considering the process portion of the evaluation, the researcher sought to clarify the strengths and weaknesses of the Business Learning Community (BLC) in an effort to assist the Research Site in strengthening the program. When considering the product portion of the evaluation, the researcher sought to evaluate if the BLC’s objectives were being achieved. By utilizing the CIPP

evaluation model, the researcher sought to determine if the BLC was being implemented with fidelity to the learning goals and working (Stufflebeam, 1972). The following central research and evaluation questions framed the study:

1. How does participation in a first-year business learning community at an urban research university contribute to the academic success and university engagement of students in their third year of undergraduate study?

o Product: To what extent do students perceive academic achievement, retention, and engagement to be attributable to their participation in the business learning community?

o Product: To what extent have third-year student outcomes been impacted by their participation or non-participation in the learning community? o Product: To what extent do students who participated in a university’s

freshman learning community have significantly different rates of retention and engagement than their respective stand-

alone counterparts two years following learning community participation? o Process: How do student perceptions reported on survey data indicate that participation in a freshman learning community experience was inherently more or less satisfying than what students had experienced in stand-alone courses?

2. To what extent does participation in a freshman learning community result in the same or better grades for cohort students than for those in respective stand-alone course comparison groups two years following learning community participation?

o Product: Is there a significant relationship between students’ academic achievement (GPA) in high school and their participation or non- participation in a freshman learning community?

3. How was the first-year business learning community implemented as designed to achieve intended and unintended outcomes?

o Process: How are the components of the learning community currently being implemented at the Research Site’s business school?

This chapter focuses on the study’s methodology and instruments. The chapter addresses the following areas: site and population, research design and rationale, research methods, and ethical considerations.

Site and Population Population Description

The study population was comprised of third-year business students at a private urban research university. All students in the population were studying towards a Bachelor of Science degree in either business or business and engineering. Within those disciplines at the BCLU, students select from the following majors: accounting,

economics, marketing, entrepreneurship, international business, business law, general business, or operations management. The students in the study ranged in age from 19 to 22, having joined the university in the fall of 2012 as freshman, two years prior to their participation in the study. Based on demographics of the 2012 BCLU, approximately 20 percent of the study participants were international students. For the purposes of this study, international students were defined as those who were born in and attended high school in a country outside the United States. Participants in the study were either enrolled in a learning community during their first year of study at the university or not. All learning community participants in the study were enrolled in multiple courses together throughout their first year of study at the university.

The population of learning community and non-learning community students consisted of approximately 50 participants in each group. In order to participate in the study, students were required to have completed two years of coursework at the

study took place at the university following a ten-week academic term, as opposed to a fifteen-week term. Students at the university are required to complete a minimum of 180 credit hours to earn an undergraduate degree. With this credit threshold in mind, students who have completed between 70.5 credits and 96 credits have gained pre-junior status. To be eligible for junior status, students must have completed between 96.5 and 129.5 credits (see Table 1). In addition, students participating in the study must have been enrolled in the five-year undergraduate program, as opposed to a four-year program. Study participants entered the university as traditional, first-term freshman majoring in either business or business and engineering and had not transferred from another university or college at the Research Site.

Table 1

Credits Completed

Credits Completed Per Academic Year

Freshman 0 – 39.5 Credits

Sophomore 40 - 70

Pre-Junior 70.5 - 96

Junior 96.5 – 129.5

Senior 130+

All students participating in the study had completed a list of prerequisite courses within the BCLU; this list is drawn from the BCLU’s published plan of study. All students in the BCLU who are enrolled in the Bachelor of Science in Business

program follow a similar plan of study for the first two years of their undergraduate career (see Table 2).

Table 2

Requisite Courses

Required Courses to Participate in Study

Foundations of Business I Financial Accounting Foundations Foundations of Business II Managerial Accounting Foundations Principles of Macroeconomics Introduction to Business Statistics Principles of Microeconomics Business Law I

The Drexel Experience International Business

All students enrolled in the BCLU are enrolled in two gateway courses during their first ten-week term, Foundations of Business I and The Drexel Experience. Foundations of Business I and the follow-up course in the sequence, Foundations of Business II, require multiple team projects and group-work activities. During both courses in the sequence, students work in teams both inside and outside class. Each of the required courses listed have identical learning outcomes for learning community and non-learning community students and, with the exception of the courses listed in Table 3, learning community and non-learning community students take courses together.

A final requirement for participation in the study was centered on the learning community students. Students in the learning community must have lived within the learning community for the entire duration of their first year on the same mixed-gender floor in the same residence hall.

Table 3

Learning Community Courses

Cohorted Learning Community Courses

Foundations of Business I Principles of Microeconomics

Foundations of Business II Principles of Macroeconomics

The Drexel Experience

Site Description

The site for the study was a private urban research university approximately one mile from the center of a major metropolitan city. The university has approximately 15,000 undergraduate students, with about 3,000 of those students in the business college. Students in the population were categorized as pre-junior or junior students within the BCLU. According to university records, 3,013 first-time, full-time, bachelor’s degree- seeking students enrolled as freshman in the fall of 2012 semester. Of those students, 2,605 (86.5%) returned to the university the following fall semester. This rate of retention represented an increase from 84.5% for the 2011 cohort. The 2012 cohort of first-year, full-time, bachelor’s degree-seeking students at the university was made up of approximately 60% male and 40% female students (see Table 4).

Table 4

First-Year, Full-Time Bachelor's Degree-Seeking Students

Total Entering Total Number Retained Percentage Retained

Females 1,224 1,086 88.7%

Site Access

Access to the study population and the Research Site was convenient for the researcher. Because the researcher works at the Research Site, the cost to complete the study was minimal. Permission to conduct the study was granted by the BCLU’s dean (see Appendix A) and acquired through the university’s Institutional Review Board (IRB).

Research Design and Rationale

This study was conducted to determine if participation in a learning community affects student academic achievement and university engagement. The researcher

followed a mixed-methods approach utilizing both quantitative and qualitative research. The CIPP evaluation model framed the evaluation design; this model contains four main areas for evaluation: context, input, process, and product. Following the CIPP, the researcher evaluated the learning community process and progress by considering the following questions:

o Product: To what extent do students perceive academic achievement, retention, and engagement to be attributable to their participation in the business learning community?

freshman learning community have significantly different rates of retention and engagement than their respective stand-

alone counterparts two years following learning community participation? o Process: How did student perceptions reported on survey data indicate that

participation in a freshman learning community experience was inherently more or less satisfying than what students had experienced in stand-alone courses?

o Product: Is there a significant relationship between students’ academic achievement (GPA) in high school and their participation or non- participation in a freshman learning community?

3. How was the first-year business learning community implemented as designed to achieve intended and unintended outcomes?

o Process: How are the components of the learning community currently being implemented at the Research Site’s business school?

o Process: What factors serve to facilitate implementation?

Within this mixed-methods study, the researcher followed methods in alignment with interactive or participatory evaluation. This approach to evaluation assumes those individuals with a vested interest in the evaluation should control its direction (Owen, 2007).

As Creswell (2012) notes, “[I]n qualitative inquiry, the intent is not to generalize to a population, but to develop an in-depth exploration of a central phenomenon” (p. 205). Additionally, “mixed methods research is a good design to use if you seek to build on the strengths of both qualitative and quantitative data” (Creswell, 2012, p. 535). Thus, the researcher adopted a phenomenological design for the study’s qualitative portion. As Creswell (2013) also notes, a phenomenological design is best if the researcher hopes to describe a common experience or meaning among several participants. Within this research study, the common experience centered on students’ time spent living in a learning community during the same period of time as the non-learning community participants. According to Mousakas (1994), considered by many to be the founder of phenomenology, the “research should focus on the wholeness of experience and a search for essences of experiences” (Simon & Goes, 2011, p. 1).

In an attempt to understand and quantify the effects of student participation in a residential business learning community on academic achievement and university

engagement, this study used existing data collected as a part of student applications to the university. The data was collected as a part of standard practice within the university’s admissions office. In addition, the researcher randomly selected samples of students who, two years prior to the study, both had and had not participated in the learning community. The students were asked to complete a questionnaire on their engagement with the BLCU and the university during their first two full years of attendance. Finally, the researcher interviewed students who had participated in the learning community. These participants were selected using a convenience sampling. As evidenced, the research consisted of two stages: quantitative and qualitative.

Research Methods Introduction – List of Methods Used

The program evaluation employed a mixed-methods research design.

Quantitative methods included questionnaires and university documents. The survey utilized in the study was adopted from the National Survey of Student Engagement (NSSE). The NSSE (2014) is used by over 1,500 colleges and universities across Canada and the United States to measure the extent to which college students use high-level learning and developmental practices. Key areas of analysis in the present study centered on course pass rates, student GPAs, and credit accumulation. The types of university record documents analyzed in the study are included in Table 5.

Table 5

University Data

University Data to be Analyzed

Gender GPA by term

Race/ethnicity SAT or TOEFL scores (including sub-scores) Number of credits completed to date Financial aid requirements/Family income

The qualitative method used in the study included face-to-face interviews. In the first stage, the researcher gathered specific data related to students’ collegiate and pre- collegiate academic achievement. The data collected included gender, high-school GPA, major area of study at the university, race, participation or non-participation in the

In the second stage, the researcher used random sampling to survey both students who had participated and had not participated in the learning community two years before the study. This random sampling was completed utilizing enrollment data

available to the researcher from the university. Students selected were sent an electronic survey utilizing the survey tool Qualtrics. In addition, the researcher used a convenience sampling method to interview five students who, two years before, had participated in the learning community. In addition, each interview allowed participants to understand the research topic and understand their rights as participants using the Informed Consent Form.

Description of Each Method Used

For the quantitative portion of the study, the researcher utilized available student data through the university’s admissions office. For the qualitative portion of the study, the researcher conducted an explanatory correlational methods approach to data-

collection and analysis (Creswell, 2012). The researcher used random sampling to select students currently enrolled in the university, as well as students who had and had not participated in the learning community.

A questionnaire based on the NSSE was distributed via email to potential participants. The questionnaire sought to identify themes within the categories listed below. In addition to these categories, NSSE measures four academic themes, including Academic Challenge, Learning with Peers, Experiences with Faculty, and Campus Environment (see Table 6). In 2014, over 470,000 students completed the NSSE, and approximately 4.5 million students have completed the survey since 2000 (NSSE, 2014).

Table 6

Questionnaire-Related Themes

Student Learning Engagement Activities Social Learning Connections

Active Learning & Teamwork Skills Social Interaction & Learning Behaviors Student Perceptions of Involvement & Support Intercultural Communications

Demographic Background

In order to best understand and discuss the demographic makeup of learning community students and non-learning community students, the data in Table 3.7 was collected from student records files available at the university.

Table 7

University Data Requirements

Data Required Section Rationale

Ethnic background To understand ethnic differences in each of the groups

SAT or TOEFL score To understand differences in standardized test scores

Family income To understand if financial backing is an advantage or has little to no effect

Gender To understand gender differences in each of the

groups

GPA To compare each of the groups based on

academic success

Participant Selection

In order to select participants for the study, the researcher followed the following protocol. First, students in the BCLU who had completed 70.5-129.5 credits were added

to the list of potential candidates. From this stage, the researcher ensured all students on the list had entered the university as traditional first-time freshman and had not

transferred to the BCLU or university from another college or institution. Next, the researcher filtered the list of students based on whether or not they had participated in the learning community during their first year. Once the list was filtered appropriately, the researcher sent an email invitation for students to complete the questionnaire. In addition, five students who were enrolled in the learning community and participated in the quantitative portion of the study were invited to take part in a face-to-face interview.

Data Collection

Data was collected from university records, face-to-face interviews, and a

questionnaire. When conducting the face-to-face interviews, the researcher recorded each interview and transcribed them at a later date. Data collected through university records and databases was saved in a file system available only to the researcher and housed on a server maintained by the BCLU. This server is backed up multiple times daily and is password-protected.

Qualitative Assessment

A phenomenological methodology was utilized to collect qualitative data. Phenomenology has been identified by the researcher due to its focus on an exploration of the individual perceptions of people who share an experience, in this case their participation or non-participation in a learning community. The descriptions of the experience, as seen through the participants’ eyes, provide a foundation to capture the essence of the experience (Moustakas, 1994).

A central portion of the study involved interviewing five learning community alumni to discuss and explore their experiences within the learning community and their time at the university since their enrollment in the learning community. The interviews allowed the researcher to gather alumni’s perceptions and discover their beliefs on the values of their learning community experience. In addition, the interview process provided a valuable avenue for students to reflect upon their experiences in the learning community and their perceptions on its implementation.

The interviews were conducted face-to-face, recorded, and subsequently transcribed. Using an open coding process, the researcher organized each interview response into themes. The process of interviewing and coding provided the researcher the opportunity to identify similarities and differences among the interviewees’

responses, as well as among the quantitative data collected. Research Questions Matrix – Mixed Methodology

Table 8

Proposed Research Questions

Research Question Mixed-Methods CIPP

Methodology Data Collection Data Analysis How does

participation in a first- year business learning community at an urban research university contribute to the academic achievement and university engagement of students in their third year of undergraduate study? Qualitative/Phenomenological Quantitative Process Standard interview protocol Adapted NSSE survey Open coding Fisher’s Exact Test

How was the first-year

In document A Program Evaluation: Examining the Effect of a Residential Business Learning Community on University Student Engagement and Academic Achievement (Page 43-61)