Student Perceptions of Effective Instruction and the Development of Critical Thinking: A Replication and Extension*
Chad N. Loes Mount Mercy University 1330 Elmhurst Drive NE Cedar Rapids, IA 52402 319-363-1323 x1536 [email protected] Mark H. Salisbury Augustana College Founders Hall 116 Rock Island, IL 61201 309-794-7504 [email protected] Ernest T. Pascarella
The University of Iowa N440 Lindquist Center Iowa City, Iowa 52242
319-335-5369 [email protected]
*The research on which this study was based was supported by a generous grant from the Center of Inquiry in the Liberal Arts at Wabash College to the Center for Research on Undergraduate Education at The University of Iowa. Corresponding author is Chad Loes <[email protected]>
Abstract
This study utilized data from the Wabash National Study of Liberal Arts Education to test the robustness of research conducted by Pascarella et al. (1996) that explored the relationship between student perceptions of exposure to organized and clear instruction and growth in critical thinking skills among college freshmen. To accomplish this, we created fully-specified models that included statistical controls for an array of potential confounding influences such as, student race, sex, pre-college critical thinking ability, precollege tested academic ability, parental
educational degree attainment, pre-college academic motivation, and a measure of interaction with high school teachers. Net of these influences, our findings generally replicate those uncovered by Pascarella et al. (1996) which suggest that student perceptions of organized instruction are positively associated with gains in critical thinking. Perceptions of instructional clarity, however, failed to exert a statistically significant influence on the dependent variable. Lastly, the results of our analyses suggest the effect of student perceptions of organized instruction on critical thinking affects students similarly, regardless of tested academic preparation (ACT or equivalent score), sex, or precollege critical thinking levels.
Keywords
Introduction
There is now a considerable body of research on effective teaching at the collegiate level (see Pascarella & Terenzini, 1991, 2005; Perry & Smart, 2007). Despite this body of knowledge, however, little is known about the relationship between student perceptions of effective teaching behaviors and one especially important outcome of the college experience – the development of critical thinking – which is described as one’s ability to clarify, analyze, evaluate, and extend arguments (ACT, 2011; Ennis, 1993; Gellin, 2003). In this paper, we estimate the impact of student perceptions of two effective instructional techniques on a standardized measure of critical thinking among college freshmen using a multi-institutional dataset. In short, our findings suggest that student perceptions of overall exposure to organized instruction are modestly, but significantly, associated with gains in critical thinking skills over the first year of college. Moreover, this relationship appears to hold true for all students in the sample, regardless of precollege critical thinking ability, sex, or tested academic preparation.
An extensive body of literature documents the relationship between specific teacher behaviors and course-related knowledge acquisition and student achievement. This literature has been summarized by a number of excellent meta-analyses and narrative syntheses ( e.g., Abrami, d’Apollonia, & Rosenfield, 2007; Braskamp & Ory 1994; Cashin, 1999; Cashin, Downey, & Sixbury, 1994; d’Apollonia & Abrami, 1997; Feldman, 1996, 1997; Greenwald, 1997; Marsh, 1987; Marsh & Dunkin, 1997; Marsh & Roche, 1997; McKeachie, 1997; Wachtel, 1998). Although there are a variety of effective teaching behaviors identified in the literature, in his synthesis of effective teaching, Feldman (1989) indentified student perceptions of two particular techniques – instructor clarity and organization – as the strongest correlates of student achievement (r = .56 and .57, respectively). Both of these forms of student perceptions of instruction – instructional clarity (clear explanations, effective use of examples) and instruction organization (use of course objectives, effective use of class time) – have been empirically established through randomized experiments (Hines, Cruickshank, & Kennedy, 1985; Schonwetter, Menec, & Perry, 1995).
It is important to note that the use of the terms, “instructional clarity” and “instructional organization” refer to measures of students’ perceptions of these effective instructional techniques. Although student perceptions are not exact measures of teacher behaviors, they are useful in measuring effective instruction. After summarizing the literature on student perceptions of instruction and instructional practices, Pascarella, Salisbury, and Blaich (2011) note, “(a) these perceptions are multidimensional, (b) they are reasonably reliable and stable, and (c) they have moderate positive correlations (e.g., .30 to .50) with various measures of course level learning such as course grade
and course final examination” (p. 5).
Despite scores of studies on effective instructional techniques and student achievement, very little empirical attention has focused on how teaching behaviors influence the development of students’ critical thinking skills. This is somewhat surprising, given the fact that, by assisting the efficient acquisition of factual knowledge and specific definitions, organized and clear instruction may allow for a more specific emphasis on critical thinking (Feldman, 1994; Pascarella, Edison, Nora, Hagedorn, & Braxton, 1996; Perry, 1991). In fact, established content acquisition may be a necessary antecedent for cognitive skills to develop (Rabinowitz and Glazer, 1985). Thus, it would seem theoretically justifiable to anticipate that overall exposure to instruction that enhances knowledge acquisition during college might also foster the development of critical thinking skills.
Although there are only a handful of studies that investigate how student perceptions of instructional techniques influence growth in students’ critical thinking skills, it is important to note that much is known about interventions designed to foster growth in this important outcome. For example, growth in critical thinking has been linked to educational activities such as: Student participation, encouragement, peer-to-peer interaction (Smith, 1977); and purposeful instruction and practice in reasoning and problem solving (Dale, Ballotti, Handa, & Zych, 1997; Halpern, 1993; Pascarella & Terenzini, 2005). The link between instruction focused reasoning skills and the development of critical thinking has also been established by means of a randomized experiment (Bailey, 1979). (See Abrami et al., 2008 for a thorough meta-analysis on antecedents associated with the development critical thinking skills.)
In our review of the literature, we uncovered only two studies that directly explore the association between student perceptions of effective teaching behaviors and the development of critical thinking. Shim and Walczak (2012) found that student perceptions of well-organized presentations and instructors who interpreted abstract concepts increased students’ self-reported gains in critical thinking; however, these same practices did not
significantly influence gains on a standardized measure of critical thinking. Although this study provides important contributions to our understanding of the influence of student perceptions of instructional techniques and critical thinking, it has some particularly important limitations. First, it partially relies on self-reported gains of problem-solving/critical thinking. Although it is sometimes practically necessary to use self-reported measures of critical thinking, standardized instruments that assess this important outcome are generally viewed as more
single-item measures to capture student perceptions of instructor organization and clarity. While this is not an especially detrimental point, it is generally preferable to use scales to assess such dynamic instructional techniques, as the (internal consistency) reliability is difficult to estimate with single-item measures (Wanous & Hudy, 2001).
Lastly, and most germane to the present investigation, Pascarella et al. (1996) sought to determine whether student perceptions of teacher organization and clarity influenced students’ general cognitive development (reading comprehension, mathematics reasoning, critical thinking, and a composite of these three measures). Net of
confounding influences, they found that student perceptions of instructor organization, but not instructor clarity, had a positive influence on all four outcome measures. Of particular relevance to the present study, their results were general rather than conditional. That is, the influence of student perceptions of teacher organization on critical thinking did not vary for students with different background characteristics.
It is also important to note that neither Shim and Walczak (2012), nor Pascarella et al. (1996) accounted for precollege academic preparation. This is an especially important point, as differences in critical thinking gains as a result of student perceptions of effective instruction could simply be an artifact of differing levels of precollege academic preparation. In addition, neither study addressed the nested nature of the data (described in more detail later in this paper). Failing to account for the aggregated nature of sampling often employed by multi-institutional educational research (rather than drawing a truly random sample across the entire target population) can lead to an increased chance for Type I error, or rejecting a true null hypothesis.
Most importantly, however, although the research by Pascarella and his colleagues (1996) signaled the possibility that student perceptions of effective instructional behaviors may indeed have a positive influence on students’ critical thinking skills during college, their findings are now almost 20 years old and have yet to be empirically replicated or extended. Several recent longitudinal studies demonstrate the degree to which today’s students bring a very different set of pre-college experiences to their postsecondary enrollment than those who began college in the early 1990s (NCES, 2005; NEA 2004, 2007; Pryor, Hurtado, Sa´enz, Santos, & Korn, 2007). Specifically, major advances in computer and communication technology and the resulting exponentially more pervasive availability of information may have fundamentally reshaped the way that present day undergraduates interact with learning and learning environments compared to the days before the development of the Internet, online education, and course management software (Mokhtari, Reichard, & Gardner, 2009). In the face of the array of sociological, cultural, and technological changes, it seems almost presumptuous to assume that the outcome of a
treatment or program identified two decades ago would hold in the present – let alone indefinitely. As Pascarella (2006) notes, “academic administrators and student affairs professionals would be provided a significantly greater margin of comfort in developing interventions or policies that are informed by replicated findings than by single sample results that have a greater probability of being artifacts” (p. 510).
The present study endeavored to replicate a key portion of the Pascarella et al. (1996) study by applying a similar analytic model to estimate the influences of student perceptions of teacher clarity and instructional
organization on growth in students’ critical thinking skills during the first year of college. Moreover, the present analysis extends this body of research by including a control for precollege academic preparation, adding a statistical procedure to adjust for the non-random nature of the data, and perhaps most importantly, analyzing data from a substantially different sample of undergraduates. Given the research reviewed above, we hypothesize that student perceptions of instructional organization and clarity will be significantly and positively associated with gains in critical thinking. Furthermore, in light of the important and substantial body of research on differential experiences by student background characteristics, the present study also sought to determine the extent to which precollege characteristics, specifically students’ tested precollege preparation (ACT or equivalent scores), sex, or precollege critical thinking levels, might moderate the impact of perceived clear and organized instruction on the development of critical thinking skills.
Thus, the specific research questions guiding our study were:
1. What are the net effects of perceived instructor organization and clarity on end of first-year critical thinking?
2. Do the net effects of perceived instructor organization and clarity on end of first-year critical thinking vary by precollege ACT score, sex, or precollege critical thinking levels?
Research Methods
Samples
Institutional Sample.
The data used in the analyses for this study came from the Wabash National Study of Liberal Arts
Education (WNSLAE). The WNSLAE is a large-scale, longitudinal study that examines the factors associated with important liberal arts educational outcomes. Over 60 colleges and universities responded to a national invitation to
participate in the WNSLAE. A purposive sample of 19 schools was drawn from the group of institutions that applied to be a part of the Wabash study. The 19 institutions were selected in an effort to diversify the institutional sample to include various geographic regions, institutional sizes, and student demographics. Because the study is mainly focused on the effects of liberal arts education, liberal arts institutions were purposefully over-represented. Three of the institutions are considered research universities, three are regional universities (non-doctoral granting), two are community colleges, and 11 are liberal arts colleges.
Student Sample.
In an effort to assess the impact of the first year of college on students, entering first-year students were surveyed in the fall of 2006 and then again in the spring of 2007. The initial sample was conducted in one of two ways. For large institutions, the initial sample was selected randomly from the incoming first-year class. In the largest participating institution, however, the sample was selected randomly from the incoming class in the College of Arts and Sciences. For a number of small liberal arts institutions, the entire entering first-year class was selected. Students were informed that they would complete a battery of assessments and questionnaires as they started college in the late summer/early fall of 2006. They were advised that they were being invited to participate in a national longitudinal study examining how a college education affects students, with the goal of improving the undergraduate experience. They were also informed that they would be invited again in the spring 2007 term (i.e., after they were essentially finished with their first year of college). All participants were given a monetary stipend of $50 for their willingness to complete the survey, and they were also assured their responses would remain confidential and never be associated with any of their institutional records.
Data Collection
Initial Data Collection.
The first panel of data was collected during the fall 2006 term. Data were collected from a total of 4,501 students from the 19 institutions. During this phase of the data collection, a precollege survey was distributed to acquire information on student demographic characteristics, family background, high school experiences, political orientation, educational degree plans, etc. Additionally, respondents also completed a battery of surveys that captured a host of intellectual and personal development measures theoretically associated with a liberal arts education. One of these was the critical thinking module of the Collegiate Assessment of Academic Proficiency
(CAAP) Due to the extensive amount of time required to complete the CAAP critical thinking module, half of the sample was randomly selected to complete this instrument during the initial data collection. The other random half of the sample completed another instrument (the Defining Issues Test) of approximately equal length.
Follow-up Data Collection.
The follow-up data collection was conducted during the spring 2007 term. This phase of the data collection took approximately two hours to complete. Respondents who participated in the follow-up data collection were each paid an additional $50 for their efforts. Of the original 4,501 respondents from the fall data collection, a total of 3,081 students returned to complete the follow-up surveys in the spring 2007 term, resulting in a response rate of 68.5%.
Two types of data were captured during the spring 2007 data collection: College experience assessments, and measures of cognitive and personal development. The college experience measures were based on questions from the National Survey of Student Engagement (NSSE) and the WNSLAE Student Experiences Survey (WSES). These particular surveys assess the extent to which students are engaged in, or exposed to, educational “good practices” (e.g., diversity experiences, clear and organized teaching) during college. To assess cognitive and personal development respondents completed a battery of follow-up (i.e., posttest) instruments. The college experience instruments were always administered and completed prior to the measures of cognitive and personal development. Recall that during the first wave of the data collection, the CAAP critical thinking test was completed by only half of the sample. That same random sample again completed the CAAP at the end of their first year of college. Useable data for our analyses were available for 1,354 students. Both phases of the data collection (fall 2006 and spring 2007) were conducted by ACT (formerly the American College Testing Program).
In an effort to make the sample more representative of the total freshman population, we created a
weighting algorithm. Within each participating institution we weighted the sample up to the first-year student by sex (male or female), race (White, Black, Hispanic, Asian/Pacific Islander, or other), and ACT score (or
COMPASS/SAT equivalent). Although the use of the weighting algorithm does not eliminate the issue of response bias, it does make the sample more representative of the population from which it was drawn.
Dependent Variable.
The dependent variable in this study is end of first-year critical thinking. The critical thinking module of the Collegiate Assessment of Academic Proficiency (CAAP) is a 40-minute, 32-item instrument. The CAAP describes critical thinking as one’s ability to clarify, analyze, evaluate, and extend arguments. The test consists of four passages in a variety of formats (e.g., case studies, debates, dialogues, experimental results, statistical arguments, editorials). Each passage contains a series of arguments that support a general conclusion and a set of multiple-choice test items. Previous research on the CAAP suggest its internal consistency reliability ranges
between .81 (Spearman-Brown) and .85 (Kuder-Richardson) among multiple populations of college freshmen (ACT, 2011). Other studies have shown that the CAAP critical thinking module correlates .75 with the multiple-choice Watson-Glaser Critical Thinking Appraisal (Pascarella, Bohr, Nora, & Terenzini, 1995) and .58 with the written Collegiate Learning Assessment (Klein, Liu, & Sconing, 2009). Although there are many instruments that could be used to capture critical thinking levels (e.g., Halpern Critical Thinking Assessment, Collegiate Learning
Assessment, California Critical Thinking Skills Test), the critical thinking module of the CAAP was chosen because it is commonly used in higher education research, is strongly correlated with a number of other instruments that assess critical thinking skills, and it is the exact tool that was used to estimate critical thinking levels in the Pascarella et al. (1996) study we sought to replicate.
Independent Variables.
Student perceptions of overall organized and clear instruction were measured in spring 2007 by two five-item scales. To assess perceptions of instructional organization, students were asked to assess the degree to which they agreed with the following statements about the instruction they received during the first year of college: The presentation of the material is well-organized; teachers are well-prepared for class; class time is used effectively; course goals and requirements are clearly explained; teachers have good command of what they are teaching. This scale had an internal consistency reliability of .84 To evaluate perceptions of teacher clarity, students were asked to consider the extent to which they agreed with the following statements about the instruction they were exposed to:
Teachers give clear explanations; teachers make good use of examples and illustrations to explain difficult points; teachers effectively review and summarize the material; teachers interpret abstract ideas and theories clearly; teachers give assignments that help in learning the course material. This scale had an internal consistency reliability of .83. Response options for both measures were 5 = “very often”; 4 = “often”; 3 = “sometimes”; 2 =
“rarely”; 1 = “never.” The question stem, items and response items were the same as those used by Pascarella et al. (1996).
Guiding Conceptual Model and Control Variables.
A variety of conceptual frameworks have been developed to better understand the influence of the college experience on students (e.g., Astin, 1993; Pascarella, 1985; Pascarella & Terenzini, 1991, 2005). According to these conceptual frameworks, in order to most accurately assess the impact of college on students, researchers must control for three sets of influences: Student background characteristics, institutional type, and other college experiences. We utilized these conceptual models to select the variables for inclusion in our equations. These models posit that learning and cognitive change are the direct result of several of influences, to wit, student
background characteristics and interaction with agents of socialization (e.g., faculty, peers). These changes are also thought to be the indirect result of other sources of influence such as the organizational characteristics of an institution (e.g., institutional type)
An additional consideration was that we wanted our analytical model to include variables that not only predicted the dependent measure, but which also predicted the independent variables – overall exposure to instructional clarity and instructional organization. While instruction received depends to a great extent on faculty behaviors, it also depends to some extent on the characteristics of individual students. Because of individual differences in response propensities and characteristics, two students could receive exactly the same instruction and respond to it differently (Pascarella, 2001). Thus, attempting to take those response propensities into account is an important consideration in estimating the unique impact of student perceptions of instruction received.
A particular strength of the WNSLAE is its longitudinal design. This permitted us to control for a wide range of potential confounding influences, similar to those taken into account in the original Pascarella, et al. (1996) study. Student pre-college/background characteristics included: race (White/student of color), sex, pre-college CAAP critical thinking score, ACT (or equivalent SAT/COMPASS) score (provided by each institution), parents’ educational degree attainment, a measure of pre-college academic motivation, and a measure of interaction with high school teachers. Institutional context was whether the student attended a liberal arts college. We chose this particular institutional characteristic because of the evidence that liberal arts colleges have a particularly unique teaching and learning environment (Pascarella, Wang, Trolian, & Blaich, 2013; Seifert, Pascarella, Goodman, Salisbury, & Blaich, 2010). Finally, other college experiences included number of courses taken in each of six
academic areas, self-reported college grades, and the average number of hours per week a student worked off-campus. (Detailed operational definitions of all variables are available from the first author, upon request.) Table 1 provides the weighted means and standard deviations of all variables, and Table 2 provides the matrix of
intercorrelations on which subsequent analyses were based.
Tables 1 and 2 about here
Analyses
We utilized fully-specified models to estimate the influence of the perceived teaching behaviors on critical thinking. To accomplish this, the analyses were carried out in two stages. In the first stage, we regressed end-of-first year critical thinking on the teacher organization and teacher clarity scales, while simultaneously accounting for the battery of control variables listed above, as well as for the clustered or nested nature of our data. The clustering effect (or “nested” nature of the data) essentially refers to the notion that students within each institution are likely to behave more similarly to one another than they are to students attending other institutions. Failing to account for this clustering effect in regression equations can lead to artificially reduced standard errors, which violates one of the assumptions of Ordinary Least Squares regression and increases the possibility of committing Type I error – rejecting a true null hypothesis (Ethington, 1997; Raudenbush & Bryk, 2001).
To address this problem, we utilized the (svy) option in Stata, the statistical program we used for our analyses, which adjusts the standard errors to account for the clustering effect. Because we had only 19 institutions (or sampling units) in our data set, we were limited in the number of variables we could include in our analyses (N-1), or 18 variables (Groves et al., 2004). Given the multi-institutional data used here, hierarchical linear modeling (HLM) might appear to be a more appropriate analytical technique. Although HLM is often useful when analyzing such data, we chose against using this technique because we analyzed only 19 aggregates. Research on the use of HLM suggests that 19 aggregates are too few to provide adequate statistical power for between-institution variables (Ethington, 1997; Raudenbush & Bryk, 2001). Moreover, because individuals (not institutions) were the unit of analysis, we chose not to use HLM in our analyses.
In the second stage of the analyses, we investigated whether the net effects of teacher organization and teacher clarity on end-of-first year critical thinking were moderated by, or conditional on students’ tested precollege
preparation (ACT or equivalent scores), sex, or precollege critical thinking levels. To determine if the effects of organized or clear instruction were conditional on student background characteristics, we created cross-product terms between tested precollege preparation (ACT or equivalent scores), sex, and precollege critical thinking level, on the one hand, and instructional organization and clarity on the other. We then added these cross-products individually to the general effects model specified in the first stage of the analyses. If any of the cross-product coefficients were statistically significant, it indicated that the effects of instruction depended on one of the three student background characteristics analyzed.
All analyses were conducted with weighted samples, adjusted to actual sample sizes for correct standard errors in significance tests. Prior to estimating our regression models we standardized all continuous variables with a mean of 0 and a standard deviation of 1. Because our prediction model controlled for a pretest measure of critical thinking, the coefficients and significance tests for all other variables in the models are exactly the same as if we were predicting a change or gain score [i.e., (2007 critical thinking – 2006 critical thinking)] (Pascarella, Wolniak, & Pierson, 2003).
We examined all the variables in our models for potential issues of multicollinearity and conducted a variance inflation factor test (see the correlation matrix in Table 2). The variance inflation factor ranged from 1.08 – 2.42, with a mean of 1.39, suggesting the multicollinearity of the variables is well within an acceptable range (Belsley, Kuh, & Welsch, 1980; Cohen, Cohen, West, & Aiken, 2003; Marquardt, 1970; Myers, 1990). Lastly, it should be noted that we use the term “effect” in reporting our results. Because our data are correlational, however, causal terms such as effect or influence should be understood in a statistical rather than an experimental sense. If a variable has a significant estimated “effect” in our results it only means that, given the other influences that we have controlled statistically, we cannot rule out a possible causal influence of that variable on critical thinking.
Results
As the correlation matrix in Table 2 shows, our regression model was reasonably well specified in terms of predictors. Unsurprisingly, the end-of-first-year (2007) critical thinking skills variable was strongly correlated with both pre-college critical thinking levels and ACT (or equivalent) score. It was also the case that the variables measuring student perceptions of exposure to organized and clear instruction during the first year of college were significantly linked to both student background characteristics and other college experiences. Both scales were significantly and positively correlated with pre-college critical thinking scores and level of academic motivation.
Similarly, during the initial year of college, students with higher grades tended to rate their overall instruction as more organized and clear than students with lower first-year grades. Perceived level of organized/clear instruction was also significantly linked to first-year coursework, with social sciences courses positively linked to instructional clarity, while humanities/arts courses were linked with perceived instructional organization.
The results of our regression analyses are summarized in Table 3. This Table shows the general effect estimates for the sample of 1,354 students. As the Table indicates, the overall regression model explained approximately 73 percent of the variance in end-of-first-year critical thinking skills. Net of all other influences, student perceptions of clear instruction failed to exert a statistically significant influence on first-year growth in critical thinking (coefficient = -.021, p > .05). However, student perceptions of organized instruction had a modest, but positive and statistically significant link with critical thinking skills (coefficient = .059, p < .05). The results of our tests for the presence of conditional effects yielded non-significant increases in the overall explained variance (R2), suggesting the effect of perceived organized and clear instruction affects all students similarly, regardless of tested precollege preparation (ACT or equivalent scores), sex, or precollege critical thinking level. This generally replicates the earlier finding of Pascarella, et al. (1996) with the National Study of Student Learning.
Table 3 about here Discussion and Implications
This study sought to replicate earlier research that found student perceptions of overall clear and organized instruction during college enhances not only specific course learning, but also the development of critical thinking skills. Using a very similar analytic model, and the exact same standardized measure of critical thinking, our results appear to largely support the thrust of prior findings. Similar to results reported nearly two decades ago (Pascarella et al., 1996), our multi-institutional findings suggest that student perceptions of organized instruction during college have a small, but statistically significant, positive effect on first-year gains in critical thinking skills.
Although student perceptions of organized instruction were significantly associated with gains in critical thinking when instructional clarity was taken in to account, perceptions of clear instruction failed to significantly influence first-year critical thinking when instructional organization was also considered. It is important to note that both measures – student perceptions of instructor organization and clarity – were positively and significantly correlated with end of first-year critical thinking (r=.20 and .13, respectively). In addition to their significant
relationship with the outcome variable, both measures were substantially correlated with each other - suggesting that effective teachers do several things well simultaneously. This also suggests that a certain amount of co-linearity existed between the two independent variables and probably led to the reverse (though non-significant) negative sign for teacher clarity in the regression results. Lastly, similar to the work we replicated, we felt it important to control for the joint effects of each measure of student perceptions of effective instruction in our analyses. (While both instructional clarity and instructional organization were vetted experimentally, the effects of each teaching dimension were considered separately.) A major purpose of our study was to estimate the net effects of each dimensions of teaching, while controlling for the influence of the other. When we did this, student perceptions of organized instruction once again were significantly associated with gains in critical thinking, even when student perceptions of teacher clarity were considered. Thus, in the first year, student perceptions of instructional organization may be more important than clarity - although both are positively correlated with the dependent variable. From a theoretical perspective this does not mean that instructional clarity is irrelevant in terms of enhancing cognitive growth during college, but only that instructional organization may be the more salient of the two.
Next, the importance of the replicated nature of our findings cannot be understated. Although replications are commonplace in medicine, chemistry, physics, and other scientific fields (Hyndman, 2010), they remain relatively scarce in the social sciences (Neuliep & Crandall, 1990), and even appear to be on the decline in some academic areas (Evanschitzky, Baumgarth, Hubbard, & Armstrong, 2007). This is somewhat troubling, as replication greatly reduces the likelihood of incorrectly concluding that an intervention or policy actually has an effect on a particular outcome (Hays, 1994). Higher education literature is replete with single-sample findings that have not been replicated. In fact, a recent analysis of the current top 100 education journals revealed that only 0.13% of studies in the field of education are replications (Makel & Plucker, 2013). This is unfortunate, as
replications in higher education research have important implications for academia. Specifically, given the dynamic and changing nature of college students over time, we cannot assume the outcome of a treatment or program will hold true indefinitely. These findings give faculty and administrators a much greater degree of confidence that a dimension of what students perceive to be effective instruction actually influences an especially salient goal of higher education – the development of students’ critical thinking skills.
Perhaps of equal importance to policy makers is the fact that delivering what students perceive to be organized classroom instruction might not be merely a function of an individual faculty member’s innate skills or dedication to teaching. In fact, Weimer and Lenze (1997) have pointed out that many of the specific skills needed to incorporate clarity and organization into instruction are eminently learnable by college faculty. Thus, institutional support for faculty development efforts that focus on teaching these rather straightforward skills may pay dividends in terms of critical thinking growth for all students, regardless of individual precollege characteristics.
Taken as a whole, our findings paint a relatively clear picture as to the influence of perceived instructional organization on critical thinking growth during the first year of college. Furthermore, similar to the Pascarella et al. (1996) findings, our results appear to be general rather than conditional. That is, perceptions of organized
instruction are positively associated with gains in critical thinking, regardless of student precollege critical thinking levels, sex, or tested precollege preparation (ACT/equivalent score). This suggests that increased focus on this particular instructional technique can be beneficial for all students, despite certain differences in individual background characteristics.
References
American College Testing Program (ACT). (2011). CAAP Technical Handbook 2011–2012. Available from http://www.act.org/caap/pdf/CAAPTechnicalHandbook.pdf
Abrami, P. C., Bernard, R. M., Borokhovski, E., Wade, A., Surkes, M. A., Tamim, R., & Zhang, D. (2008). Instructional interventions affecting critical thinking skills and dispositions: A stage 1 meta-analysis.
Review of Educational Research, doi:10.3102/0034654308326084.
Abrami, P., d’Apollonia, S., & Rosenfield, S. (2007). The dimensionality of student ratings of instruction: What we know and what we do not. In R. Perry & J. Smart (Eds.), Effective teaching in higher education:
Research and practice (pp. 385-456). The Netherlands: Springer.
Astin, A. W. (1993). What matters in college? Four critical years revisited. San Francisco, CA: Jossey-Bass.
Bailey, J. F. (1979). The effects of an instructional paradigm on the development of critical thinking of college students in an introductory botany course. Dissertation Abstracts International, 40(6), 3138A.
Belsley, D. A., Kuh, E. & Welsch, R. E. (1980). Regression diagnostics: Identifying influential data and sources of collinearity, doi:10.1002/0471725153.
Bowman, N. A. (2010). Can 1st-year college students accurately report their learning and development? American Educational Research Journal, doi:10.3102/0002831209353595.
Braskamp, L. A., & Ory, J. C. (1994). Assessing faculty work. San Francisco, CA: Jossey-Bass.
Cashin, W. E. (1999). Student ratings of teaching: Uses and misuses. Unpublished manuscript, Kansas State University, Manhattan.
Cashin, W. E., Downey, R. G., & Sixbury, G. R. (1994). Global and specific ratings of teaching effectiveness and their relation to course objectives: Reply to Marsh. Journal of Educational Psychology, doi:10.1037/0022-0663.86.4.649.
Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences. L. Erlbaum Associates Mahwah, NJ.
Dale, P. M., Ballotti, D., Handa, S., & Zych, T. (1997). An approach to teaching problem solving in the classroom. College Student Journal, 31, 76-79.
doi:10.1037/0003-066X.52.11.1198.
Ennis, R. H. (1993). Critical thinking assessment. Theory into Practice, doi:10.1080/00405849309543594.
Ethington, C.A. (1997). A hierarchical linear modeling approach to studying college effects. In J.C. Smart (Ed.),
Higher education: Handbook of theory and research, Volume XII (pp. 165-194). New York: Agathon Press.
Evanschitzky, H., Baumgarth, C., Hubbard, R., & Armstrong, J. (2007). Replication research’s disturbing trend.
Journal of Business Research, doi:10.1016/j.jbusres.2006.12.003.
Feldman, K. A. (1989). The association between student ratings of specific instructional dimensions and student achievement: Refining and extending the synthesis of data from multisection validity studies. Research in Higher Education, doi:10.1007/BF00992392.
Feldman, K. (1994, January). Identifying exemplary teaching: Evidence from course and teacher evaluations. Paper commissioned by the National Center on Postsecondary Teaching, Learning, and Assessment. Stony Brook, NY: State University of New York at Stony Brook.
Feldman, K. A. (1996). Reflections on the study of effective college teaching and student ratings: One continuing quest and two unresolved issues. Unpublished manuscript, State University of New York, Stony Brook. Feldman, K. A. (1997). Identifying exemplary teachers and teaching: Evidence from student ratings. In R. Perry & J.
Smart (Eds.), Effective teaching in higher education: Research and practice (pp. 368-395). New York, NY: Agathon.
Gellin, A. (2003). The effect of undergraduate student involvement on critical thinking. Journal of College Student Development, doi:10.1353/csd.2003.0066.
Greenwald, A. G. (1997). Validity concerns and usefulness of student ratings of instruction. American Psychologist, doi:10.1037/0003-066X.52.11.1182.
Groves, R., Fowler, F., Couper, M., Lepkowski, J., Singer, E., & Tourangeau, R. (2004). Survey methodology. Hoboken, NJ: Wiley-Interscience.
Halpern, D. F. (1993). Assessing the effectiveness of critical-thinking instruction. The Journal of General Education, 42, 238-254.
Harper, S. R., & Quaye, S. J. (Eds.) (2008). Student engagement in higher education: Theoretical perspectives and practical approaches for diverse populations. New York: Routledge.
Hays, W. (1994). Statistics (5th ed.). Fort Worth, TX: Harcourt Brace.
Hines, C., Cruickshank, D., & Kennedy, J. (1985). Teacher clarity and its relationship to student achievement and satisfaction. American Educational Research Journal, doi:10.3102/00028312022001087.
Hyndman, R. J. (2010). Encouraging replication and reproducible research. International Journal of Forecasting, doi:10.1016/j.ijforecast.2009.12.003.
Klein, S., Liu, O., & Sconing, J. (2009). Test validity study (TVS) report. Washington, DC: Fund for the Improvement of Postsecondary Education.
Makel, M. C., & Plucker, J. A. (2013). Facts are more important than novelty: Replication in the education sciences. Manuscript submitted for publication.
Marsh, H. W. (1987). Students' evaluations of university teaching: Research findings, methodological issues, and directions for future research. International Journal of Educational Research, doi:10.1016/0883-0355(87)90001-2.
Marsh, H. & Dunkin, M. (1997). Students’ evaluations of university teaching: A multi-dimensional
perspective. In R. Perry & J. Smart (Eds.), Effective teaching in higher education: Research and practice
(pp. 241-320). New York, NY: Agathon.
Marsh, H. W., & Roche, L. A. (1997). Making students' evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, doi:10.1037/0003-066X.52.11.1187.
Marquardt, D.W. (1970). Generalized inverses, ridge regression and biased linear estimation. Technometrics,
doi:10.2307/1267205.
McKeachie, W. J. (1997). Student ratings: The validity of use. American Psychologist,
doi:10.1037/0003-066X.52.11.1218.
Mokhtari, K., Reichard, C. A., & Gardner, A. (2009). The impact of internet and television use on the reading habits and practices of college students. Journal of Adolescent & Adult Literacy, doi:10.1598/JAAL.52.7.6. Myers, R. (1990). Classical and modern regression with applications. (2nd ed.). Boston, MA: Duxbury Press. National Center for Education Statistics. (2005). The condition of education (NCES 2005-094). Washington,
National Endowment for the Arts. (2004). Reading at risk: A survey of literacy reading in America. Washington, DC: Author.
National Endowment for the Arts. (2007). To read or not to read: A question of national consequence. Washington, DC: Author.
Neuliep, J. W., & Crandall, R. (1990). Editorial bias against replication research. Journal of Social Behavior & Personality, 5(4), 85-90.
Pascarella, E. T. (1985). College environmental influences on learning and cognitive development: A critical review and synthesis. Higher Education: Handbook of Theory and Research, 1(1), 1–61.
Pascarella, E. T. (2001). Using student self-reported gains to estimate college impact: A cautionary tale. Journal of College Student Development, 42(5), 488–492.
Pascarella, E. T. (2006). How college affects students: Ten directions for future research. Journal of College Student Development, doi:10.1353/csd.2006.0060.
Pascarella, E.T., Blaich, C., Martin, G. L., & Hanson, J. M. (2011). How robust are the findings of Academically Adrift? Evidence from the Wabash National Study. Change, doi:10.1080/00091383.2011.568898. Pascarella, E., Bohr, L., Nora, A., & Terenzini, P. (1995). Cognitive effects of 2-year and 4-year colleges: New
evidence. Educational Evaluation and Policy Analysis, doi:10.3102/01623737017001083.
Pascarella, E., Edison, M., Nora, A., Hagedorn, L., & Braxton, J. (1996). Effects of teacher organization/preparation and teacher skill/clarity on general cognitive skills in college. Journal of College Student Development, 37, 7-19.
Pascarella, E. T., Salisbury, M. H., & Blaich, C. (2011). Exposure to effective instruction and college student persistence: A multi-institutional replication and extension. Journal of College Student Development, doi:10.1353/csd.2011.0005.
Pascarella, E. T., & Terenzini, P. T. (1991). How college affects students: Findings and insights from twenty years of research. San Francisco, CA: Jossey-Bass.
Pascarella, E.T, & Terenzini, P.T. (2005). How college affects students (Vol. 2): A third decade of research. San Francisco, CA: Jossey-Bass.
Pascarella, E., Wang, J., Trolian, T., & Blaich, C. (2013). How the instructional and learning environments of liberal arts college enhance cognitive development. Higher Education, 1-15.
Pascarella, E. T., Wolniak, G. C., & Pierson, C. T. (2003). Explaining student growth in college when you don't think you are. Journal of College Student Development, doi:10.1353/csd.2003.0007.
Perry, R. P. (1991). Perceived control in college students: Implications for instruction in higher education. In J.C. Smart (Ed.), Higher education: Handbook of theory and research (pp. 1-56). New York: Agathon. Perry, R. P., & Smart, J. C. (2007). The scholarship of teaching and learning in higher education: An
evidence-based perspective. The Netherlands: Springer.
Pike, G. R. (1996). Limitations of using students’ self-reports of academic development as proxies for traditional achievement measures. Research in Higher Education, doi:10.1007/BF01680043.
Pryor, J. H., Hurtado, S., Sa´enz, V. B, Santos, J. L., & Korn, W. S. (2007). The American freshman: Forty year trends. Los Angeles: Higher Education Research Institute.
Rabinowitz, M., & Glaser, R. (1985). Cognitive structure and process in highly competent performance. In F. Horowitz and M. O’Brien (Eds.), The gifted and talented: Developmental perspectives, (pp. 75-98). Washington, DC: American Psychological Association, doi:10.1037/10054-003
Raudenbush, S.W., & Bryk, A.S. (2001). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage Publications.
Schonwetter, D., Menec, V., & Perry, R. (April, 1995). An empirical comparison of two effective college teaching behaviors: Expressiveness and organization. Paper presented at the annual meeting of the American Educational Research Association. San Francisco, CA.
Seifert, T., Pascarella, E., Goodman, K., Salisbury, M., & Blaich, C. (2010). Liberal arts colleges and good practices in undergraduate education: Additional evidence. Journal of College Student Development,
doi:10.1353/csd.0.0113.
Shim, W. J., & Walczak, K. (2012). The impact of faculty teaching practices on the development of students' critical thinking skills. International Journal of Teaching and Learning in Higher Education, 24(1), 16-30.
Smith, D. G. (1977). College classroom interactions and critical thinking. Journal of Educational Psychology, 69(2), 180-190. doi: 10.1037/0022-0663.69.2.180.
Wachtel, H. K. (1998). Student evaluation of college teaching effectiveness: A brief review. Assessment & Evaluation in Higher Education, doi:10.1080/0260293980230207.
Wanous, J. P., & Hudy, M. J. (2001). Single-item reliability: A replication and extension. Organizational Research Methods, doi:10.1177/109442810144003.
Weimer, M., & Lenze, L. (1997).Instructional interventions: Review of the literature on efforts to improve instruction. In R. Perry & J. Smart (Eds.), Effective teaching in higher education: Research and practice
Table 1
Weighted Means and Standard Deviations for All Variables (n = 1,354)
Variable Mean SD
Critical Thinking - End of First Year 62.598 5.782
Male 0.466 0.499
White 0.831 0.375
Tested Academic Preparation/ACT 24.853 4.928
HS Teacher Interaction 3.246 1.070
Academic motivation 3.504 0.567
Parents’ Education 14.137 10.156
Critical Thinking - Precollege 62.475 5.295
Attended a Liberal Arts College 0.232 0.422
Hours/Week Working at a Job 7.969 11.451
College Grades 6.067 1.629
Courses taken in the natural sciences 1.197 1.303
Courses taken in engineering 0.107 0.416
Courses taken in humanities 2.327 1.589
Courses taken in social sciences 1.461 1.116
Courses taken in mathematics 1.165 0.829
Courses taken in business 0.203 0.593
(Teacher) Organizationa -0.029 0.961
(Teacher) Claritya 0.031 1.005
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 1. Critical Thinking - End of First Year 1.00
2. Male 0.03 1.00
3. White 0.23 0.05 1.00
4. Tested Academic Preparation/ACT 0.68 0.08 0.19 1.00
5. HS Teacher Interaction 0.02 -0.06 -0.02 0.02 1.00
6. Academic motivation 0.07 -0.10 -0.06 0.06 0.32 1.00
7. Parents’ Education 0.28 0.08 0.18 0.39 0.01 -0.05 1.00
8. Critical Thinking - Precollege 0.79 0.06 0.24 0.70 0.03 0.08 0.32 1.00
9. Attended a Liberal Arts College -0.05 0.05 0.04 -0.08 0.02 0.03 -0.01 -0.08 1.00
10. Hours/Week Working at a Job -0.20 -0.05 -0.13 -0.28 0.05 0.03 -0.24 -0.19 -0.01 1.00
11. College Grades 0.37 -0.07 0.16 0.35 0.09 0.15 0.16 0.37 -0.06 -0.10 1.00
12. Courses taken in the natural sciences 0.14 0.05 0.03 0.22 0.02 0.10 0.07 0.15 -0.21 -0.12 0.01 1.00
13. Courses taken in engineering 0.05 0.21 -0.02 0.15 0.00 -0.02 0.08 0.06 -0.11 -0.05 -0.05 0.18 1.00
14. Courses taken in humanities 0.32 -0.05 0.07 0.30 0.04 -0.01 0.16 0.30 0.16 -0.11 0.20 -0.23 -0.11 1.00
15. Courses taken in social sciences 0.07 -0.03 0.01 0.10 0.07 -0.01 0.06 0.06 0.10 0.01 0.08 -0.18 -0.13 -0.01 1.00
16. Courses taken in mathematics -0.01 0.13 -0.01 0.02 -0.03 0.02 0.02 0.01 -0.23 -0.02 -0.04 0.09 0.24 -0.22 -0.15 1.00
17. Courses taken in business -0.15 0.07 0.01 -0.11 0.03 -0.06 -0.05 -0.15 -0.04 0.08 -0.05 -0.08 0.04 -0.15 0.07 0.10 1.00
18. (Teacher) Organization
0.20 0.02 0.05 0.11 0.04 0.14 0.08 0.13 0.05 -0.04 0.18 0.02 -0.06 0.09 0.02 0.01 -0.04 1.00
19. (Teacher) Clarity
0.13 0.01 0.00 0.06 0.06 0.18 0.04 0.09 0.07 0.02 0.17 -0.02 -0.04 0.08 0.01 -0.01 -0.05 0.68 1.00
Note: Correlations of .07 or greater are significant at p < .01; correlations of .09 or greater are significant at p < .001. Table 2: Correlations Among All Variables
Table 3
Regression Estimates for Effects of Perceived Instructional Organization and Clarity on End of First-Year Critical Thinking using the Wabash National Study of Liberal Arts Education (n = 1,354) End of First-Year Critical Thinking Variablesa Coef. SE Male -0.116 * 0.053 White 0.045 0.037
Tested Academic Preparation/ACT 0.292 *** 0.045
HS Teacher Interaction 0.083 * 0.033
Academic motivation -0.020 0.017
Parents’ Education 0.007 0.023
Critical Thinking – Precollege 0.554 *** 0.036
Attended a Liberal Arts College 0.023 0.053
Hours/Week Working at a Job -0.036 0.026
College Grades 0.000 0.012
Courses taken in the natural sciences 0.025 0.028
Courses taken in engineering 0.008 0.020
Courses taken in humanities 0.042 0.033
Courses taken in mathematics -0.021 0.023
Courses taken in business -0.005 0.020
(Teacher) Organization 0.059 * 0.024
(Teacher) Clarity -0.021 0.049
R2 0.729 ***
aAll continuous variables were standardized prior to analysis
* p < 0.05, ** p < 0.01, *** p < 0.001
