The steps for conducting research on teaching and learning mirror most of the steps used to conduct research on any topic (Sansone, Morf, & Panter, 2003). First, the teacher identifies a question of interest and then reviews what has been published on the topic. Next, the teacher ascertains how to answer the research question. One common approach to this sort of research is to measure relevant aspects of what the students are learning, make a change or introduce a new method or assignment, and then measure students’ learning again to determine the extent to which the manipulation affected it (see Bishop-Clark & Dietz-Uhler, 2012; Gurung & Schwartz, 2012, for exemplars of conducting SoTL research). This process becomes even easier if you identify a ScoL finding or prescription (e.g., repeated testing aids learning) and want to change your class design accordingly.
Similar to conducting psychological science research in general, you can start with descriptive studies (e.g., how are my students learning?), move on to correlational designs (e.g., what is student learning associated with?), and then design experiments (e.g., if I change this assignment, or use a ScoL finding, or introduce a new way of talking about this concept, will learning change?). In designing classroom experiments, teachers can choose from a wide variety of models. Bartsch (in press) presents a fuller description of different research designs and particular adaptations for classroom research.
At its core, research consists of testing hypotheses. Researchers may choose from three distinct approaches to hypothesis testing: validation, falsification, and qualification. The development and nature of the experimental paradigm began a little over 80 years ago with the work of Fisher, who first formalized essential elements of research, including the manipulation of independent and dependent variables, and randomization (Pelham & Harton, 2006). Other essential characteristics of research you need to be familiar with are validity, reliability, and measurement scales (Morling, 2012). Validity and reliability are crucial to both experimental and passive observational research (naturalistic research that does not involve the manipulation of variables).
The best way to assess the use of a ScoL finding is to conduct an experiment. A form of research that affords the most control is the true experiment, in which the researcher has control over all of the independent variables of interest and uses random assignment. The independent variable is what is manipulated. For example, if you were to have one section of your class take multiple quizzes based on your reading of the ScoL literature on repeated testing, and compare their exam scores with another section that did not take multiple quizzes, the independent variable would be ‘use of multiple quizzes’. True experiments are further broken down into one-way designs and factorial designs. In some studies, such as repeated measures designs, researchers hold some control over individual differences by exposing participants to more than one level of an independent variable. For example, you may use a pre and post test of learning (i.e., the repeated measure) to compare a section of a course that takes three quizzes as a study aid, with a section of a class that takes five practice quizzes. You cannot always manipulate factors such that one course section or class gets nothing as you may be keeping something valuable from that section. There are research designs to help control as many factors as possible and do research in the real world setting of the classroom (Morling, 2012). This is the tip of the iceberg in terms of research methods terminology, representing some of the most common elements of research design. You can familiarize yourself with many of these terms, issues, and research skills by reading
peer-reviewed scholarly articles published in journals in your field. Reading a research methods book provides an essential foundation, but reading journal articles is what keeps your research mind sharp.
Basic Ways to Assess Pedagogical Innovations
One easy way to start is to pick a course in which you test students many times or courses that are offered consistently every semester and year. This type of Repeated Measures Design (RMD)works well in course designs that have several similar exams or assignments. The term “RMD” is used in research when the assessment is identical such as when department learning outcomes or course objectives are measured year after year in the same college student sample. Unlike the example in the previous section, where the pre and post test repeated measure is used with the same group of students, this form of design tests different cohorts. In such designs, the key is to identify changes in responses to similar question(s) over time. The measure used repeatedly consists of the same number of questions asked in the same order, and differences in the responses will be taken to indicate changes in
knowledge.
However, using identical questions is not always practical or possible in most courses. To avoid this problem, many teachers modify the RMD to include a pretest and a posttest. For example, many large general education courses give the same test at the beginning and end of the semester (e.g., a test of knowledge of governmental policy in a political science course). To test whether learning is changing over the course of the semester, the teacher can test if the class average is changing over time, test if learning has changed from the beginning of the semester (using a class average) or even compare a single student’s score to his or her previous score to determine if a student is improving over the course of the semester. If there is a significant difference in student learning between the two assessments, it could be due to the instruction the teacher provided in between the pretest and the posttest. Of course, a teacher cannot be sure that the change was due only to instruction unless she has measured and controlled for many other possible factors such as how much and how the student studied. If, while holding other variables constant, a teacher finds a significant difference between the pretest and posttest measures, then she may be confident that her finding is a good indicator that instructional changes produced increases in learning.
This basic idea – measure learning (pre-test), introduce a change (e.g., repeat quizzing), measure learning again (post-test) – allows you to test the utility of any pedagogical innovation. The ScoL
provides a number of good practices (Dunlowsky, Rawson, Marsh, Nathan, & Willingham, 2013); you can take any one of them, introduce it into your class and see how learning changes. Although many such investigations can be conducted on a single class in a single semester, a pretest/posttest approach introduces the problem of testing effects when the original test affects later measurements. For example, within a social psychology class, you may first give students a 15-item measure assessing understanding of the cognitive dissonance or the bystander effect. You then discuss the concept in class using a new approach you read about in the ScoL literature and give them the same test to see if their knowledge increases. The problem is that taking the first learning measure, students might perform better on the second measure simply because they had taken the first measure and not due to your instruction. To avoid students recognizing a past measure, a good solution is to have two measures for the same construct (e.g., a form A and form B of a measure with different questions on each, Bartsch, Engelhardt Bittner, & Moreno, 2008).
Sometimes a teacher may not even use a pre measure of learning. The simplest type of design for comparing two-groups is called a two-group post-only design. In the two-group post-test only design,
one group receives the ScoL innovation and another does not, and then both are measured on the same assessment. The two groups could be two sections of a class or one class divided into two or the same class over two consecutive semesters. For example, in a test of whether giving PowerPoint lecture notes to students before the lecture was given in class, Noppe gave one section of her class notes in advance and did not provide notes to the other section (Noppe, 2007). She used two separate sections of the same course but used the same exam. The class that did not receive the notes scored lower on the exam. There was no pretest of learning used in this case but Noppe checked to see if the grade point averages of students in both sections were similar at the start of the class to eliminate the possibility that students in the section that received the notes were not as good students, which could have accounted for the same result.
Sometimes you can make sure all students get the innovation, or different levels or types of an
intervention. This type of design is called a within-participants design. This type of design is useful when you want to test different variations on a theme. For example, one study examined the effectiveness of PowerPoint presentations (Bartsch & Cobern, 2003). In this study researchers compared the
effectiveness of different forms of presentation format and content, a popular object of study in the ScoL (e.g., Mayer, 2011). Bartsch and Cobern used three forms of delivery: overhead transparencies, plain PowerPoint slides, and PowerPoint slides with pictures, graphs, transitions, and sound effects. Conditions rotated each week, and each week researchers quizzed the students.
Designing a sound way to test if your innovation worked is important, but do not forget to take the time to also design sound learning assessments. There are also good practices for writing test questions, essay prompts, and other forms of tests. Taking the time to fine tune one’s rubrics for grading and aligning assessments with one’s learning outcomes is time well spent. Stevens and Levi (2011) and Suskie (2009) provide indispensable guides for rubric development and assessment.