Limitations and future research

This study has some limitations. The study took place in a general chemistry course at a large Midwestern university, therefore the participants of this study represented a well-defined, selected group and our tests included only chemistry-related questions. These features restrict our findings to this subject group and material only. Consequently, future research needs to further investigate testing mode with more diverse participants and material.

Another limitation of this study was measuring cognitive load using subjective measures.

Although we supplemented the subjective measure of item difficulty with the objective measure (i.e., item difficulty index), mental effort was investigated solely with subjective ratings. Future

studies on testing mode, may incorporate physiological measures to record changes in cognitive load in real time as students take tests in either testing mode. This step has become possible with a variety of physiological measures such as functional magnetic resonance imaging (fMRI, Brandt et al., 2015; for a review see Whelan, 2007), electroencephalography (EEG, Antonenko, Paas, Grabner, & van Gog, 2010; Krigolson, Hassall, Satel, & Klein, 2015; Thilaga et al., 2015;

Vijayalakshmi et al., 2015), heart rate monitoring (Cranford, Tiettmeyer, Chuprinko, Jordan, &

Grove, 2014; Durantin, Gagnon, Tremblay, & Dehais, 2014), and eye-tracking (Jarodzka, Janssen, Kirschner, & Erkens, 2015; Mcewen & Dubé, 2015; Park, Korbach, & Brünken, 2015;

van Gog & Jarodzka, 2013).

Finally, in this study we only measured whether or not students used scratch paper. In the future studies, researchers may choose to measure how students used scratch paper (e.g., how much they write and what they write) when taking tests on paper versus computer. Such research could shed light on why students may use more scratch paper during a paper-based test than during an online test. The current study is only the first step in what we believe is a promising field of study in the area of testing mode.

Acknowledgments

We thank Dr. Appy and Logan Fischer who helped us to run the experiment. We also would like to thank Dr. Baluyut, Dr. Reed, and Gauri Ramasubramanian for helping create chemistry material, all student teaching assistants for proctoring and grading quizzes, and students for participating in this study. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

References

Antonenko, P., Paas, F., Grabner, R., & van Gog, T. (2010). Using electroencephalography to measure cognitive load. Educational Psychology Review, 22(4), 425–438.

https://doi.org/10.1007/s10648-010-9130-y

Ayres, P. (2006). Using subjective measures to detect variations of intrinsic cognitive load within problems. Learning and Instruction, 16(5), 389–400.

https://doi.org/10.1016/j.learninstruc.2006.09.001

Baddeley, A. D. (2002). Is Working Memory Still Working? European Psychologist, 7(2), 85–

97. https://doi.org/10.1027//1016-9040.7.2.85

Barrett, L. F., Tugade, M. M., & Engle, R. W. (2004). Individual Differences in Working Memory Capacity and Dual-Process Theories of the Mind. Psychological Bulletin, 130(4), 553–573. https://doi.org/10.1037/0033-2909.130.4.553

Bennett, R. E., Braswell, J., Oranje, A., Sandene, B., Kaplan, B., & Yan, F. (2008). Does it matter if I take my mathematics test on computer? A second empirical study of mode effects in NAEP. The Journal of Technology Learning and Assessment, 6(9).

Blin, F., & Munro, M. (2008). Why hasn’t technology disrupted academics’ teaching practices?

Understanding resistance to change through the lens of activity theory. Computers and Education, 50(2), 475–490. https://doi.org/10.1016/j.compedu.2007.09.017

Borges, S. S., Reis, H. M., Marques, L. B., Durelli, V. H. S., Bittencourt, I. I., Jaques, P. A., &

Isotani, S. (2016). Reduced GUI for an interactive geometry software: Does it affect students’ performance? Computers in Human Behavior, 54, 124–133.

https://doi.org/10.1016/j.chb.2015.07.064

Brandt, C. L., Kaufmann, T., Agartz, I., Hugdahl, K., Jensen, J., Ueland, T., … Westlye, L. T.

(2015). Cognitive Effort and Schizophrenia Modulate Large-Scale Functional Brain Connectivity. Schizophrenia Bulletin, 41(6), 1360–1369.

https://doi.org/10.1093/schbul/sbv013

Bratfisch, O., Borg, G., & Dornic, S. (1972). Perceived Item-Difficulty in Three Tests of Intelectual Performance Capacity. Stockholm: Institute of Applied Psychology (Vol. 29).

Stockholm.

Castillo-Manzano, J. I., Castro-Nu, M., Opez-Valpuesta, L. L., & Teresa Sanz-Díaz, M. (2017).

To take or not to take the laptop or tablet to classes, that is the question. Computers in Human Behavior, 68, 326–333. https://doi.org/10.1016/j.chb.2016.11.017

Clariana, R., & Wallace, P. (2002). Paper-based versus computer-based assessment: Key factors associated with the test mode effect. British Journal of Educational Technology, 33(5), 593–602. https://doi.org/10.1111/1467-8535.00294

Cranford, K. N., Tiettmeyer, J. M., Chuprinko, B. C., Jordan, S., & Grove, N. P. (2014).

Measuring load on working memory: The use of heart rate as a means of measuring chemistry students cognitive load. Journal of Chemical Education, 91(5), 641–647.

https://doi.org/10.1021/ed400576n

Ding, L., & Beichner, R. (2009). Approaches to data analysis of multiple-choice questions.

Physical Review Special Topics - Physics Education Research, 5(2), 1–17.

https://doi.org/10.1103/PhysRevSTPER.5.020103

Durantin, G., Gagnon, J. F., Tremblay, S., & Dehais, F. (2014). Using near infrared spectroscopy and heart rate variability to detect mental overload. Behavioural Brain Research, 259, 16–

23. https://doi.org/10.1016/j.bbr.2013.10.042

Emerson, L., & MacKay, B. (2011). A comparison between paper-based and online learning in higher education. British Journal of Educational Technology, 42(5), 727–735.

https://doi.org/10.1111/j.1467-8535.2010.01081.x

Engle, R. W. (2002). Working Memory Capacity as Executive Attention. Society, 11(1), 19–23.

https://doi.org/10.1111/1467-8721.00160

Hart, Sandra, G. (2006). NASA-task load index (NASA-TLX); 20 years later. In Human Factors and Ergonomics Society Annual Meting (pp. 904–908). Sage publications.

https://doi.org/10.1037/e577632012-009

Hochlehnert, A., Brass, K., Moeltner, A., & Juenger, J. (2011). Does medical students’

preference of test format (computer-based vs. paper-based) have an influence on

performance? BMC Medical Education, 11(1), 89. https://doi.org/10.1186/1472-6920-11-89 Holme, T., & Murphy, K. (2011). Assessing Conceptual and Algorithmic Knowledge in General

Chemistry with ACS Exams. Journal of Chemical Education, 88(9), 1217–1222.

https://doi.org/10.1021/ed100106k

Jarodzka, H., Janssen, N., Kirschner, P. A., & Erkens, G. (2015). Avoiding split attention in computer-based testing: Is neglecting additional information facilitative? British Journal of Educational Technology, 46(4), 803–817. https://doi.org/10.1111/bjet.12174

Kalyuga, S. (2011). Cognitive load theory: How many types of load does it really need?

Educational Psychology Review, 23(1), 1–19. https://doi.org/10.1007/s

Keng, L., McClarty, K. L., & Davis, L. L. (2008). Item-Level Comparative Analysis of Online and Paper Administrations of the Texas Assessment of Knowledge and Skills. Applied Measurement in Education, 21(3), 207–226. https://doi.org/10.1080/08957340802161774

Kennedy, C., Rhoads, C., & Leu, D. J. (2016). Online research and learning in science: A one-to-one laptop comparison in two states using performance based assessments. Computers &

Education, 100, 141–161. https://doi.org/10.1016/j.compedu.2016.05.003

Kim, H. J., & Kim, J. (2013). Reading from an LCD monitor versus paper: Teenagers’ reading performance. International Journal of Research Studies in Educational Technology, 2(1), 15–24. https://doi.org/10.5861/ijrset.2012.170

Krigolson, O., Hassall, C., Satel, J., & Klein, R. (2015). The impact of cognitive load on reward evaluation. Brain Research, 1627, 225–232. https://doi.org/10.3758/s13423-014-0772-5 Marshall, C. C. (1997). Annotation: From paper books to the digital library. In Proceedings of

the second ACM international conference on Digital libraries - DL ’97 (pp. 131–140).

https://doi.org/10.1145/263690.263806

Mcewen, R. N., & Dubé, A. K. (2015). Engaging or Distracting : Children’s Tablet Computer Use in Education. Educational Technology & Society, 18(4), 9–23.

Meyer, A. J., Innes, S. I., Stomski, N. J., & Armson, A. J. (2015). Student performance on practical gross anatomy examinations is not affected by assessment modality. Anatomical Sciences Education, 9(2), 111–120. https://doi.org/10.1002/ase.1542

Murphy, G., Groeger, J. A., & Greene, C. M. (2016). Twenty years of load theory—Where are we now, and where should we go next? Psychonomic Bulletin and Review, 1–25.

https://doi.org/10.1177/0963721410370295

Naismith, L. M., & Cavalcanti, R. B. (2015). Validity of Cognitive Load Measures in Simulation-Based Training. Academic Medicine, 90(11), S24–S35.

https://doi.org/10.1097/ACM.0000000000000893

Noyes, J., Garland, K., & Robbins, L. (2004). Paper-based versus computer-based assessment: Is workload another test mode effect? British Journal of Educational Technology, 35(1), 111–

113. https://doi.org/10.1111/j.1467-8535.2004.00373.x

Paas, F. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84(4), 429–434.

https://doi.org/10.1037/0022-0663.84.4.429

Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent development. Educational Psychologist, 38(1), 1–4.

Paas, F., Renkl, A., & Sweller, J. (2004). Cognitive load theory: Instructional implications of the interaction between information structures and cognitive architecture. Instructional Science, 32, 1–8.

Paas, F., van Merrienboer, J., & Adam, J. (1994). Measurement of cognitive load in instructional research. Perceptual and Motor Skills, 79(1), 419–430.

https://doi.org/10.2466/pms.1994.79.1.419

Park, B., Korbach, A., & Brünken, R. (2015). Do Learner Characteristics Moderate the Seductive-Details-Effect ? A Cognitive-Load-Study Using Eye-Tracking. Educational Technology & Society, 18(4), 24–36.

Prisacari, A. A. (2015). The testing effect in general chemistry: Effects of repeated testing on student performance across different test modes. MS Thesis, Iowa State University, Ames,

IA.

Prisacari, A. A., & Danielson, J. (2017). Rethinking testing mode: Should I offer my next chemistry test on paper or computer? Computers & Education, 106, 1–12.

https://doi.org/10.1016/j.compedu.2016.11.008

Schmeck, A., Opfermann, M., van Gog, T., Paas, F., & Leutner, D. (2015). Measuring cognitive load with subjective rating scales during problem solving: differences between immediate and delayed ratings. Instructional Science, 43, 93–114. https://doi.org/10.1007/s11251-014-9328-3

Selwyn, N. (2007). The use of computer technology in university teaching and learning: A critical perspective. Journal of Computer Assisted Learning, 23(2), 83–94.

https://doi.org/10.1111/j.1365-2729.2006.00204.x

Smith, K. C., Nakhleh, M. B., & Bretz, S. L. (2010). An expanded framework for analyzing general chemistry exams. Chem. Educ. Res. Pract., 11(3), 147–153.

https://doi.org/10.1039/C005463C

Suhr, K. A., Hernandez, D. A., Grimes, D., & Warschauer, M. (2010). Laptops and fourth grade literacy: Assisting the jump over the fourth-grade slump. The Journal of Technology, Learning and Assessment.

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. https://doi.org/10.1016/0364-0213(88)90023-7

Sweller, J. (2010). Element interactivity and intrinsic, extraneous, and germane cognitive load.

Educational Psychology Review, 22, 123–138. https://doi.org/10.1007/s10648-010-9128-5

Sweller, J., Ayres, P., & Kalyuga, S. (2011). Cognitive load theory. New York: Springer.

Sweller, J., & Chandler, P. (1994). Why Some Material Is Difficult to Learn. Cognition and Instruction, 12(3), 185–233. https://doi.org/10.1207/s1532690xci1203_1

Sweller, J., van Merrienboer, J., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251–296.

Thilaga, M., Vijayalakshmi, R., Nadarajan, R., Nandagopal, D., Cocks, B., Archana, C., &

Dahal, N. (2015). A heuristic branch-and-bound based thresholding algorithm for unveiling cognitive activity from EEG data. Neurocomputing, 170, 32–46.

https://doi.org/10.1016/j.neucom.2015.03.095

van Gog, T., & Jarodzka, H. (2013). Eye tracking as a tool to study and enhance cognitive and metacognitive processes in computer-based learning environments. In R. Azevedo & V.

Aleven (Eds.), International Handbook of Metacognition and Learning Technologies (Vol.

28, pp. 143–156). New York: Springer. https://doi.org/10.1007/978-1-4419-5546-3

van Gog, T., Kirschner, F., Kester, L., & Paas, F. (2012). Timing and frequency of mental effort measurement: Evidence in favour of repeated measures. Applied Cognitive Psychology, 26(6), 833–839. https://doi.org/10.1002/acp.2883

Vijayalakshmi, R., Nandagopal, D., Dasari, N., Cocks, B., Dahal, N., & Thilaga, M. (2015).

Minimum Connected Component - A Novel Approach to Detection of Cognitive Load Induced Changes in Functional Brain Networks. Neurocomputing, 170, 15–31.

https://doi.org/10.1016/j.neucom.2015.03.092

Whelan, R. R. (2007). Neuroimaging of cognitive load in instructional multimedia. Educational Research Review, 2(April 2006), 1–12. https://doi.org/10.1016/j.edurev.2006.11.001

Zheng, B., Warschauer, M., & Farkas, G. (2013). Digital writing and diversity: The effects of school laptop programs on literacy processes and outcomes. Journal of Educational Computing Research, 48(3), 267–299. https://doi.org/10.2190/EC.48.3.a