New technologies in
psychological assessment:
The example of
computer-based collaborative problem
solving assessment
Katarina Krkovic, Anita
Pásztor-Kovács, Molnár Gyöngyvér & Samuel
Greiff
University of Luxembourg &
University of Szeged
[email protected]
Abstract
Computer-based assessment is a relatively new but exponentially growing field in
psychological assessment. The advantages are numerous – flexibility of
application, possibility to use video and audio material, construction of dynamic
tasks, and the availability of log-file data. Moreover, computers allow us to capture
the whole process of solving one task. Since collaborative problem solving (ColPS)
is a skill, which is becoming increasingly important in 21
stcentury, constructing an
appropriate assessment tool for it is a high priority in research at the moment.
Computer-based assessment of ColPS appears to be a logical solution in order to
capture the entire process of collaboration. However, the construction of
computer-based ColPS assessment comes with many questions, which need to be
discussed - Can we use computer-agents instead of real humans to simulate the
collaboration and how the social presence influences test-taker`s behavior? How
can we assess communication? Can chat substitute a real-life communication?
How can we properly use data obtained from log-files? These and some other
questions are addressed in this paper.
Introduction
Fascinating developments in informational technology – new software, internet, and social networks - caused tremendous changes in many life sectors including the science. Many disciplines have changed significantly due to computerization and they benefit from it in different ways. For instance, in medicine, many medical devices, such as pacemaker or prosthetics, function with the help of a computer.
In psychology, the computerization is still work in progress. In some areas of psychology computers are already showing to be exceptionally efficient. For instance, computer-generated virtual reality simulations already offer relatively successful therapy for different phobia such as fear of flying, or fear of small animals (Rothbaum et al., 1997). Moreover, in clinical psychology there are several programs available offering a cognitive training for patients who suffered neurological damages (Gontkovsky et al., 2002). However, in the field of psychological assessment scientists are still discovering how to properly adapt paper-pencil versions of psychological assessments into computerized versions. The importance of adaptation procedures and equivalence studies is often underestimated, and can result in computer tests with poor psychometric quality. Besides adapting existing paper-pencil assessment tools into computer-based assessments, the expansion of informational technologies brings possibilities of
constructing new, innovative computer-based instruments. Moreover new technologies are enabling us to assess some aspects of human behavior and cognitive possibilities that we were not able to assess before. For instance, computer-based assessment enables the implementation of various types of material – videos, audio material, drag-and-drop tasks, dynamical items, etc. (Parshall et al., 2010). The implementation of that kind of items can make it possible to assess some more complex skills, such as dynamic problem solving, creativity or collaborative problem solving (ColPS) skills.
What is collaborative problem solving?
ColPS as a construct is defined inconsistently by different authors in the literature. For instance, O`Neil et al. (2003) describe ColPS as searching for the path from the initial state to the goal state while interacting with others working on a shared goal (O`Neil et al., 2003). Other attempt of an appropriate definition comes from one of the most recognized large-scale assessments worldwide, the Programme for International Student Assessment (PISA). There, ColPS is defined as "…the capacity of an individual to
effectively engage in a process whereby two or more agents attempt to solve a problem by sharing the understanding and effort required to come to a solution and pooling their knowledge, skills and efforts to reach that solution" (OECD, 2012).
21st century skills are skills essential for a potential worker in order to be successful in today`s world, and ColPS is one of these skills (Binkley et al., 2012). This comes from the fact that we are confronted with solving problems in a group in various life areas. Being aware of it or not, we are collaboratively solving problems in many aspects of our lives - in our families, when making decisions and solving household problems, in schools where solving problems in collaboration with other students is required in regular team activities in classes, and finally, in our work, where we collaborate on different levels with
colleagues exchanging knowledge and expertise and trying to reach common goals.
Collaborative problem solving assessment
Notwithstanding the fact that various authors define ColPS differently, there is a certain consistency when it comes to the fact that an important part of ColPS is the process itself. Accordingly, computer-based assessment of ColPS is a natural choice since assessing a
process without the use of computer would mean observational procedures, which often lack objectivity and reliability. Unfortunately there is still no standardized computer-based assessment instrument of ColPS. However, due to emerging importance of ColPS in education, there is a strong initiative to finally provide a sound measurement of this skill. There are several open questions that researchers have to face and adequately address in order to construct a valid and scalable ColPS assessment tool. One of the biggest
challenges in creating a suitable instrument is to achieve a standardized environment in which ColPS will occur. In reality, we collaborate with different kinds of people – friends or strangers, highly competent or incompetent people, and the result of the work is not only depending on us, but also on people we are working with. Additionally, situations in which we are collaboratively solving a problem can be different – working under time pressure, conflict, in stressful or in peaceful environment. From a psychometrical point of view, in order to be able to assess ColPS of individuals and compare their results we have to place each individual in the same situation enabling a standardized assessment
procedure.
The challenges of computer-based collaborative
problem solving assessment
Using computer agents as collaborators
In order to assess the collaboration part of ColPS we need to have someone in the assessment situation to collaborate with. If we use actual people, whose behavior is generally unpredictable, it becomes impossible to construct a standardized setting for the assessment. Computers offer a sophisticated solution to this problem: using computer agents as collaborators in the assessment instead of real humans (OECD, 2012). The application of a computer agent capable of written or oral speech and even gesturing seems to be a very promising way to optimize the conditions for standardization
(Graesser, Jeon & Dufty, 2008). However, from the psychological point of view, the use of computer agents comes with significant constraints. Firstly, the face validity of a human agent vs. computer agent setting is questionable. Against developments in programming computer`s behavior (i.e., communicating emotions, reacting adequately to other’s emotions), generating and managing conflicts are still barely expected from a computer agent software. Thus, besides the great potential of achieving a relatively high
standardization, the level of closeness of this solution to the real life collaboration is arguable. Secondly, different experiments verify the important role of social presence in human behavior: humans act differently when they act with a computer than with a real human (Weinel et al., 2011). Miwa and Terai (2012) outline that person`s behavior is influenced by the instruction whether the partner is a computer or a human, and not by the partner’s behavior itself. Therefore, it can be assumed that using a computer agent as a partner and giving a proper instruction about it will not trigger the same behavior as the collaboration with a real human. Finally, the question that arises is if constructing a
computer agent which is so realistic that the person working with it does not notice that she/he is working with a computer is even ethical, and if it can be seen considered as a deception of the test-taker.
All in all, a compromise must be made and scientists have to decide if the high standardization that computers offer is worth sacrificing the face validity that human-based collaboration offers.
Assessing communication
Another challenge in computer-based assessment of ColPS is to adequately assess communication. The problem arising from using computer-based assessment is how to properly structure the conversation. Since the goal is to make an assessment that will be applicable in different settings, for example in a large-scale assessment or in a school setting where special equipment (e.g., headphones, high screen resolution, or high internet speed) is not available, a video or audio chat is not a good option. However, when we exclude image and sound, one big part of communication is lost: nonverbal communication (i.e., gestures, touch, physical distance, facial expression, or eye contact). Nevertheless, nonverbal communication is a very important aspect of communication considering that two thirds of human conversation is nonverbal (Nistor, 2012). One possibility to structure communication in a ColPS task is to use chat. Although losing the nonverbal information, there is the advantage of more or less automatic scoring of the obtained data. Considering the fact that the form of written ColPS is more and more common in our daily life, the idea of assessing ColPS skill by chat is understandable and should not be seen as a constraint (Hermann, Rummel & Spada, 2001).
Usage of the log-file data
Recent research results show that the collection and analysis of many kinds of assessment tasks data can be easily done by computers. However, to assess ColPS, some meta-data, which influence the achievement, are also necessary (e.g., number of trials, time spent on a task, or testing time). These meta-data can be provided by log-files. Several studies report using log-files to examine think-aloud protocol, eye tracking, head movement or facial expressions recorded by computer (c.f., Van Gog, Paas & van Merriënboer, 2005; Csapo, Lorinz & Molnar, 2012). Considering the results of aforementioned studies, log-file data can be very useful for ColPS assessment. For instance, mouse movements can inform us about who is the first one to reacts, or who gives the actual answer. This kind of information may let us draw conclusions of the collaboration dynamics. However in order to use log-file data, a consensus must be found, which information is actually important for ColPS.
Further on, different types of content-analysis software enable relatively detailed and reliable content-analysis of the communication in ColPS assessment. However, also here there are constraints – such software is not available for all languages, their development is expensive, their application is time consuming and therefore not possible in a large-scale assessment. For the purposes of the large-large-scale assessment the automatic scoring appears to be necessary. The application of predefined chat messages could be a solution for automatic scoring. A chat with predefined messages mostly relies on pre-studies that investigate with an open chat which are the important and most common messages (Hsieh & O’Neil, 2002, Rosen & Tager, 2013). However, besides underlining the
inflexibility of this solution, which can cause serious frustration to the participants, we also have to address the question if predefined messages are suggestive – what effect has, for example, the sequence of messages (i.e., primacy/recency effect: the first or the last message we see in the list influences which message we are going to choose).
Summarized, although by using computers we can collect more information than with a conventional test formats, it is essential to consider how this information can be used and correctly interpreted.
Conclusion
enormous advantages, which would be a waste not to exploit. However, as we show on the example of ColPS assessment, there are some significant constraints and even ethical concerns regarding the application of computers in measurement. Many questions are still open: Is there a way to achieve face-validity in computer-based assessment? Is
computer-based assessment useful for every situation, and if not, where are the limits? Finally the most important question and challenge for the researchers is how to make the best out of the computer-based assessment – how to overcome mentioned constraints and exploit the advantages to the fullest.
References
Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., & Rumble, M. (2011). Defining 21st Century Skills. In P. Griffin, B. McGaw, & E. Care (Eds.). Assessment and teaching 21st century skills. Heidelberg: Springer.
Csapo, B., Lorincz, A., & Molnar, G. (2012). Innovative assessment technologies in educational games designed for young students. In D. Ifenthaler, D. Eseryel, X. Ge (Eds.), Assessment in game-based learning: foundations, innovations, and
perspectives. (pp. 235-254). New York: Springer.
Graesser, A. C., Jeon, M., & Dufty, D. (2008). Agent technologies designed to facilitate interactive knowledge construction. Discourse Processes, 45, 298–322.
Gontkovsky, S. T., McDonald, N., B., Clark, P. G., & Ruwe, W. D. (2002). Current directions in computer-assisted cognitive rehabilitation. NeuroRehabilitation, 17(3), 195-199.
Hermann, F., Rummel, N. & Spada, H. (2001). Solving the case together: The challenge of net-based interdisciplinary collaboration. In P. Dillenbourg, A. Eurelings, & K. Hakkarainen (Eds.), Proceedings of the first European conference on computer-supported collaborative learning (pp. 293-300). Maastricht, NL: McLuhan Institute. Hsieh, I.-L. & O’Neil, H. F. Jr. (2002). Types of feedback in a computer-based
collaborative problem solving group task. Computers in Human Behavior, 18(1), 699-715.
Miwa, K., & Terai, H. (2012). Impact of two types of partner, peceived or actual, in human-human and human-agent interaction. Computers in Human Behavior, 28, 1286-1297.
Nistor, G. (2012). The Role of the Nonverbal Communication in Interpersonal Relations.
Procedia - Social and Behavioral Sciences, 47, 552-556.
OECD (2012) Draft PISA 2015 Collaborative Problem Solving Assessment Framework (EDU/PISA/GB(2012)11). Presented at the 33rd meeting of the PGB, Tallinn, Estonia, 16-18 April 2012
Parshall, C. G., Harmes, C., Davey, T., & Pashley, P. J. (2010). Innovative items for computerized testing. In W. J. van der Linden & C. A. W. Glas (Eds.). Computerized adaptive testing: Theory and practice (2nd ed.). Norwell, MA: Kluwer Academic Publishers.
Rosen, Y. & Tager, M. (2013). Computer-based Assessment of Collaborative Problem Solving Skills: Human-to-Agent versus Human-to-Human Approach. Retrieved May 23, 2013 from
Rothbaum, B. O., Hodges, L., & Kooper, R. (1997). Virtual reality exposure therapy. The Journal of Psychotherapy Practice and Research, 6(3), 219-226.
Van Gog, T., Paas, F., & van Merriënboer, J. J. G. (2005). Uncovering expertise related differences in troubleshooting performance: combining eye movement and concurrent verbal protocol data. Applied Cognitive Psychology, 19, 205-221.
Weinel, M., Bannert, M., Zumbach, J., Hoppe, H. U., & Malzahn, N. (2011). A closer look on social presence as a causing factor in computer-mediated collaboration. Computers in Human Behavior, 27, 513-521.