HY
PER
C
U\B/E
S
LEARNING COMPUTATIONAL THINKING
THROUGH EMBODIED SPATIAL PROGRAMMING
IN AUGMENTED REALITY
Anna Fuste Lleixa
BEng., La Salle, Universitat Ram6n Llull, 2013. BComms., Universitat Pompeu Fabra, 2010.
Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning,
in partial fulfillment of the requirements for the degree of Master of Science in Media Arts and Sciences
at the Massachusetts Institute of Technology September 2018
@2018 Massachusetts Institute of Technology. All rights reserved.
Signature of Author:
Signature redacted
Anna Fuste
Lleixa
Program in Media Arts and Sciences September 2018
Signature redacted
Certified by:
Chris Schmandt
Principal Research Scientist
Thesis Supervisor
Signature redacted
Accep~ted by: Thrd d- nuc r MA1HUtTI IbNoSTIITUTE OF TECHNOLOGYOCT
16
2018
LIBRARIES
ARCHIVES
Academic Head, Program in edia Arts and SciencesHYPERCUBES
Learning Computational Thinking through Embodied
Spatial Programming in Augmented Reality
Anna Fuste Lleixa
ABSTRACT
Computational thinking has been described as a basic skill that should be included in the educational curriculum. Several online screen-based platforms for learning computational thinking have been developed during the past decades. In this thesis we propose the concept of Embodied Spatial Programming as a new and potentially improved programming paradigm for learning computational thinking in space. We have developed HyperCubes, an example Augmented Reality authoring platform that makes use of this paradigm. With a set of qualitative user studies we have assessed the engagement levels and the potential learning outcomes of the application. Through space, the physical environment, creativity and play the user is able to tinker with basic programming concepts that can lead to a better adoption of computational thinking skills.
HYPERCUBES
Learning Computational Thinking through Embodied
Spatial Programming in Augmented Reality
Anna Fuste Lleixa
This thesis has been examined by the following reader:
Dr. Pattie Maes
Professor of V TeehnoLogx, MIT Media Lab.
Signature redacted
MIT Prog iArts and Sciences.HYPERCUBES
Learning Computational Thinking through Embodied
Spatial Programming in Augmented Reality
Anna Fust6 LleixA
This thesis has been examined by the following reader:
Amit Pitaru
Creative Director, Google Creative Lab. CoFounder, School For Poetic Computation.
Signature redacted
ACKNOWLEDGEMENTS
These have been two incredible years that have deeply inspired me; not because of the place or the projects around me but because of the amazingly talented people that have influenced and helped me evolve in such thriving environment as the MIT Media Lab.
First and foremost, I would like to express a special thanks of gratitude to my mentor, Chris Schmandt. Thank you for accepting me into the group. Your help and feedback have allowed me to make the right decisions to reach a higher quality of research. Thank you for always being straight forward and for giving me so much freedom to explore. Thank you as well for showing me the desert and the mountains and for believing in me and my ideas every step of the way.
I would also like to thank my two readers Pattie Maes and Amit Pitaru. Pattie, your inspiring mentorship and your guidance have given me the perfect criticism to perfect my research and ask the right questions. Amit, thank you for giving me the opportunity to spend some time at the Creative Lab where all this thesis was originated. Your feedback has inspired me to look further and beyond.
This thesis project also had the opportunity to be evaluated through a set of qualitative user studies as a deployment in a non-lab setting, thanks to a special group of people. I would like to thank first Luba Elliot, Tracy Harwood and Tina Barton for inviting me to the Leicester Art-Al Festival and for allowing a set of workshops with the most amazing kids. It was a pleasure to attend and witness their reactions to my work. I would also like to thank Parts & Crafts for having me during their Girls Invention Week and allowing the children to play with the project. Finally, I would like to thank the MIT Museum for being interested in my work and letting me perform a workshop at their IdeaHub program where I had the chance to test the application and record the families playing with it.
A massive thank you to all my lab friends that have been there during fun times and hard times these past two years. To Javi and Oggi, for your advice in my research but more importantly for making me laugh, for your constant visits to my office, for teaching me about affective computing and personalized machine learning and for supporting me during my stay at the lab.
To Pedro, Nikhita and Chrisoula. 'Papa', thanks for always being there, for giving me plenty of good moments and helping me through the struggle. Nikhita thanks for all the support, the 'calm down' talks and for inspiring me with your amazing design and art skills. Chrisoula you have been an inspiration and I have admired you since I met you. Your talent and your resilience will take you very far. Thank you for being my rock.
To my best friend Judith. You not only give me motivation for my research work but you are also an inspiration in my life. We both started this research together and it is thanks to you that this project has been possible. Thank you for believing in me and for inspiring me in so many different domains in my life.
To my grandparents, my uncles, aunts and cousins. My work reflects the values and principles that I have gained during my entire life and most of them are influenced by all of you. A big thank you for shaping all these values and for supporting me in every decision even if it means being far away.
To my sister Laura and my parents Teresa and Jordi. Laura, you are and always will be my soul mate and my other raindrop. No matter how far you are you will always inspire me to thrive. Mama, thank you for your encouragement, for guiding me in my research and for all the love you give to me. Papa, thank you for believing in me in every step that I have taken and for all your excitement and admiration towards me. I look up to the three of you. You have made of me the person that I am today and I deeply thank you for that.
And finally, a tremendous thank you to my husband, Jordi. Thank you for everything you do for me every day. Your constant support and the love you give to me keep me pushing forward and being creative in all the work that I embark on. You inspire every single one of my movements and decisions and give me all the happiness in this life. You give me all I need and I thank you for that.
TABLE OF CONTENTS
1
INTRODUCTION
9
1.1
Motivation
9
1.2
Research goals
11
1.3
Thesis structure
12
2
BACKGROUND
14
2.1
About experiential learning and computational thinking
14
2.1.1 Towards a constructionist theory of learning 14
2.1.2 Learning through play 18
2.1.3 Computational thinking 21
2.1.4 Learning about programming 26
2.2
About space, tangibility and Augmented Reality
29
2.2.1 Space and embodiment for STEM 29
2.2.2 Tangibility and materiality 33
2.2.3 Augmented Reality 37
3
EMBODIED SPATIAL PROGRAMMING
42
3.1
Humans and our spatial mental models
42
3.2
Augmented reality, the body and space
48
3.2.1 Do not gesture nonsense in space 48
3.2.2 Towards a graspable future 52
3.3
Authoring to own the medium
55
3.4
Embodied spatial programming for computational thinking
57
4
HYPERCUBES
61
4.1
Overview
61
4.1.1 HyperCubes outline 61
4.1.2 DIY culture and maker culture 61
4.1.3 Tangibility and raw materials 63
4.1.4 Mobile device as an accessible medium 63
4.1.5 Embodied Spatial Programming in AR 64
4.1.6 Paper Cubes and HyperCubes 64
4.2
Application design
66
4.2.1 Learning computational thinking with HyperCubes 66
4.2.2 The cycle as a programming loop or main function. 80 4.2.3 Cubes and Parameters as Functions and Variables 81
4.2.4 Cubes as Programming Functions and Events 86
4.3 HyperCubes as an authoring tool for creativity 88
4.3.1 Playfulness and creativity for learning 88
4.3.2 Low floors and high ceilings 88
4.3.3 Collaboration in space 89
4.4 System overview and implementation 90
4.4.1 Design and UX 90
4.4.2 Main Software Architecture 94
4.4.3 Augmented Reality technologies 96
4.5 Evaluation 98
4.5.1 Pilot study 98
4.5.2 Qualitative user study 104
5
CONCLUSIONS
111
6
FUTURE WORK
114
7
REFERENCES
116
8
TABLE OF FIGURES
121
HYPERCUBES
Learning Computational Thinking through Embodied
Spatial Programming in Augmented Reality
1
INTRODUCTION
1.1 Motivation
Since Logo (Papert, 1980), the first programming language to teach programming skills to
children, a great number of tools have been developed to improve computational literacy.
Computational thinking is now considered a basic skill that should be included in the
educational curriculum in schools (Lockwood & Mooney, 2017). It has also been regarded
as a learning framework that can incorporate other basic subjects such as mathematics,
engineering or science (Wing, 2006).
The majority of the tools that are being developed for computational thinking are
screen-based online tools such as Scratch (Resnick, et al., 2009) and Alice (Cooper, Dann, &
Pausch, 2000). In making use of these tools, the child engages in an activity in front of a
screen that isolates her from her physical surroundings.
These experiences make no use of space and embodiment, which are regarded as basic
and crucial to the learning process by well-known cognitive and psychological theories (Antle,
2009; Newcombe, 2017; Wai, Lubinski, & Benbow, 2009). As Antle (2009, p. 27) explains,
"An embodied approach suggests that humans should be considered first and foremost as
active agents rather than as disembodied symbol processors." She also highlights "The
importance of understanding the role of action and the environment in the development of
children's thinking skills."
Parallel to this, Augmented Reality (AR) has gained a lot of attention in recent years with
improved hardware and software and new devices for the masses. New gestural interfaces
have been designed that do not take advantage of the physicality of our environment and
our human spatial mental models. AR is a very powerful technology that lets us make use of
9
the flexibility of the computational medium without losing the physical texture of our environment. We can still interact with real objects instead of just staring at a flat screen. We can grasp and hold things and we can have a digital layer that provides an information bridge to our mental constructs.
To this end, we propose the concept of Embodied Spatial Programming as a framework to learn computational thinking. The principles of Embodied Spatial Programming are based on the idea of using physical tokens and objects to program digital interactions in space. AR's capability to bridge the physical and virtual worlds makes it a suitable technology to apply Embodied Spatial Programming principles to the interaction design on new educational interfaces and applications.
We also present HyperCubes as an example application of Embodied Spatial Programming. HyperCubes is an AR platform for children to create and learn by spatially orienting handmade paper cubes. HyperCubes sets a successful precedent for the advantages of applying Embodied Spatial Programming to an AR creation and educational tool.
With new technologies that make use of space and the physical environment of the user, such as AR, we can create improved interactions to teach computational thinking to new generations, using the full agency of their bodies, without losing the benefits of the computer as a medium.
1.2 Research goals
The main research goals of this thesis are the following:
1. Propose a new interaction framework for learning computational thinking that makes use of space and the physical environment of the user.
2. Develop and validate this new interaction model with an Augmented Reality authoring platform using physical paper cubes in space with the aim of teaching computational thinking concepts in a creative and playful environment.
The first research goal focuses on the idea of designing a new interaction framework based on a body of work related to spatial thinking and user interaction with the environment. Considering the background work and previous interaction theories, we want to propose an improved interaction model that makes use of space and the physical environment in order to design programming interfaces.
The second research goal is focused on trying to consolidate this model by designing and validating an AR authoring platform that makes use of the principles of this interaction model. The main intention is to have a solid project based on our theory to be able to test it on a non-laboratory setting where we can evaluate and get some feedback on the design advantages or disadvantages of the proposed model.
The evaluation performed for the project is a qualitative evaluation adjusted to the thesis scope. This qualitative evaluation has given us a sense of the positive and negative impact that this interaction model can have on the user. In order to demonstrate learning outcomes, we will need to perform a quantitative evaluation and more qualitative assessments as discussed in the Future Work section.
1.3 Thesis structure
The thesis starts off with an overview of the related work and theories that have been explored in the past. This background chapter is split in two sections.
The first section is focused on two main points: giving a background about psychology theories related to learning and defining computational thinking as a key subject that should be included in the educational curriculum.
The first subsection goes through well-known cognitive development psychology theories such as the Constructivist theories of Jean Piaget or the Constructionist theories of Seymour Papert. The second subsection talks about the creative learning approach of Mitchel Resnick and gives some examples of this approach such as the Explorable Explorations platform. In the third subsection, computational thinking is explained and examples of its application are mentioned. The last subsection dives into the concept of learning about programming, what it means and why is it important.
The second section of the background is focused on space, tangibility and Augmented Reality. It starts with a subsection on Visual Programming as the seed to using space for our everyday interfaces. It later goes on to explain cognitive theories that put spatial thinking as a key framework for learning in STEM fields. It then explains tangibility and materiality as two key aspects when designing interfaces. Finally, Augmented Reality is presented as a recently boosted technology that can help leverage all the cognitive spatial and tangible models that humans already have.
The second chapter of the thesis proposes Embodied Spatial Programming as a framework to design new interfaces for learning about programming. The first section talks about the spatial mental models that humans develop and why and how we can take advantage of them. The second section talks about Augmented Reality as a powerful technology for leveraging these spatial mental models and it presents good and bad practices that have already been seen using this technology. The third section talks about the need for developing tools native to this new spatial medium and the importance of these creation tools. The final section dives deeper into the concept of Embodied Spatial Programming, its definition and uses.
The third chapter is focused on HyperCubes, an example application that makes use of Embodied Spatial Programming. This chapter contains a first overview section with an introduction to the application, why it was developed and what principles were used in its design. The second section explains the application design in more depth and explains how the application works. The third section talks about the system implementation and the technologies used for the development of the application. The last section explains the evaluation performed of the application and gives some initial qualitative results extracted from the interactions in a pilot study and in a user study.
Finally, the thesis presents a Conclusions section with final remarks, lessons learned and the possible improvements for the project. After the Conclusions, a Future Work section talks about the potential for the application and what the next steps are in order to improve it and take it further.
2 BACKGROUND
2.1 About experiential learning and computational thinking
The educational curriculum has evolved through the last decades towards learning-by-doing methodologies and experiential learning approaches. Different psychology theories have tried to find a proper means of improving the cognitive development of a child considering different maturity stages and levels. Technology has played a key role in this evolution, giving place to a set of good practices that take advantage of the computational medium but also a vast number of methodologies that lack some of the essential ingredients for the proper cognitive development of a child. It is worth reviewing all these theories and approaches in order to design improved interfaces that can help advance the educational curriculum.
2.1.1 Towards a constructionist theory of learning
Piaget's theory of cognitive development
"Each time one prematurely teaches a child something he could have discovered himself, that child is kept from inventing it and consequently from understanding it completely." - Jean Piaget
In order to build a framework for a learning environment, one needs to fully understand all the factors, context and processes that will affect a child during the course of her cognitive development cycles. There has been a vast body of work exploring the mental processes of the development of a child. Based on the work of Jean Piaget, the father of constructivism, others focused on a more practical approach such as the constructionist theories of Seymour Papert or the creative learning approaches of Mitchel Resnick. It is worth reviewing all these theories in order to be able to design proper systems founded on the psychology of the human and from the ground up.
Jean Piaget (Piaget, 1971) is considered one of the main pioneers of constructivism in the psychology field. He introduced his theory of cognitive development based on the idea that
the development process is conformed by a cycle of steps. These steps are based on experience and experimentation with actions and they build up after each cycle. He claimed that children had an active role in the learning process and that through experimenting and play, they would build upon the knowledge they had already acquired.
Piaget presented four different stages in the development of a child: - "Sensorimotor" stage: birth to 2 years
- "Preoperational" stage: ages 2 to 7
- "Concrete operational" stage: ages 7 to 11 - "Formal operational" stage: ages 12 and up
In the first "sensorimotor" stage the child needs to be in contact with her physical surroundings. Piaget assumed that the mind in a child developed endogenously until the infant had the cognitive capacity to assimilate language. In the "preoperational" stage, the child begins to learn to speak, however, she is not able to reason or apply logic. It is a stage that he considered related to play and pretending. In the "concrete operational" stage, children can think logically, but their reasoning is limited to the physical possibilities. Finally, in a more advanced stage, the "formal operational" stage, the child can deal with symbols and abstract thinking.
Years after, researchers would start realizing how these stages should not be treated as sequential but rather parallel ongoing events. The acquisition of sensorimotor skills and the principles of cognitive development were presented as events that should be intertwined in order to boost the learning processes.
The constructionism
of Seymour Papert
Following Piaget's theories of constructivism, Seymour Papert developed what he called constructionism. Constructionism is a ramification of constructivism and is based on the idea that children learn by creating mental models in their brains drawing from experience and action. Discovery learning is key and it builds upon knowledge previously acquired. Papert was a big advocate for project-based learning. He was a mathematician, computer scientist and educator and one of the pioneers of artificial intelligence. He spent most of his career
15
working on learning theories and used new technologies as a playground for the
Constructionists learning methods.
The principles of constructionism say that the learning process is more effective when
students are engaged in making and interacting with tangible objects in the physical
environment. As Papert (Papert, p. 1, 1987) defined the concept himself:
"The word constructionism is a mnemonic for two aspects of the theory of science education
underlying this project. From constructivist theories of psychology, we take a view of learning
as a reconstruction rather than as a transmission of knowledge. Then we extend the idea of
manipulative materials to the idea that learning is most effective when part of an activity the
learner experiences as constructing a meaningful product."
-
-
~ir0
Figure 1. From left to right, Seymour Papert with the Logo Turtle'
and two children playing with it2
Papert strongly advocated the use of the computer in the classroom. Constructionism was
initially used for teaching mathematics, however Seymour Papert started putting emphasis
1 From Matematicamente.it (2014)
on what he called computational thinking as a new way of engaging with the new era of technology. Several programming languages and environments have been developed as platforms for Constructionist learning and computational thinking. Papert himself developed the language Logo which introduced the turtle concept [Figure 1] to elementary schoolchildren. Other languages and platforms following the constructionist principles are SmallTalk (Kay, 1996), Etoys (Kay, 2005) and Scratch (Resnick, et al., 2009).
2.1.2 Learning through play
Lifelong Kindergarten
Mitchel Resnick, professor at the MIT Media Lab, starts his book Lifelong Kindergarten (Resnick, 2017) by giving the example of Chen Jining, the president of Tsinghua University, the leading engineering university in China. He explains how Chen Jining was worried that the Chinese students, although having very high standards and good grades, were not really prepared for the needs of our constant changing and evolving society. The students knew the theory, they knew how to solve math problems, but they didn't have the ability to think creatively, to ask their own questions, to pose their own queries and solve them. Resnick, influenced by the constructionist theories of Papert, talks about the need to forge a new generation of students that are curious, that think further and can formulate new challenges built upon the knowledge that they acquire. In order to prepare students for this ever-changing future, we need to carve creative thinkers.
Resnick runs his group Lifelong Kindergarten at the MIT Media Lab, where he works towards a vision of the educational curriculum much more open to creativity and free play. He talks about the creative process as a "Creative Learning Spiral" [Figure 2] significantly related to the cognitive development theories of Piaget and the constructionist theories of Papert. In his "Creative Learning Spiral" there are five different stages: "Imagine", "Create", "Play", "Share" and "Reflect". It is defined as an infinite cycle that builds up at each iteration:
01(
Figure 2. Creative Learning Spiral by Mitchel Resnick3
At the Lifelong Kindergarten group, Resnick has developed a framework for creative thinking
based on four basic principles [Figure 3]: "projects", "passion", "peers" and "play". Resnick
explains how creating "projects" is essential for the Creative Learning Spiral and highlights
some of the examples in the Scratch community where users create projects constantly that
help them develop their skills and creativity. As Piaget said: "Children learn best when they
are actively engaged in constructing something that has personal meaning to them", that is
why "passion" is fundamental for the creative process. "Peers" refers to the collaboration and
sharing of projects with others. As Resnick states: "Creativity is a social process". Finally, the
last core is "Play". Playful experimentation is key in order to successfully thrive in the creative
process.
projects
- *bpla~y
passion
peers
Figure 3. Projects, Passion, Peers and Play by Mitchel Resnick'
Explorable Explanations
This playful experimentation is key for the learning experience and has been included as a
guiding principle in many learning environments. An interesting example of using play for the
learning process is found in Explorable Explanations. Explorable Explanations is defined as a
community of artists, coders and educators that develop interactive projects to learn through
play. As Bret Victor explains (Victor, 2011), Explorable Explanations is an "umbrella project
4 From Resnick, Mitchel (2017). Courtesy of the author
for ideas that enable and encourage truly active reading". Bret Victor talks about active readers as the ones that "ask questions, consider alternatives, question assumptions, and even question the trustworthiness of the author". This definition aligns considerably with the new generation of creative thinkers that Resnick talks about. In Explorable Explanations, the user can learn about a topic by engaging with an interactive platform or project that allows for play and exploration. After developing some of the Explorables in the platform, Nicky Case (2018) talks about four design patterns for teaching through play that he has been able to extract from the Explorable Explanations experiments. These design patterns are the following ones:
- "Puzzle it out": Case defends the idea that puzzles make the user think. And by thinking the user can learn and assimilate new concepts.
- "Place your bets": let the user future-cast what the solution will be and compare her solution to the correct answer so she can see how her expectations match or don't match reality.
- "Role play": this design patter helps not only using empathy but making the user think about ideas, reasoning and pondering on issues and situations.
- "Sandbox mode": this design pattern is based on giving the user a simulation to play with. Case highlights the fact that this pattern needs a ground-up approach where the user starts off with a very basic simulation and everything leads to the whole sandbox.
These patterns define a practical framework guided by the principles of learning through play that Resnick defends. These practical guidelines are some of the basic foundations that should be considered when designing new learning environments for playful experimentation.
2.1.3 Computational thinking
Origins and definition
"If children became fluent in thinking in the medium of the computer; If programming was a form of basic literacy like reading and writing, they'd become adults with new forms of critical thought." - Alan Kay
The term computational thinking was first introduced by Seymour Papert in his book
"Mindstorms: Children, computers and powerful ideas" in 1980. Since then, there has been
a significant debate on what exactly computational thinking is, how we can benefit from it
and why we should learn about it.
"Mindstorms" marked a turning point in the educational landscape. Teachers and educators
started introducing computers in the classroom as a new creative pedagogical tool. The
resilience of the computational medium offered limitless possibilities for children to explore
new ideas while boosting their cognitive development. Papert invested much of his time
working on his platform Logo, that he designed by applying computational thinking principles
for children. Logo allowed children to program using a high level language to draw shapes
on the screen [Figure 4] or drive a robotic turtle that drew them on the floor.
hal f-petal draw arc 50 rotate left 45 and
C
petal draw half-petal rotate left 100 draw half-petal end flower petal rotate right 90 petal rotate right 90 petal rotate right 90 petal rotate right 90 end new-flower repeat 0 petal rotate right 360/S andNOTE Logo code was edited for clarity
Figure 4. An example of a Logo problem to draw a flower5
, From Ferster, Bill (2013). Adapted from Papert (1993). Retrieved from flickr.com
As Papert introduced it, the term computational thinking not only is a way of conceiving and unraveling the computational medium. It is also a way to enhance the understanding of our reality and the way we interact with it. It is a framework for our mind to help us in being better problem solvers and creators.
Skills and approaches
Computational thinking defines a set of skills and approaches that go beyond basic coding skills. Most of these skills and approaches can be found in a regular computer science pedagogical curriculum but can be applied in any situation where a person may need to learn or carry out a task. These are the skills and approaches [Figure 5] that I consider to be more significant and that I put together based on the "Mindstorms" book and other sources (Resnick, 2017; Bers, 2017). The skills that constitute the basics of computational thinking are the following:
- Logic: we are able to reason to solve a problem, situation or procedure. - Decomposition: we have the ability to break problems down into parts. - Pattern recognition: we can find similarities around us and predict events. - Abstraction: we are able to work with broader ideas and symbols.
- Algorithms: we are capable of developing abstract computation sequences. - Simulations: we can create rules and put them to practice in parallel environments.
All these abilities can be mastered by taking some basic approaches that boost their strength such as tinkering, creating, persevering, debugging or collaborating. Modern educational platforms have taken into account all these approaches to build systems that enhance the child's cognitive development (Cooper, Dann, & Pausch, 2000; Lockwood & Mooney, 2017).
Tinker Logic Creae - i'~aooDecomposition create 0 \ Patterns Persevere Computational Thinking Abstraction Debug Algorithms Cottaborate 0Simulation
Figure 5. Computational thinking skills and approaches
Beyond Papert
"Children learn best when they are actively engaged in constructing something that has personal meaning to them - be it a poem, a robot, a sandcastle or a computer program." - Seymour Papert
Logo (Papert, 1980) was the first programming language created for teaching Computational
Thinking concepts. Since its creation, numerous other platforms and programming
languages have been developed with the same objective (Cooper, Dann, & Pausch, 2000;
Kay, 2005). Some of these platforms draw from Papert's constructionist learning approach,
such as Scratch (Resnick, et al., 2009), a free programming language and online community
where users can create their own interactive stories, games, and animations [Figure 6].
23
AFiure Crab 5nifa=
Lnaws
*~f
2Sond em, & G 2Ld
ch*rnuin opttonltikn c lOpts~ [Fgue ]
th trurlsin erm, f
Figure 6. Scratch Interface 6
Scratch and many others make use of visual programming. Alice (Cooper, Dann, & Pausch,
2000), AgentSheets (Repenning, 1993), MIT App Inventor (Pokress & Dominguez Veiga,
2013), 0 Unplugged (Bell, Alexander, Freeman, & Grimley, 2009) or the LEGO Mindstorms
(Kiassner & Anderson, 2003) are just some of the platforms focused on teaching coding to
children using computational thinking concepts [Figure 7].
(~~rIr~f~~ MIT Q AgentSheets
APP INVANTOP mind5STersmrn Afice4J.
Figure 7. Computational thinking pedagogical platforms7
Likewise, during these past decades several research studies and experts have claimed how
computational thinking is essential for the development of children. Since Papert introduced
the term, the research community has been pushing for the inclusion of computational
thinking into the educational curriculum. However, although a lot of work has been done
SFrom Scratch.mit.edu (2018)
SFrom Scratch.mit.edu / Appinventor.mit.edu / Osunplugged.org / Lego.com / Agentsheets.com /
Alice.org (2018). Logos retrieved and adapted.
around the world in many different educational contexts, the work relating to computational thinking incorporated in the classroom is still in its infancy (Lockwood & Mooney, 2017). The main reason lies in the fact that we need children to engage and play with their environment. And having them in front of a screen typing lines of code is not ideal for their own development. The vast majority of these educational tools are developed in 2D environments where the child is interacting solo with a screen and not making use of the space around her. Others such as the LEGO Mindstorms, are based on robotics and mechanisms and the programming gets limited by the physical affordances.
There are many open paths and opportunities to explore and there is also plenty of room for improvement in the educational arena. Computational Thinking conforms a framework that not only is good for learning about the computing medium but can also become a solid infrastructure for learning in other areas such as Mathematics, Physics, Literature or Art.
2.1.4 Learning about programming
Computational Thinking is much more than knowing how to code. Computational thinking is a way of understanding and interacting with the world. Computational thinking does not mean "thinking like a machine", the same way as learning about programming does not mean learning how to code. As Bret Victor says (Victor, 2012): "A person is not a machine, and should not be forced to think like one". Learning about programming means being able to understand certain coding constructs that will boost our way of approaching a problem and finding a solution. As we have already mentioned, Papert put a lot of effort in developing Logo, a programming language intended for educational purposes and focused on learning about programming. Several other pioneers in the field have presented other platforms and languages as well as relevant research on the matter of learning about programming.
One of the most noticeable examples was developed by Alan Kay, a computer scientist who has dedicated great part of his life to the development of educational technology. Kay designed the Dynabook concept, in 1968 (called The KiddiComp back then), as "a personal computer for children of all ages" (Kay, 1972). The Dynabook concept was conceived way before the first tablet was released to the market. He also developed various programming languages and educational platforms such as Etoys. Etoys is an authoring environment and visual programming system for children built in Squeak, an open-source implementation of SmallTalk. Etoys served as an inspiration for the development of Scratch later on. Kay didn't envision Squeak and Etoys as tools to learn how to code. He designed them as platforms for a greater purpose; platforms to learn about programming as an umbrella for the assimilation of many more subject areas such as physics or mathematics.
Kay gives and example of a typical 5th grade math-science project sequence in his research note "Squeak Etoys, Children & Learning" (2005):
" draw objects (such as cars)
* write scripts to move them, have them follow edges, etc. Every fifth frame of video
Dropping objects
think about speed and acceleration by leaving trails
observe moving objects in the real-world: dropping weights, etc. get detailed records of the drop using video cameras
* look at every 5th frame and measure the speeds
* compare the difference in speed between each frame
-
see that the differences (the acceleration) is constant
-
write a script to move a simulated ball with constant accelerated motion
* find the constant that will match up the simulation with the real-world
It is clear how the student not only learns about scripting but explores and tinkers with a
wider range of concepts and transversal competences such as physics, mathematics, writing
or documenting. Kay has done substantial work on the area of visual and interactive
programming. He created SmallTalk as an educational programming language that was
object-oriented, dynamically typed and reflective tailored to the new 'human-machine
symbiosis' era [Figure 8].
Figure 8. SmallTalk being used on an Alto computer, by children
8All these characteristics are basic to design a technological platform for learning. As Bret
Victor points out (Victor, 2012), a platform for learning about programming should aim for
these two main goals:
"
Support and encourage powerful ways of thinking
-
Enable programmers to see and understand the execution of their programs
8 From Computer History Museum (1970) PARC (Palo Alto Research Center, Incorporated)
27
A system to learn about programming should be reflective; affording the ability to examine, introspect and modify its own behavior at runtime. And this, not only applies to a platform to learn about programming but also any other natural interface. A digital drawing program should also be reflective, such as Sketchpad, the first graphical computer-aided design program or ChalkTalk (Perlin & Nunes, 2017), a conversational and performative sketch language [Figure 9]. ChalkTalk was designed and implemented by Ken Perlin. ChalkTalk is an interactive blackboard where the user can draw glyphs and they come to live. It is meant to be used as an educational and conversational platform to teach and discuss topics.
Figure 9. ChalkTalk sketch example9
Michael Nielsen put it in these words when examining the ChalkTalk system (Nielsen, 2017): "There is a sweet spot for internalizing operations: they must be performable on a timescale of no longer than a few seconds. That is, there is a natural speed of thought, and what we internalize must match this speed."
The best way of learning what gravity is, is to release an object in mid-air and observe what happens next. The same applies to programming. If we want someone to learn programming constructs we need a system that can react instantaneously to the user actions the same way as objects in the physical world react instantaneously to gravity.
2.2 About space, tangibility and Augmented Reality
2.2.1 Space and embodiment for STEM
Visual programming
The research community has put emphasis on the importance of space and embodiment used in educational methodologies for STEM fields. The body and the relationship with the environment is not only essential for the development of motor skills but it can greatly benefit the understanding of more abstract concepts such as mathematics or physics. We humans exist in a three dimensional world and our brain takes this 3D environment as a reference. This reference has served in the past and has be considered as a powerful conduit to teach science and technology concepts to children. The use of spatial principles to make an interface more usable and instinctive is not something new in technology systems. This idea has been applied to computer interfaces since the start of the computing era. Spatial information and awareness has been used as a tool for designing interfaces better adapted to our human nature.
One of the clear examples of this is visual programming. Visual programming is a powerful and prevalent programming paradigm that has been evolving since the 60s. It makes use of visual references and blocks arranged in 2D space in order to program a regular sequence of commands. Visual programming is the first programming paradigm that leveraged our spatial mental schemes in order to design a better interface for a sequential task. The first visual programming interface was created in 1968. Back then Tom Ellis created GRAIL (Ellis, Heafner, & Sibley, 1969), GRAphical Input Language, where users draw flowcharts in order to write software [Figure 10].
Figure 10. GRAIL Interface (1966)
Since then, plenty of other programming platforms and environments have been developed,
specially for learning about programming using spatial principles. An example of this is
AgentSheets (Repenning, 1993), a domain-oriented visual programming system; VVVV
(VVVV group, 1998) a node based programming platform to program graphics; or the Reality
Editor (Heun, Hobin, & Maes, 2013), a tool to program smart objects in Augmented Reality.
From action to abstraction: 'Doing with Images makes Symbols'
Kay worked on trying to apply the principles of visual programming to learning environments
for children. Inspired by the theories of Seymour Papert and Jean Piaget, Kay tried to
understand how spatial notion was essential when abstracting from reality and trying to think
in the computational medium.
If we look back at Piaget's theories of cognitive development, he explained how there are
different stages to the development of a child. A first sensorimotor stage where the child
needs to be in contact with his/her physical surroundings; a more advanced stage where the
child can recognize patterns and the final stages where the child can deal with symbols and
abstract thinking. Years after, researchers started realizing how these stages should not be
treated as sequential but rather parallel ongoing events.
To make a simile with Piaget's theories, Alan Kay (Kay, 1987) talked about the idea of 'Doing
with Images makes Symbols' [Figure 11]. As mentioned before, Kay is a computer scientist
and he is well-known for his work on visual programming languages. He tried to explain how
the learning process would become much richer if we combined Piaget's learning stages
altogether. If children could make use of space, the physicality of their environment as they
can recognize patterns and understand symbols, the learning process would be much richer
and faster.
DOING
with
IMAGESmakes SYMBOLS
From Piaget to Alan Kay
The body
representation
The visualK
The language
X~Y 7
0
Figure 11. Doing with Images makes Symbols"
A lot of the inspiration for Kay's work came from Papert's theories. Papert talked about the
idea of body-syntonic learning. He realized how children were able to reason using their
bodies and create programs for the Logo Turtle by making a simile with their movements in
space. Embodiment was key in understanding some of the programming constructs in the
Logo environment. He described how concepts such as variables or recursion could be more
naturally internalized by using the body and space. For example, a child tried to program the
turtle to make a circle. The child would then see how if she moved forward one step and
rotated x degrees and then repeated that sequence over and over she would be able to
complete a circle in space.
" Adpated from Kay, 1987. "Doing with images makes symbols"
Spatial thinking for STEM fields
Nora Newcombe is a researcher in cognitive development and cognitive psychology. Her body of work focuses on spatial reasoning and the development of spatial representation. She talks about the importance of spatial thinking for STEM fields, and how it relates to episodic memory (Newcombe, 2017). She conducted several research studies where she found that children that play with blocks and puzzles develop a better spatial reasoning ability. And this spatial reasoning ability is very beneficial for STEM education. Mathematics or physics are great examples where spatial reasoning is often essential to understand abstract concepts. Newcombe talks about more recent studies where there is evidence of the causal links between spatial thinking and STEM fields. She highlights how these studies "show that high spatial ability early on as a child can predict later interest and success in STEM fields".
Newcombe presents a view of spatial reasoning inspired by Piagetian theories but opposing some of their principles. She talks about the fact that infants are born with spatial coding abilities that evolve during development and life. This view contradicts the Piagetian view that focused more on the idea that children only developed spatial reasoning in the late stages of childhood.
This spatial information encoded in the child's mind since birth and the evolution of spatial reasoning during childhood is what can be leveraged in order to develop and acquire new knowledge, especially in the technology and science fields as Newcombe points out.
2.2.2 Tangibility and materiality
Tangible user interfaces
In parallel to all the theories relating to spatial reasoning and cognitive development, there
has been a lot of research trying to create learning experiences that enhance each one of
the Piagetian development stages separately. But not much effort has been put in trying to
combine them together for learning. Current approaches for teaching computational thinking
and programming are online screen based. However, we are physical creatures. As Bret
Victor mentions (Victor, 2011), 'We live in a three-dimensional world. Our hands are designed
for moving and rotating objects in three dimensions, for picking up objects and placing them
over, under, beside, and inside each other. No creature on earth has a dexterity that
compares to ours'. Piaget also advocated for introducing physical interactions when
designing educational methods. He highlighted the importance of physical actions towards
objects in order to abstract from reality: "...there are individual actions such as throwing,
pushing, touching, rubbing. It is these individual actions that give rise most of the time to
abstraction from objects"".
Following these same principles, Graspable User Interfaces were created. They are
nowadays called Tangible User Interfaces (TUls) and they are based on the manipulation of
physical tokens in space in order to control a digital interface. In 1997 at MIT Media Lab,
Hiroshi
Ishii
and Brygg Ullmer (Ishii & Ullmer, 1997) published an article where they talked
about the connection between bits and atoms. This article focused on the junction between
the cyberspace and the physical environment.
Ishii
and Ullmer pointed out that we had gone
from using physical objects to measure events in the world, to using only computers and
screens as unique measuring tools. In this article,
Ishii
and Ullmer tried to express their
willingness to unify the richness of the physical world with the human-computer interaction.
Tangible User Interfaces are a response to this need. One of the examples they presented in
their research is MetaDesk. MetaDesk [Figure 12] is a platform developed at the Tangible
Media group at MIT Media Lab in order to explore TUls. The platform allows the use of 2D
12
From Jean Piaget Quotes (2016). Retrieved from jeanpiaget.org (2018)
and 3D graphics on a screen, and the manipulation of various physical tokens detected by an array of optical, mechanical and electromagnetic sensors.
passWeLENS aciveLENE
~ ~j
-iistrurmevit phcorm
Figure 12. MetaDesk platform (Ishii & Ullmer, 1997)
Ishii and Ullmer's research triggered the development of a bunch of new projects and more research related to this area. One of the most successful projects using TUIs to the moment is Reactable (Jordt, Geiger, Alonso, & Kaltenbrunner, 2007), a musical edition platform. Several research studies have claimed how Tangible User Interfaces are beneficial for the usability and natural interaction of a system (Schneider, Jermann, Zufferey, & Dillenbourg,
2011). Not only they improve the human-computer interaction methods but they also
increase the learning outcomes if used in an educational setup (Skulmowski, Pradel, KOhnert, Brunnett, & Daniel Rey, 2016).
Tangible programming
Similar to how TUIs can increase the performance in learning environments, there is evidence on how tangible programming can help users in better understand their programming systems (McNerney, 2000).
The vast majority of platforms teaching computational thinking are online screen-based where the student gets disconnected from his own physical reality and isolated from others, sitting stationary looking at their own device. This is why, in parallel to all this movement of
online screen-based systems, researchers have been realizing the importance of embodiment and physical manipulation in learning experiences (Antle, 2009). Tangible programming draws from these cognitive theories and different tools have been developed following these lines of thought. AlgoBlock (Hideyuki & Kato, 1993), Tangible Programming Bricks (McNerney, 2000) or CyberPLAYce (Soleimani, Green, Herro, & Walker, 2016), amongst many others (Bers, 2010; Good et al., 2008; Peng, 2012; Weller, Yi-Luen Do & D. Gross, 2008), are some of the uncountable examples of programming platforms that make use of physical bits in order to teach about programming. If we take AlgoBlock as an example, the system allows students to plan and build procedures in group using physical blocks. The child does not program lines on code on a screen but instead, they use physical bits to design their programs and sequences. They can then observe the outcome of their actions on a screen.
Figure 13. Children using AlgoBlock system (Hideyuki & Kato, 1993)
A similar approach is used by systems like Tern (Horn & Jacob, 2007) or Quetzal (Horn, 2006), more recent programming languages that make use of the same principles [Figure 14]. They were both developed at Tufts University by Michael S. Horn and focus on the idea of making programming interfaces more accessible for the classroom, using simple materials such as paper to create programming inputs in space.
Figure 14. Tern and Quetzal programming interfaces (Horn, Tangible programming with quetzal: Opportunities for education, 2006)
The projects mentioned are good examples of tangible programming. The inputs of the
system are laid in space making use of the physical environment of the user. The output is
visualized on a 2D screen. This detaches the input actions of the user from the results that
the user achieves. The tangible bits that the user can manipulate are physically disconnected
from the digital output that happens in a different environment. In order to have an interactive
and reflective system that naturally reacts to the user actions, the digital content should be
attached to the physical tokens and react to those instantaneously so that the user
understand their functionality automatically. Learning computational thinking with a tangible
programming system can greatly benefit from overlaying digital content in space.
2.2.3 Augmented Reality
Augmented Reality in Education
With the advent of Virtual and Augmented Reality (AR) and the latest advances in image processing and spatial tracking, these new technologies setup an ideal territory for educational experimentation. AR, in particular, lets us put in practice a lot of the concepts related to spatial cognition and cognitive development in the K-1 2 curriculum.
The affordances of AR allow us to take advantage of space and merge the virtual and physical worlds in a way that can greatly benefit educational and creative activities. Some researchers have already explored using AR as a means to improve learning and creativity approaches in education. However, the applications developed so far don't make use of the full potential of this technology. The new tracking systems such as ARCore (Google) or ARKit (Apple) and the new head-mounted displays such as Hololens (Microsoft) or Meta 2 (Meta) that have been released to the market in the last years open a new door to the exploration of new learning methodologies involving space and physical environment.
In different attempts to blend the digital on-screen based approaches with the tangible ones, some researchers have made use of AR as a flexible medium that can leverage spatial thinking capabilities in learning environments [Figure 15]. AR Scratch (Radu & MacIntyre, 2009) is a children's authoring environment for augmented-reality experiences that makes use of the Scratch programming language in AR. The system allows for 2D content in a screen, making use of image markers in front of the computer webcam.
A similar approach more focused on blending the digital with the physical environment is found in Color Code (Rosenbaum, 2010) by Eric Rosenbaum. Instead of image markers, Rosenbaum designed a set of experiments that overlaid digital content depending on color codes in the environment captured by the webcam.
Another example of the use of AR for education is Code Bits (Goyal, Vijay, Monga, & Pratul, 2016). Code Bits is an inexpensive tangible computational thinking toolkit that makes use of flat patterns and AR to teach programming skills. The user only needs paper to create image markers and a smartphone in order to visualize the AR content in space. In this case, the markers are flat and they need to remain in view in order for the application to display the
37
virtual content. The user needs to point at the board at all times in order for the digital content to appear. The application is not spatially aware.
Figure 15. ARScratch (Radu & Maclntyre, 2009), Color Code (Rosenbaum, 2010) and CodeBits (Goyal, Vijay, Monga, & Pratul, 2016) Interfaces
Finally, AvatAR (Fernandez-Baena, Adso, & Miralles, 2014) is an interaction system that makes use of tangible interaction and AR to control the character animation of a virtual avatar. In this case, a physical cube is used to blend between animations and control how the character is animated in 3D. The cube has to remain in view and the application is not spatially aware. A related system is used in SmartAvatars (Amores, Benavides, Comin, Fust6, Pla, & Miralles, 2012) where the user can interact with objects through virtual avatars in space. The application is not spatially aware and the object requires specific electronics in order to get connected to the smart device.
DynamicLand
Another system that makes use of AR for computational thinking in learning environments is RealTalk (Victor, Schachman, Te, Horowitz, & lannini, 2016). RealTalk was released in 2017 and it is "An environment for authoring and using computational media with ordinary physical objects - paper, pebbles, whatever's at hand - recognized and brought to life by technology built into the ceiling". The system was developed by a research group led by Bret Victor. Later on they created a whole environment on top of RealTalk, called DynamicLand (Victor B. , DynamicLand, 2017), a space for collaboration, programming and tinkering. Bret described DynamicLand as "a medium that we can see, feel and manipulate". DynamicLand is a space where people can interact with objects and the physical tokens get merged with the digital content projected from technology installed on the ceiling [Figure 16].
-- -- I
The 2D digital content gets projected onto the surface of a table where the user can interact with it using any type of physical object. The interaction is bound to the table where the technology is set up.
Figure 16. DynamicLand (Victor, 2017)
Spatial Object Programming
AR has been used also as an interface to manipulate our physical environment. The visual component creates a direct link between the digital content and the physical objects linked to it. This is what Valentin Heun explores in his research and the development of his AR system, the Reality Editor (Heun, Hobin, & Maes, 2013). In Heun's words, "If an object can become digitally linked with a virtual object, one can form methods for an expressive form of programming that provides a new form of control over physical space. Such a programming language needs to be easily accessible and operable so that everyone can adjust the world around them".
Heun presented what he called Smarter Objects, which could be programmed with the Reality Editor. The Reality Editor is an AR interface to create programs and manipulate the physical environment around the user [Figure 17]. Heun's goal is "...to create physical objects that have a virtual component to benefit from their physical design while making them
39
W
adaptable and extensible". He thought about the AR interface as a visual elongation of the physical qualities of the objects in the environment.
Figure 17. Reality Editor (Heun, Hobin, & Maes, Reality editor: programming smarter objects, 2013)
Augmented Creativity
Following the principles that define the Lifelong Kindergarten group at the MIT Media Lab, other researchers have used AR to boost creativity and play in an AR environment. ZOnd, Fabio, et al. (Z(ind, et al., 2015) introduced the concept of Augmented Creativity as a way of bridging the real and virtual worlds to enhance creative play. They present different applications that make use of AR to boost the creative abilities of children. In one of their examples a system is developed to enhance robot programming using AR [Figure 18] (Magnenat, Morderchai, Klinger, & Sumner, 2015).
recording time run/pause indicator button robot dropping AR Seven-actions pairs tracking ndicators ocal timeline global timeline
Figure 18. Enhancing Robot Programming with Visual Feedback and Augmented Reality (Magnenat,
Morderchai, Klinger, & Sumner, 2015)
AR is setting the stage for a new era in the educational landscape. The use of spatial awareness and physicality in the digital medium allows for a vast amount of new interactions that remain still to be explored.
3 Embodied Spatial Programming
3.1 Humans and our spatial mental models
We have talked about theories of spatial cognitive development and how different
approaches see the evolution of spatial domains in the brain as an earlier or later event in
childhood. Yet, all of the cited research concurs that spatial models created in our brain from
early stages can be leveraged to enhance other human capabilities linked directly to those
brain areas.
We humans, as a physical entity, live and exist in a four dimensional world. At every
moment in time we exist at a position in space and we relate to objects and other entities
around us that exist at another position in space. Everything we do happens in space. Even
our dreams take place within locations with spatial impressions, though they are abstract
and imaginary. We relate all of our actions to space since the moment we are born. This
leads us to build a set of spatial models in our brain that govern our way of interacting with
things almost in an unconscious manner. When we walk up the stairs, we do not stop to
calculate how far we have to move our leg in order to get to the next step; our leg moves the
appropriate distance because we have a mental model in our brain that has evolved from
necessity and that has permeated in our brain until it is part of it.
Human evolution has shown us that we have benefited from these base spatial models to
build improved interfaces. Let's take an early example, so ancient that its exact origin is still
unknown; the abacus. The abacus is a tool developed for practical mathematical
calculations.
Figure 19. Chinese Abacus13 Figure 20. Modern Abacus (
20' Century)1 4 13
From Wikipedia (2018). Retrieved from en.wikipedia.org/wiki/Abacus
The abacus has a physical and spatial interface. Slide right to add, slide left to subtract in powers of 10 (in more modern versions of the tool). Adding and subtracting are two mathematical constructs that humans found to be abstract and difficult to escalate without a physical concrete representation. The solution to learning these difficult mathematical constructs ended up being an interface that made use of spatial models in our brain. We found a way of using our spatial notion of an object position to represent and operate with mathematical abstraction.
We have used these techniques for years and nowadays rely on them even more. One quintessential example of this is the Graphical User Interface. Graphical User Interfaces were introduced as a way of simplifying the steep learning curve of command-line interfaces. In 1945, Vannevar Bush published an essay that he titled "As we may think", talking about the problem of the current selection and database systems. He highlighted the fact that the human mind works according to association not according to alphabetical or numerical systems. This early essay inspired the creation of SketchPad (Sutherland, 1964), the first graphical computer-aided design program, created by Ivan Sutherland.
Figure 21. Ivan E. Sutherland demonstrating Sketchpad"5
" Inernet Archive (Kay, 1987). Retrieved from Wikipedia (2018).
SketchPad [Figure 21] made use of spatial mental models to design geometry in a 2D
interface. This early vision of a computer interface inspired the creation of the On-Line System (NLS) (Englebart, 1986) in the late 60s by Douglas Engelbart, who introduced the mouse as a new input device to control these 2D interfaces. Similar to the abacus, the mouse allows the user to move their hand along a plane in space to control an information system. It is another example of how we have benefited from our spatial models to control a non-tangible information system.
Later on, Alan Kay developed a new GUI for the Xerox Alto computer [Figure 22] that was based on windows, folders, containers and bins placed in space and controlled by the
mouse, the first computer "desktop".
Figure 22. The first Graphical User Interface on the Xerox Star 8010 (Bewley, Roberts, Schroit, & Verplank, 1983)
This new Graphical User Interface placed elements in different positions in the 2D plane and made use of skeuomorphism to make the items represented resemble their real-world counterparts. As well as making use of visual mental models that the user already had, it made use of spatial mental models in order to manipulate information in the system.
Our interfaces started being three dimensional; like the abacus. We used 3D objects that we could manipulate in space. Yet, with the computing era our main interfaces shifted towards 2D screens anchored in a single point in space where the main computing unit was. The first
comput
