Embodied Learning and Gaming

Exploring the Affordances of Augmented Reality for Learning Abstract Mathematics

1 Augmented Reality

Human beings have been fantasizing about mixing virtual worlds into the real world for decades [Bradbury, 1951; Dick, 1968]. First matters of fiction, these notions became more tangible as researchers started exploring mixed-reality technologies. In 1995, Milgram and Kishino formalised a notion of continuum between reality and virtuality (Figure 1), describing Augmented Reality (AR) as embedding virtual content into the real world [Milgram et al., 1995]. AR should not be confused with Virtual Reality or Virtual Environments: in VR, the real world is occluded from the user’s perception with devices such as goggles or headphones and replaced by a virtual world.

Figure 1: Reality-Virtuality Continuum

On a more technical level, a system is characterized as AR if it (a) combines real and virtual, (b) is interactive in real time, and (c) is registered in three dimensions [Azuma, 1997]. Examples of AR systems can be found on Figure 2.

In this project, we are interested in several properties of AR. AR is embedded in the real world and therefore includes tangible components: these objects can be moved without the need to learn new interaction techniques, they can be contextualised in a way that is meaningful for the activity at hand, they can be shaped and designed to grow specific affordances or affects. Making use of its software component, AR is also interactive: the user can benefit from direct, automated, synchronous, and personalised feedback throughout their manipulation of the system. Finally, AR supports collaboration rather effortlessly compared to other computer-based systems: all the users can see the real world components with their bare eyes, and they can easily access the augmented components by using their own AR device or sharing the device of a nearby user.

Because of these properties, previous researchers also saw great potential in AR for education [Yuen et al., 2011; Ibáñez, 2018], and showed that AR can be used to engage students to explore materials from different spatial perspectives [Shelton et al., 2004]; help teach subjects where students cannot gain real-world first-hand experience [Shelton and Hedley, 2002]; enhance collaboration [Billinghurst and Kato, 2002]; foster creativity and imagination [Klopfer and Yoon, 2004]; and help students take control of their learning [Hamilton and Olenewa, 2011].

AR can be implemented in different ways [Van Krevelen, 2007], the most common instances in education being see-through AR and spatial AR (Figure 2). In see-through AR, a device (mobile device or head-mounted display) is used to capture the environment and display augmentation on top of the video feed. However, in spatial AR, the virtual components are directly projected onto the environment. Because spatial AR comes with many restrictions (sensitivity to light conditions, extra calibration step, cumbersome physical installation), and devices supporting see-through AR are now widely available, we will focus on mobile-based see-through AR throughout this project.

Even in this reduced context, building a meaningful AR activity is not trivial. Many components are involved and need to be considered. In this project we will focus on: the physical objects (tokens, images, or surfaces) supporting the augmentation and keeping it embedded in the tangible world; the visual augmentation itself and its role of representation and visualization; and the Human-AR interaction, including how a user can interact with the augmentation and how the system reacts to the user’s actions (Figure 4).

Getting a better understanding of all these components and their role in the resulting effect of an AR activity is a crucial step towards making AR more consistently beneficial for its users.

Figure 2: See-through vs Spatial Augmented Reality

2 AR for Learning Mathematics

Mathematics is a living subject which seeks to understand patterns that permeate both the world around us and the mind within us [Council et al., 1989].

Learning abstract mathematics is hard. This issue might have several origins. First, because of their abstract nature, the topics of such domain might be difficult to grasp: if one cannot see and cannot manipulate, one might be restricted in their ways of building and grounding knowledge. Second, the pedagogy used to teach these topics also has an impact on the learning outcome: is the teacher only instructing the students? Or do the students have space to explore on their own? Third, the state of the learner, and in particular their beliefs, can impact their motivation, their confidence, and therefore the outcome of the learning activity. Students hold several unproductive beliefs about mathematics [Schoenfeld 1992] that could have a role in this issue, including:

  • Mathematics learnt in school have little or nothing to do with the real world
  • Formal proof is irrelevant in the process of discovery or invention
  • Ordinary students can’t expect to understand mathematics, they just memorize it
  • Mathematics problems have only one correct answer
  • Mathematics is a solitary activity done by individuals in isolation

In summary, the difficulty of learning abstract mathematics might originate from the domain, from pedagogy, or from the learner. In this section we will present different mechanisms of mathematics education that can benefit from AR, and how they can address these three aspects.

In general, the way mathematics education is implemented has been highly criticized. Already in 1954, Pólya mentioned that “mathematics students should be given the opportunity to guess mathematical facts before proving them” [Pólya, 1954]. In 1992, Schoenfeld presents a thorough review of mathematics education and offers new directions such as creating learning environments where students feel safe to try out new ideas; or insisting on the importance of sense-making over accepting the teacher’s solution as a unique truth [Schoenfeld, 1954]. In 2013, Thompson deplores the lack of focus on meaning in american mathematics education and suggests the development of assessments targeting Mathematical Meaning for Teaching [Thompson, 2013]. In 2014, Kapur shows that including a phase of failure-driven problem-solving before instruction (Productive Failure) can greatly improve the conceptual understanding and ability to transfer to novel problems [Kapur, 2014]. We can also clearly see how these methods could also address most of the students wrong beliefs about mathematics; and how AR seems particularly relevant to support such approaches, by providing a self-space to perform sense-making interactive exploration and providing a multi-modal representation of the mathematical situation.

The range in which an individual can develop their knowledge, benefiting from collaboration or expert guidance, is called the Zone of Proximal Development [Chaiklin, 2003]. In this context, we call scaffolding “the process that enables a child or novice to solve a problem, carry out a task, or achieve a goal which would be beyond his unassisted efforts” [Wood et al., 1976]. The idea of scaffolding is to provide support to an individual’s learning process, and to fade away once said individual reaches a level of independence.

Following this definition, see-through AR seems particularly suitable for scaffolding: by using a mobile device, a learner can access and manipulate extra information regarding the topic at hand. Once the student becomes more independent, they can simply stop using the mobile device. Moreover, as much as AR is light in terms of cognitive load compared to computer interfaces [Tang et al., 2003], see-through AR is still physically heavy as one has to carry a mobile device wherever they want to benefit from the AR overlay. We believe that, just as bike training wheels, this physical cumbersomeness could be used to encourage students towards growing independence. Finally, as students use behaviours more similar to face-to-face collaboration when using an AR interface than when using a screen interface [Billinghurst et al., 2002], AR seems to be a good way to benefit from collaboration and social interaction, as well as fading and adaptive scaffolding.

Finally, the context in which learning happens can have a strong impact on the outcomes of said learning. Situated learning can help bridge the gap between the culture in which learning occurs and the culture in which the knowledge will be applied [Bujak et al., 2013]. Moreover, connecting abstract concepts and physical objects can also help support memory and understanding of symbolic representations [Tversky, 2001]. Embodiment has also showed potential with respect to teaching mathematics [Trninic 2012]. In many of these cases, AR can help making these connections explicit. In the specific case of abstract Mathematics, AR can be particularly relevant. Indeed, in 1980 Papert suggested that, as the best way to learn French is still to spend some time in France, a good way to learn Mathematics would be to spend some time in Mathland [Papert, 1980]. Following this idea, see-through AR can be a way to access “Mathland”, using a mobile device as a portal to create an explicit connection between the task at hand and its mathematical meaning, and benefiting  from embodiment and physical semiotics.


Figure 4: Project Structure Overview

In conclusion, AR can support learning by providing a multi-modal embodied or situated experience, and by implementing an adaptive and fading scaffolding to support a learner in their exploration, while addressing students’ beliefs about mathematics (Figure 4). In particular, the difficulties inherent to the domain might be reduced with embodiment and physical semiotics, as well as multi-modal representation, as these features help the learner connect the abstract concepts to the more tangible world. Second, pedagogy would benefit from interactive exploration and fading and adaptive scaffolding, by giving the learner more power and control over their learning. Finally, difficulties arising from the learner’s beliefs can be addressed with interactive exploration as well as collaboration and social interaction, as these mechanisms would make the learner more aware of the different perspectives existing around one problem (Figure 5).


Figure 5: Influence of the different mechanisms on the origins of the difficulty of abstract mathematics education

3 Research Goals

Augmented reality games have the unique capacity to allow novices to experience intellectually productive problems central to science in a psychologically safe place where they can experiment with new ideas and identities and learn through failure [Koutromanos, 2015].

In this project, we will explore how and why an augmented reality learning environment can improve learning and transfer of abstract mathematics concepts. We will focus mainly on Higher Education and mobile-based see-through AR.

Figure 4 shows an overview of the research trajectory. The next sections describe the different phases in more detail.

For each of our steps, the AR learning environments will be evaluated from an Human-Computer Interaction perspective, including standard tests (NASA-TLX [Hart 1988], SUS [Brooke 1996]); and from a Learning Sciences perspective, in terms of learning, transfer, building conceptual understanding, and affecting beliefs students have about mathematics.

3.1 Design an Augmented Reality Learning Environment for Mathematics

The goal of this phase is to design an AR Learning Environment to teach Mathematics to students in Higher Education. For this project, we focus on the learning of Linear Algebra. We believe this topic is particularly interesting to explore, as, although is has a lot of concrete geometrical and physical interpretations, it is still difficult for students to make sense of it. Linear Algebra also has many applications in various fields, such as engineering, physics, or machine learning.

In order to build this learning environment, we will use design thinking iterative methods, taking into account design principles for AR activities in the classroom: integration, empowerment, awareness, flexibility, and minimalism [Cuendet et al., 2013; Kerawalla et al., 2006]

Our design will focus mainly on the three main components of the AR application: the physical objects, the visual augmentation, and the Human-AR Interaction.

For each component, we will identify the different goals and constraints, explore the state of the art, interview target users, and design appropriate solutions. We will then compare the different solutions through different studies.

This design process and follow-up studies will also help us get a better understanding of how to design AR learning environments for higher education mathematics.

Figure 6: Example of see-through AR activity for Linear Algebra

3.2 Understand the Mechanisms Involved in Augmented Reality Learning

Once we designed a system satisfying our education and human-computer interaction goals, we will explore the mechanisms involved in the production of the outcomes, following the idea of conjecture maps [Sandoval 2014]. In this context, we would like to explore the role of some of the different features of the AR system:

  • Embodiment and Physical Semiotics – The physical actions performed to interact with the AR system as well as the physical representation of the mathematical concepts at hand.
  • Interactive Exploration – The digital interactivity provided by AR to support one’s exploration of a mathematical system.
  • Fading and Adaptive Scaffolding – The digital adaptivity of AR scaffolding as well as its fading by digital design or physical constraints.
  • Multi-modal Representation – The access to a multi-modal representation of a same mathematical situation.
  • Collaboration and Social Interaction – The collaboration with the teacher or other students as well as the awareness of the situation of the surrounding students.

From this list, we will identify the salient mechanisms involved in the learning outcomes of the activity, and focus only on these ones, due to time constraints. We will design studies to get a better understanding of their effect. Such studies will involve modifying the AR features relevant to said mechanism, and measure the effect on our different metrics.

3.3 Understand the Context most Suitable for Augmented Reality Learning

As an AR Learning activity is physically, temporally, and socially embedded, different contexts might produce different outcomes. Regarding this aspect, we will focus on the following questions: is an AR Learning activity self sufficient, or more beneficial when used as a preparation for future learning [Schwartz 2004], or as an elaboration of past learning? What is the effect of resources enrichment and resources impoverishment on the outcome of an AR Learning activity?

To answer these questions, we are interested in different conditions:

  • I – Teacher Instruction: the teacher gives a classic lecture on the topic at hand
  • AI – AR Teacher Instruction: the teacher uses the AR system to support the instruction
  • E – Students Exploration: students explore a Linear Algebra question with pen and paper
  • AE – AR Students Exploration: students explore a Linear Algebra question with our AR system

With these notations, AR as preparation for future learning can be described as AE-I or AE-AI, and AR as elaboration of past learning can be described as I-AE or AI-AE (Figure 7 – Left). The first question can be answered by comparing the effect of these different conditions.

Regarding the second question, we are interested in the effect of introducing AR in an activity, compared to removing it during the activity. In one case, the students start exploring with pen and paper only, and are then given the AR application (E-AE), and in another case, they start with the AR right away but finish their exploration without it (AE-E)  (Figure 7 – Right).

Figure 7: (Left) AR activity as Preparation for Future Learning vs AR activity as Elaboration of Past Learning. (Right) Enrichment of AR Resources vs Impoverishment of AR Resources


[Azuma 1997] R. T. Azuma. A survey of augmented reality. Presence: Teleoperators & Virtual Environments, 6

(4):355–385, 1997.

[Billinghurst 2002] M. Billinghurst and H. Kato. Collaborative augmented reality. Communications of the ACM, 45 (7):64–70, 2002.

[Billinghurst 2002] M. Billinghurst, H. Kato, K. Kiyokawa, D. Belcher, and I. Poupyrev. Experiments with face-to-face collaborative ar interfaces. Virtual Reality, 6(3):107–121, 2002.

[Bradbury 1951] R. Bradbury. The Veldt. 1951.

[Brooke 1996] J. Brooke. SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194), 4-7, 1996.

[Bujak 2013] K. R. Bujak, I. Radu, R. Catrambone, B. Macintyre, R. Zheng, and G. Golubski. A psychological perspective on augmented reality in the mathematics classroom. Computers & Education, 68: 536–544, 2013.

[Chaiklin 2003] S. Chaiklin. The zone of proximal development in Vygotsky’s analysis of learning and instruction.Vygotsky’s educational theory in cultural context, 1:39–64, 2003.

[Council 1989] N. R. Council et al. Everybody counts: A report to the nation on the future of mathematics education. National Academies Press, 1989.

[Cuendet 2013] S. Cuendet, Q. Bonnard, S. Do-Lenh, and P. Dillenbourg. Designing augmented reality for the classroom. Computers & Education, 68:557–569, 2013.

[Dick 1968] P. K. Dick. Do Androids Dream of Electric Sheep? 1968.

[Hamilton 2014] K. E. Hamilton and J. Olenewa. Augmented reality in education. Proc. SXSW Interactive, 2011. M. Kapur. Productive failure in learning math. Cognitive Science, 38(5):1008–1022, 2014.

[Hart 1988] S. G. Hart & L.E. Staveland. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology (Vol. 52, pp. 139-183). North-Holland, 1988.

[Ibáñez 2018] M. B. Ibáñez, and C. Delgado-Kloos. Augmented reality for STEM learning: A systematic review. Computers & Education, 123, 109-123, 2018

[Kerawalla 2006] L. Kerawalla, R. Luckin, S. Seljeflot, and A. Woolard. “making it real”: exploring the potential of augmented reality for teaching primary school science. Virtual reality, 10(3-4):163–174, 2006.

[Klopfer 2004] E. Klopfer and S. Yoon. Developing games and simulations for today and tomorrow’s tech savvy youth. TechTrends, 49(3):33–41, 2004.

[Koutromanos 2015] G. Koutromanos, A. Sofos, & L .Avraamidou. The use of augmented reality games in education: a review of the literature. Educational Media International, 52(4), 253-271, 2015.

[Milgram 1995] P. Milgram, H. Takemura, A. Utsumi, and F. Kishino. Augmented reality: A class of displays on the reality-virtuality continuum. In Telemanipulator and telepresence technologies, volume 2351, pages 282–293. International Society for Optics and Photonics, 1995.

[Papert 1980] S. Papert. Mindstorms: Children, computers, and powerful ideas. Basic Books, Inc., 1980. 6

[Pólya 1954] G. Pólya. Mathematics and plausible reasoning: Induction and analogy in mathematics, volume 1. Princeton University Press, 1954.

[Sandoval 2014] W. Sandoval. Conjecture mapping: An approach to systematic educational design research. Journal of the learning sciences, 23(1), 18-36, 2014.

[Schoenfeld 1992] A. H. Schoenfeld. Learning to think mathematically: Problem solving, metacognition, and sense making in mathematics. Handbook of research on mathematics teaching and learning, 334370, 1992.

[Shelton 2002] B. E. Shelton and N. R. Hedley. Using augmented reality for teaching earth-sun relationships to undergraduate geography students. In The First IEEE International Workshop Augmented Reality Toolkit,, pages 8–pp. IEEE, 2002.

[Shelton 2004] B. E. Shelton, N. R. Hedley. Exploring a cognitive basis for learning spatial relationships with augmented reality. Technology, Instruction, Cognition and Learning, 1(4), 323, 2004.

[Schwartz 2004] D. L. Schwartz, T. Martin. Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184, 2004.

[Tang 2003] A. Tang, C. Owen, F. Biocca, and W. Mou. Comparative effectiveness of augmented reality in object assembly. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’03). ACM, New York, NY, USA, 73-80. DOI=http://dx.doi.org/10.1145/642611.642626, 2003

[Thompson 2013] P. W. Thompson. In the absence of meaning…. In Vital directions for mathematics education research (pp. 57-93). Springer, New York, NY, 2013.

[Trninic 2012] D. Trninic and D. Abrahamson. “Embodied Artifacts and Conceptual Performances” 1 (n.d.): 8.

[Tversky 2001] B. Tversky. Spatial schemas in depictions. In Spatial schemas and abstract thought, pages 79–111, 2001.

[Van Krevelen 2007] R. Van Krevelen. Augmented reality: Technologies, applications, and limitations. 04 2007. doi: 10.13140/RG.2.1.1874.7929.

[Wood 1976] D. Wood, J. S. Bruner, and G. Ross. The role of tutoring in problem solving. Journal of child psychology and psychiatry, 17(2):89–100, 1976.

[Yuen 2011] S. C.-Y. Yuen, G. Yaoyuneyong, and E. Johnson. Augmented reality: An overview and five directions for AR in education. Journal of Educational Technology Development and Exchange (JETDE), 4(1):11, 2011.

Julia Chatain

Bob Sumner

Manu Kapur