Grounding Abstract Mathematics through Embodied Interaction in Virtual Reality

Keywords

Embodied interaction, embodied cognition, abstract mathematics, virtual reality, multiple representations, concreteness fading, higher education

Introduction

Mathematics is a useful skill to learn, even for students who do not wish to become mathematicians. One of the goals of mathematics is to gain understanding of the patterns of the world, in order to model and predict future outcomes. Recent years shed light on the importance of mathematics, for example to understand and take action during the COVID-19 pandemic or the general climate crisis. However, most students find mathematics useless and disconnected from the real world, and several countries are witnessing a worrisome decline in mathematics ability. This means, we argue, that we should challenge our assumptions about how to teach mathematics, specifically as the way we teach mathematics currently does not enable students to understand it well enough to transfer their skills to other classes, nor outside of the classroom setting.

Researchers and educators tackling this issue have argued that, although mathematics is often considered as a Platonic ideal that cannot directly be sensed or manipulated, mathematics rather is a social and malleable process that arises from our sensorimotor experiences of the world. For example, arithmetic can be seen as manipulation of object collections. Similarly, the concept of infinity can be related to our way of expressing something too large for our senses to perceive, while continuity may arise from sequences of elements too small for us to sense. Simply put, mathematics is embodied, and, although not all experts agree on the implications of this theory, there is undeniable evidence that our bodies play an important role in learning mathematics. As a consequence, when designing learning activities, we need to facilitate gesture production, and support sense-making of bodily actions.

Meanwhile, Virtual Reality (VR), a technology heavily focused on bodily movement and manipulations, became more affordable and widespread. Using a wireless Head-Mounted Displays, VR can immerse learners’ sensory channels into another world, digitally manufactured. Using hand-tracking technologies, VR seems particularly suitable to support embodied learning activities. With VR, we could create the MathLand imagined by Papert, a world of mathematics where learners can explore and manipulate mathematical objects. But is VR truly the solution to support the transition towards embodied learning of mathematics?

a person is manipulating a function's graph in VR. In the background is a mountain and a water pipe system.

Research questions

Although the idea of using VR to implement embodied learning activities seems promising, VR interaction research does not leverage actual theories of embodied learning. Specifically, we identified three main challenges in this field. First, there is a lack of empirical studies evaluating the effect of VR embodied learning activities. Second, only a handful of studies address higher education specifically. Third, there are no design guidelines for embodied learning activities in VR focusing specifically on interaction. Considering these challenges, our work addresses the following research question:

How to design embodied interaction to support embodied sense-making of mathematics?

We ground our work in three meanings of embodiment: Embodied cognition relates to the role of learners’ bodies in cognition, embodied interaction relates to the role of users’ bodies in interaction, and avatar embodiment relates to the perception of a digital body as one’s own. Within this framework, we address these challenges through six contributions, summarized below.

State of the project

Designing the digital avatar

Looking at embodied interaction at the level of the avatar, we propose “Digital Gloves”, a novel input mechanism that supports embodied interaction and reduces split-attention effect by co-locating input and display on the users’ hands. Through two user studies, we demonstrate the potential of our mechanism for more intuitive, enjoyable, and effective gaming and learning experiences. We offer recommendations to best design activities using our mechanism as well as suggestions for future applications.

The images is split into 4 vignettes. The first one shows a person with a VR headset, looking at her hand. The second one shows a pinball game displayed on one hand. The third one shows a maze game on one hand. And the fourth one shows a two hands game where each hand contains an item and the hands are positioned to mimic a twin-pan balance.

Digital Gloves [Chatain 2020]

Designing the interaction

Focusing on the interaction level, we offer an empirical evaluation of the impact of the degree and type of embodiment on usability and learning outcomes. To do so, we designed an embodied activity to learn about derivatives. Our results reveal that although the degree of embodiment only impacts the duration of the activity, the type of embodiment impacts manipulations and learning outcomes. We offer an explanation of these results in terms of mathematical meanings highlighted by different types of embodiment, and conclude with design recommendations for VR embodied learning activities.

Three different ways of learning derivatives using embodied interaction: direct-embodied interaction on tablet, direct-embodied interaction in Virtual Reality, and enacted interaction in Virtual Reality [Chatain 2022]

Designing the context of the interaction

Looking at the context of the interaction, we then conceptualize embodiment as a form of concreteness, and demonstrate the grounding affordances of embodied concreteness. To do so, we designed an embodied activity in VR to teach graph theory to bachelor students. Our activity builds on embodied metaphors by representing graphs as water flow systems. Our results show that students using our activities feel that the activity highlights the relevance of the topic best, and also feel better prepared for the subsequent lecture. Moreover, unlike an activity focusing on manipulation only, our activity does not impair transfer abilities.

The figures contains three pictures, in doodle style. On the first figure, a character solves a graph theory problem on paper with an abstract representation. On the second figure, on tablet with an abstract representation. On the third figure, in Virtual Reality with a relatable and embodied representation. The graph is always the same and the character always interacts with the same edge.

Three approaches to graph theory exercises: abstraction on paper, manipulated concreteness on tablet, and embodied concreteness in Virtual Reality [Chatain 2023]

Designing for individual differences in learners

As learners are at the core of the learning experience, we also explore the design space of embodied interaction for sense-making focusing on the individual differences among learners. Specifically, we explore two contexts. First, we look into an intuition-building activity where learners are directed towards specific bodily actions. Second, we look into an intuition-probing activity where learners spontaneously perform bodily actions. Specifically, we look at the role of individual differences and we aggregate the results from both studies to offer general design recommendations as well as directions for future research on sense-making embodied interaction in VR. (Work to be published)

The left picture shows an anonymized participant representing a slope in the air using both pointing fingers. On their side is a picture of a function's graph that they are describing. The right picture shows a woman manipulating a function's graph on a tablet using both pointing fingers on a specific point of the function to adjust its slope. In the middle, there is a system representation with several elements. "Learner" in a circle with bidirectional arrows towards a "bodily actions" box. This box has bidirectional arrows towards a "Concept" circle. The box also has a double arrow towards a "learning" box. The arrow from the learner to the bodily action contains the label "Design for spontaneous bodily actions", and the arrow on the other direction contains the label "Design for directed bodily actions".

Spontaneous bodily actions are performed spontaneously while solving a task (right). Directed bodily actions are performed as explicitly asked by a task (left).

Supporting transition to formalisms

As our activities so far focus on intuition building, we then explore how to reconnect the embodied activities with formal representations of mathematical objects. In this project, we designed and evaluated two prototypes for input of mathematical expressions in Virtual Reality.

The image has three vignettes. First, the VR environment. A room with windows, green walls, wooden floor, and a plant in the corner. The second vignette shows a stick-figure user pressing buttons on a virtual keyboard to add mathematical symbols to the expression in front of them. The last vignette shows the user dragging and dropping the mathematical symbols directly onto the expression.

Two input techniques for mathematics in Virtual Reality [Sansonetti 2021]

Defining the embodiment landscape

Finally, based on this experience, we offer an interdisciplinary framework for embodied learning and describe its relevance through several use cases. From this we draw several design considerations for VR embodied learning interactions: (1) consider the meaning highlighted by the interaction and its context and how it relates to the meaning being learned, (2) consider desirable difficulties in interaction to support deeper sense-making of bodily actions, and (3) consider breaking avatar-body morphological similarity to design semantic avatars, that is, avatars designed to apprehend a specific concept and highlight a specific meaning.

Three pictures. 1. A person has an arm folded and an arm extended. These two arms represent the x and y axis in a horizontal plane. The plane is distorted, as one arm is longer than the other. On the plane are 3D houses and trees to show the distortion. 2. A woman interacts with a function curve in VR. The display also shows the derivative, which the woman manipulates through a digital rotating angle. 3. A man interacts with a pipe system representation of a graph in VR. He presses a button on a pipe with one hand, and adjusts the level of the water in the pipe with the other hand.

Three VR embodied learning activities: (a) the digital avatar is used to embody mathematical concepts, (b) the learner interacts with mathematical objects [Chatain 2022], (c) the virtual world grounds abstract mathematics in embodied concreteness, that is a highly embodied, situated and relatable context [Chatain 2023].

Publications

Doctoral dissertation

🏅 Chatain, J. (2023, March) Embodied Interaction in Virtual Reality for Grounding Mathematics. https://doi.org/10.3929/ethz-b-000613388 or (pdf) or (video) or (thread). Recipient of the ETH medal for outstanding dissertation.

Papers

Chatain, J., Kapur, M., Sumner, R. W. (2023, April) Three Perspectives on Embodied Learning in Virtual Reality: Opportunities for Interaction Design. In CHI EA ’23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems.
https://doi.org/10.1145/3544549.3585805 or (pdf) or (video)

Chatain, J., Sumner, R. W., Kapur, M. (2023, August) Embodied interaction in virtual reality for learning mathematics. In the 20th Biennial EARLI Conference.

Chatain, J., Varga, R., Fayolle, V., Kapur, M., Sumner R. W. (2023, February) Grounding Graph Theory in Embodied Concreteness with Virtual Reality. In TEI ’23: Proceedings of the Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction.
https://doi.org/10.1145/3569009.3572733 or (pdf) or (video)

Chatain, J., Ramp, V., Gashaj, V., Fayolle, V., Kapur, M., Sumner, R. W., Magnenat, S. (2022, June). Grasping Derivatives: Teaching Mathematics through Embodied Interactions using Tablets and Virtual Reality. In Interaction Design and Children (IDC’22).
https://doi.org/10.1145/3501712.3529748 or (pdf) or (video)

Sansonetti, L., Chatain, J., Caldeira, P., Fayolle, V., Kapur, M., Sumner, R. W. (2021, September). Mathematics Input for Educational Applications in Virtual Reality. In International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments (ICAT-EGVE). https://doi.org/10.2312/egve.20211324  or (pdf)

Chatain, J., Sisserman, D. M., Reichardt, L., Fayolle, V., Kapur, M., Sumner, R. W., Zünd, F., Bermano, A. H. (2020, November). DigiGlo: Exploring the Palm as an Input and Display Mechanism through Digital Gloves. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (CHI Play). https://dl.acm.org/doi/abs/10.1145/3410404.3414260 or (pdf) or (video)

Poster

Chatain, J., Ramp, V., Gashaj, V., Fayolle, V., Kapur, M., Sumner, R. W., Magnenat, S. (2022, May). ETH-EPFL Grasping Derivatives with Embodied Interactions. Joint Doctoral Program for Learning Sciences Kick-off Event (JDPLS). (pdf)

Dr. Julia Chatain

Prof. Dr. Manu Kapur

Prof. Dr. Robert W. Sumner