Visualization Systems for Student Reflection in Serious Games

Figure 1: UXR pipeline adopted to understand student expectations, build a visualization system and evaluate the visualization system.

Generative Steps

In the generative step, we aim to understand how instructors prompt reflection and learning in a serious game called 'Parallel'. Through focus groups, we identify how instructors prompt reflection, discussion and learning in class. Then through playtesting and semi-structured interviews, we ascertain what students expect from a visualization system that supports learning and peer reflection.

Instructor Focus Group

To understand how human instructors facilitate learning in these topics, we conducted workshops with two parallel programming instructors. Instructors were presented them with a recording that showcased four students attempting to solve a level in Parallel. Instructors were then prompted with a series of tasks: Firstly, they were asked to discern the variations observed in the playtraces and the students' respective approaches to the parallel programming challenges. Secondly, they were tasked with ranking these playtraces based on performance, substantiating their rankings with explanations. Lastly, the instructors were asked to pinpoint specific segments in the same playtraces that the student should reflect upon For these highlighted segments, instructors also provided textual feedback, suggesting ways the student could ameliorate their gameplay in that particular segment. In the end, we opened the floor for them to pose any questions and comments.

Insights from Instructor Interviews

• Breaking Down the Playtrace: The instructors regularly spoke about playtraces in chunks to explain or deduce what students were doing - "Yeah, I think Steps 2 and Step 3, is where the student is trying to figure out the Switch and see what happens."
• Identifying Important Spatial Zones: While examining student's playtraces, instructors often referred to how students have used different sections of the spatial map in analogous to parallel programming concepts - "They seem to be placing the semaphore in this area to prevent this thread from reaching before the other thread"
• Comparing Parts of Playtraces: Instructors compared and contrasted strategies used by students to solve parallel programming challenges within the game. They discussed how certain players' strategy was more efficient than others - "If you compare the two players in the beginning, they seem to be tackling different problems. The first student places a semaphore here to work on the direction box, while the other student"
• Ranking based on Parallel Programming Metrics: Instructors, when asked to rank play traces, compared playtraces based on (a) How accurately did the player identify critical sections? (b) Reducing the size of critical sections by placing semaphores as close as possible (c) and Identifying opportunities within the play trace to further optimize their playtrace by exploring alternate solutions.

Student Playtesting and Personas

We invited 10 players to play the serious game 'Parallel'. Post the playtesting, we noted verbal cues, and followed up by a semi-strucutred interview to understand where students struggled, were looking for additional help and areas where the felt they did well. We performed thematic analysis on the responses and identified two personas that captured student expectations from a visualization support system that would help them better understand educational concepts embedded in the serious game.

Insights from Personas

1. Improve Solution Efficiency:
   • Players aim to enhance the efficiency of their gameplay solutions.
   • Example: Jamie wishes to "locate and select more efficient players" to reflect and improve her gameplay.
2. Easy Gameplay Comparison:
   • Players desire an effortless way to compare their gameplay with peers from multiple perspectives.
   • Example 1: Jamie seeks a high-level community view to gauge her solution efficiency and identify better-performing players.
   • Example 2: Alex prefers a detailed perspective to explore players' gameplay actions and "examine other potential moves not encountered in her gameplay."
3. Suggestions for Gameplay Improvement:
   • Players appreciate advice on enhancing their gameplay both during and post-gameplay.
   • Example 1: Alex wishes to view other players' solutions when facing challenges.
   • Example 2: Jamie prefers figuring out solutions independently but is keen on exploring alternative approaches post-gameplay.

Figure 2: The designed and developed Visualization System

Design and Development

Based on the insights generated form instructors and student personas, we used the Shniederman Visualization Design framework to design and build the visualization system. In this case study, I will not share details with regards to the development and design but stick to the UXR methods and process used. The visualization system designed and developed is presented in Figure 2.

Evaluative Research

To understand if our visualization system prompts reflection and learning, we invited 8 participants to use the visualization system, think out aloud and we then followed it up with semi-structured interviews. From these interviews, we identified various themes (shown in Figure 3) which show how and why the participants used the visualization system and how the system helps in learning and reflection.

Figure 3: Various ways students interacted with the visualization system.

Conclusion

Our UX research makes contributions and has implications to various fields. Details as follows:

Implications towards Open Learner Models:

• Open Learner Models (OLMs) are key in representing student learning journeys.
• Modern classrooms are leaning towards game-centric learning.
• There's a need to integrate OLMs within video games.
• Our project is a trailblazer, combining video games and OLMs.
• We introduce an OPM that takes traditional OLM principles and embeds them in a gaming environment.
• Aim: Spur game designers and learning experts to create diverse OPMs for enhanced game-based learning experiences.

Implications towards Player Facing Visualization Systems:

• The growth of video games necessitates player-centric visualization systems promoting reflection and learning.
• Most systems focus on aesthetics, aggregation, accuracy, readability, and interpretability.
• Learning and reflection aspects are often sidelined.
• We're pioneering by designing a visualization system informed by OLMs.
• Our system stands as a testament to merging game design with learning literature principles.
• Goal: Promote discussions about leveraging OLM principles in game visualization systems.

Implications towards Parallel Research Platform:

• The serious game, Parallel, is a popular tool for several visualization and learning analytics studies.
• Parallel is recognized and used for teaching parallel programming in classrooms.
• Our contribution:
  • Introduce a new learner model for classrooms within Parallel.
  • Provide a visualization system that features this learner model.
  • Enhance gameplay reflections and discussions through our visualization system.