Visualization Systems for Student Reflection in Serious Games

The Challenge: Bridging Gameplay and Learning

Educational games, like 'Parallel' which teaches parallel programming concepts, are powerful learning tools generating vast amounts of player data. However, a significant gap exists: how can this data be effectively visualized to help students reflect on their own learning process and understand the strategies used by their peers? Existing tools often lack features specifically designed to foster this kind of reflective learning.

Our core challenge was to design a visualization system that moves beyond simple data presentation to actively guide students in analyzing gameplay, comparing strategies, and ultimately deepening their understanding of complex concepts taught through the game.

Understanding the Users: Instructors & Students

To ensure our solution was grounded in real needs, we employed a Design-Based Research (DBR) approach, incorporating UXR methods to understand both instructor goals and student expectations.

Instructor Focus Groups: How Experts Facilitate Reflection

We conducted workshops with two parallel programming instructors. They analyzed anonymized student gameplay recordings from 'Parallel', identifying different strategies, ranking performance, and pinpointing moments ripe for student reflection. They also provided the type of feedback they would give.

Key Insights from Instructors:

• Chunking Gameplay: Instructors analyzed playtraces in segments to understand student actions (e.g., "...Steps 2 and Step 3, is where the student is trying to figure out the Switch...").
• Spatial Awareness: They connected gameplay actions in specific game areas to programming concepts (e.g., "They seem to be placing the semaphore in this area...").
• Comparative Analysis: Instructors frequently compared different students' strategies to highlight efficiency or alternative approaches.
• Performance Metrics: Ranking was based on core parallel programming concepts like identifying critical sections, minimizing their size, and recognizing optimization opportunities.

Student Playtesting & Personas: What Learners Need

We observed 10 students playing 'Parallel' and followed up with semi-structured interviews to understand their struggles, successes, and desires for a support system. Thematic analysis led to two key personas representing student needs.

Key Insights from Student Personas:

Designing the Visualization System

Guided by the insights from instructors and students, and utilizing Shneiderman's Visual Information-Seeking Mantra ("Overview first, zoom and filter, then details-on-demand") as a framework, my team and I designed and developed the OPM Viz system. While detailed development specifics are omitted here, the core design principles aimed to directly address the identified user needs.

Key design goals included:

• Providing an **overview** comparing a student's performance against the community (addressing Persona Insight 2).
• Allowing users to **filter** and select specific peer playtraces for deeper comparison (Insight 1 & 2).
• Offering a synchronized, **detailed side-by-side view** of playtraces, highlighting key actions and game states (Instructor Insight 1 & 3, Persona Insight 2).
• Incorporating metrics relevant to parallel programming efficiency (Instructor Insight 4, Persona Insight 1).
• Suggesting moments for reflection based on common patterns or deviations (Persona Insight 3).

The Solution: OPM Viz

OPM Viz is an interactive visualization system designed to integrate seamlessly with the 'Parallel' serious game. It empowers students to reflect on their gameplay, compare strategies with peers, and gain deeper insights into parallel programming concepts.

The system allows students to:

• View their own performance metrics.
• Compare their solution path against anonymized peers.
• Filter peers based on performance or specific strategies.
• Replay their own and selected peers' games side-by-side.
• Identify key differences and potential areas for improvement.

Evaluation: Does it Foster Reflection?

To assess the effectiveness of OPM Viz, we conducted an evaluative study with 8 student participants. They used the system after playing 'Parallel' while thinking aloud, followed by semi-structured interviews.

Our analysis focused on *how* and *why* students used the system and whether it prompted the desired reflection and learning.

Key Findings from Evaluation:

• Facilitated Comparison: Participants actively used the system to compare their strategies with peers, often expressing surprise or gaining new ideas (e.g., "Oh, I didn't realize you could solve it that way!").
• Identified Inefficiencies: The visualizations helped students pinpoint specific parts of their gameplay that were less efficient than others'.
• Sparked Curiosity: Exploring different solutions through the system encouraged students to think more deeply about the underlying programming concepts.
• Supported Self-Assessment: Participants used the comparative data to gauge their own understanding and performance relative to others.
• Highlighted Trade-offs: Observing different strategies allowed students to understand the trade-offs between various parallel programming approaches within the game context.

Overall, the evaluation indicated that OPM Viz successfully prompted reflection and supported learning by making gameplay data accessible, comparable, and insightful.

Impact and Contributions

This project demonstrates the potential of integrating player-facing visualization systems, informed by Open Learner Model (OLM) principles, into serious games to enhance learning.

Key Contributions:

• Bridging OLMs and Games: Introduced an Open Player Model (OPM) approach, adapting OLM principles for the dynamic context of serious games.
• Learning-Focused Visualization: Designed and validated a player-facing visualization system prioritizing reflection and learning over purely aesthetic or performance-tracking goals.
• Enhancing 'Parallel': Provided a novel tool for the established 'Parallel' research platform, enabling new avenues for studying game-based learning and reflection.
• Published Research: Shared findings and the system design with the academic community via publication at ACM CHI 2024, contributing to HCI and Learning Sciences fields.

The positive evaluation results and academic recognition underscore the value of this user-centered approach to designing learning support tools within educational games.

Reflections and Future Directions

Key Learnings:

• The Power of Comparison: Directly comparing solutions was a highly motivating factor for student engagement and reflection.
• Instructor Insights are Crucial: Understanding how experts analyze and provide feedback was invaluable in designing meaningful visualizations.
• Balancing Guidance and Discovery: Finding the right level of information presentation—enough to guide reflection without overwhelming or giving away answers—is a key design challenge.

Future Directions:

• Exploring adaptive feedback based on student interaction patterns within the visualization.
• Integrating the visualization system more tightly within the game environment itself.
• Testing the system in authentic classroom settings over longer periods.