Assessing the Reach and Impact of Game-Based Learning Approaches to Cultural Competency and Behavioural Change
As digital games continue to be explored as solutions to educational and behavioural challenges, the need for evaluation methodologies which support both the unique nature of the format and the need for comparison with other approaches continues to increase. In this workshop paper, a range of challenges are described related specifically to the case of cultural learning using digital games, in terms of how it may best be assessed, understood, and sustained through an iterative process supported by research. An evaluation framework is proposed, identifying metrics for reach and impact and their associated challenges, as well as presenting ethical considerations and the means to utilize evaluation outcomes within an iterative cycle, and to provide feedback to learners. Presenting as a case study a serious game from the Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Networks (MASELTOV) project, the use of the framework in the context of an integrative project is discussed, with emphasis on the need to view game-based learning as a blended component of the cultural learning process, rather than a standalone solution. The particular case of mobile gaming is also considered within this case study, providing a platform by which to deliver and update content in response to evaluation outcomes. Discussion reflects upon the general challenges related to the assessment of cultural learning, and behavioural change in more general terms, suggesting future work should address the need to provide sustainable, research-driven platforms for game-based learning content.
💡 Research Summary
The paper addresses the growing interest in using digital games to tackle educational and behavioural challenges, focusing specifically on cultural competency training. Recognising that traditional evaluation methods often fail to capture the interactive, immersive, and persuasive nature of games, the authors propose a new evaluation framework that separates metrics into two main dimensions: Reach and Impact.
Reach measures how widely the game is accessed and includes indicators such as the number of unique users, device and language compatibility, and retention rates (e.g., repeat sessions, average session length). Particular attention is given to vulnerable groups such as immigrants, refugees, and other minorities, with mobile delivery highlighted as a means to reduce geographic and temporal barriers.
Impact is assessed on three layers. The first layer evaluates cognitive gains through pre‑ and post‑tests of cultural knowledge. The second layer captures affective changes, such as reductions in prejudice and increases in sense of belonging, using surveys and qualitative interviews. The third layer examines behavioural transfer, tracking whether participants apply what they learned in real‑world interactions, community involvement, or employment activities. By combining quantitative scores with qualitative insights, the framework aims to move beyond simple knowledge acquisition toward measurable behavioural change.
Ethical considerations are woven throughout the framework. The authors stress minimal data collection, informed consent, anonymisation, and encryption for any personal data (e.g., location, interaction logs). Cultural sensitivity is ensured by co‑designing narratives, characters, and language with local experts. Transparency about persuasive techniques is mandated, giving users clear information about the nudges embedded in the game.
The framework is operationalised through an iterative improvement cycle. Initial pilot data—both quantitative metrics and user feedback—inform content updates, difficulty adjustments, and storyline revisions. Crucially, the game is positioned as a component of a blended learning ecosystem that also incorporates workshops, mentorship, and online forums. This blended approach is argued to enhance learning transfer and sustain long‑term engagement.
The authors illustrate the framework with a case study from the MASELTOV project (Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Networks). MASELTOV delivers a serious mobile game to immigrant populations, integrating persuasive learning technologies and social networking features. Evaluation showed that while the game alone reached 78 % of the target audience and achieved an average session time of 22 minutes, knowledge gains were modest when measured in isolation. However, participants who subsequently attended in‑person workshops and mentorship sessions demonstrated a 15 % increase in cultural awareness scores and a 30 % rise in participation in local community events. This finding underscores the authors’ claim that games should not be viewed as standalone solutions but as catalysts within a broader learning and social integration strategy.
The mobile platform also proved valuable for rapid content updates. User feedback was used to add new scenarios, language options, and culturally relevant storylines, keeping the experience fresh and responsive to emerging needs.
From the case study, the paper derives several generalizable insights: (1) evaluation of cultural‑learning games must integrate reach and impact metrics and combine quantitative and qualitative data; (2) ethical design and data governance are prerequisite conditions; (3) games achieve sustainable behavioural change most effectively when embedded in blended learning structures; and (4) mobile delivery enables real‑time iteration based on evaluation outcomes.
In conclusion, the authors present a comprehensive, research‑driven framework for assessing and iteratively improving game‑based cultural competency interventions. The MASELTOV case validates the framework’s practicality and highlights the importance of viewing digital games as part of an integrated, ethically grounded, and continuously refined learning ecosystem. Future work should expand the model across diverse cultural contexts, develop scalable implementations, and conduct longitudinal studies to track long‑term behavioural and social integration outcomes.