As the online learning landscape evolves, the need for personalization is increasingly evident. Although educational resources are burgeoning, educators face challenges selecting materials that both align with intended learning outcomes and address diverse learner needs. Large Language Models (LLMs) are attracting growing interest for their potential to create learning resources that better support personalization, but verifying coverage of intended outcomes still requires human alignment review, which is costly and limits scalability. We propose a framework that supports the cost-effective automation of evaluating alignment between educational resources and intended learning outcomes. Using human-generated materials, we benchmarked LLM-based text-embedding models and found that the most accurate model (Voyage) achieved 79% accuracy in detecting alignment. We then applied the optimal model to LLM-generated resources and, via expert evaluation, confirmed that it reliably assessed correspondence to intended outcomes (83% accuracy). Finally, in a three-group experiment with 360 learners, higher alignment scores were positively related to greater learning performance, chi-squared(2, N = 360) = 15.39, p < 0.001. These findings show that embedding-based alignment scores can facilitate scalable personalization by confirming alignment with learning outcomes, which allows teachers to focus on tailoring content to diverse learner needs.
Online education has expanded markedly in recent years, driven by learners' routine use of online platforms, the COVID-19 pandemic, and a renewed emphasis on lifelong learning, positioning digital tools as critical for equity and inclusiveness [1][2][3]. In this online landscape, the demand for personalization and inclusiveness underscores the challenge of curating content that aligns with both target learning outcomes and diverse learner needs [3,4]. Although many resources are available, teachers find the process of identifying suitable materials time-consuming and inefficient [5,6]. Moreover, prioritizing personalization can inadvertently weaken alignment, necessitating deliberate safeguards [7]. Prior work has sought technological remedies: early systems aggregated open educational resource repositories [8,9] but did not aid selection because they lacked ranking sensitive to pedagogical context; subsequent approaches used semantic technologies and knowledge graphs to encode teaching context [10,11], yet they face scalability limits due to high development and maintenance costs [12]. More scalable machine-learning techniques-such as Learning to Rank (LTR) and topic modeling-have also been explored [13,14], but accuracy remains subpar.
Teachers typically engage in three key tasks: aligning instructional content with intended learning outcomes, delivering instruction, and personalizing learning experiences [15]. Of these, alignment is particularly amenable to automation, whereas effective teaching and personalization depend on students’ contexts and therefore require teachers’ nuanced judgment and interaction [15]. Large Language Models (LLMs), with their advanced natural language processing and reasoning capabilities [16], offer a promising new direction. They can potentially support teachers by efficiently identifying those resources that are constructively aligned [17] with learning outcomes. Furthermore, LLMs can generate new educational materials, expanding the range of resources available to educators [4] and effectively reducing their workload.
Despite the promise of LLMs, incorporating LLMs in education still presents significant challenges. To date, generated educational content requires careful verification to ensure alignment with learning outcomes [17,18], and the high computational costs of these models limit their accessibility and scalability [19]. While recent research has explored using LLMs to create content in narrowly defined domains such as programming [4,18,20], these efforts highlight the persistent risks of hallucination and the high costs associated with quality control [21,22].
In this paper, we explore cost-effective and scalable LLM-based techniques to support teachers by providing resource rankings. These rankings, that are based on the alignment of a resource with intended learning outcomes, can be applied to evaluate either existing or generated educational content. By relying on these rankings, teachers can ensure their pedagogical goals are met, while simultaneously allowing them to tailor resources to diverse learner needs, such as accessibility requirements or varying levels of prior knowledge. To achieve this, we set out to answer the following research questions:
(1) Can text embeddings effectively expose the alignment between a candidate educational resource and teachers’ intended learning outcomes?
(2) If so, can embedding-based alignment rankings of LLM-generated resources be validated through expert evaluation and, subsequently, shown to predict improved learning performance?
To answer these questions, we conducted two studies. The first study evaluated how effectively various text embedding models-from prominent models such as Google Gemini and OpenAI ChatGPT to open-source alternatives [23]-ranked existing resources from YouTube against intended learning outcomes. We developed a scoring metric inspired by Kendall’s tau [24] to assess ranking quality, prioritizing models that ranked resources aligned with intended learning outcomes above those not aligned. The second study exploited the best-performing model from the first study to rank LLM-generated resources in an effort to examine whether it would also be deemed superior by experts and yield enhanced learning performance. Here, six different LLMs were prompted to generate educational content tailored to specific personalization and inclusiveness use cases. These generated resources were again labeled by experts, and we used our ranking score to assess the ability of the optimal model to evaluate them. As an extension, we further tested whether these rankings are associated with superior learning performance. Using an experimental design with 360 participants, we found that higher-ranked resources consistently resulted in better performance outcomes, demonstrating that our ranking approach not only aligns with expert judgment but also predicts learning performance.
The findings from this three-level evalua
This content is AI-processed based on open access ArXiv data.