Cold Start Problem: An Experimental Study of Knowledge Tracing Models with New Students

KnowledgeTracing (KT) involves predicting students’ knowledge states based on their interactions with Intelligent Tutoring Systems (ITS). A key challenge is the cold start problem, accurately predicting knowledge for new students with minimal interaction data. Unlike prior work, which typically trains KT models on initial interactions of all students and tests on their subsequent interactions, our approach trains models solely using historical data from past students, evaluating their performance exclusively on entirely new students. We investigate cold start effects across three KT models: Deep Knowledge Tracing (DKT), Dynamic Key-Value Memory Networks (DKVMN), and Self-Attentive Knowledge Tracing (SAKT), using ASSISTments 2009, 2015, and 2017 datasets. Results indicate all models initially struggle under cold start conditions but progressively improve with more interactions; SAKT shows higher initial accuracy yet still faces limitations. These findings highlight the need for KT models that effectively generalize to new learners, emphasizing the importance of developing models robust in few-shot and zero-shot learning scenarios

💡 Research Summary

The paper tackles the cold‑start problem in Knowledge Tracing (KT), which arises when an Intelligent Tutoring System (ITS) must predict a brand‑new learner’s knowledge state with little or no prior interaction data. While most prior work evaluates KT models by training on each student’s initial responses and testing on their later responses, this study adopts a more realistic scenario: models are trained exclusively on historical data from past students and are evaluated solely on completely unseen learners.

Three widely used deep‑learning KT models are examined: Deep Knowledge Tracing (DKT), Dynamic Key‑Value Memory Networks (DKVMN), and Self‑Attentive Knowledge Tracing (SAKT). The authors use three ASSISTments datasets (2009, 2015, 2017). For each dataset they randomly select five disjoint groups of ten students, remove all of these students’ interactions from the training pool, and use them as test sets. This strict segregation guarantees that the models have never observed any of the test students during training, thereby mimicking a real‑world deployment where a newly enrolled student arrives with no historical record.

Experimental design varies the number of observed interactions per test student—from 3 up to 20 questions (30 for the 2017 dataset). Accuracy is measured after each incremental interaction, producing learning curves that reveal how quickly each model adapts to a new learner.

Results show a consistent pattern across datasets. DKT, which relies on an LSTM‑based recurrent architecture, starts with low accuracy (≈0.45) but improves steadily as more questions are observed, reaching ≈0.75 by the 20th interaction. DKVMN, which stores concept‑level keys and student mastery values in an external memory, begins with higher accuracy (≈0.55–0.60) and rapidly climbs above 0.75 after roughly ten interactions. Its memory‑centric design appears to extract useful concept‑level signals even from very sparse data. SAKT, built on self‑attention mechanisms inspired by Transformers, exhibits the strongest early performance (≈0.60–0.65 after only three to five questions) but its gains plateau after about fifteen interactions, and in some cases a mild performance dip is observed. This suggests that attention is effective at capturing immediate patterns but does not automatically guarantee sustained long‑term adaptation.

The discussion interprets these findings in light of the three research questions. All models struggle initially, confirming that limited prior data hampers early prediction (RQ1). DKVMN adapts fastest with minimal data, while DKT shows the most robust long‑term improvement (RQ2). Although SAKT’s attention mechanism yields better early accuracy, it does not fully solve the cold‑start issue, as evidenced by its later plateau (RQ3).

The authors conclude that effective cold‑start KT requires a blend of rapid initial adaptation and sustained learning capacity. They propose future work on hybrid architectures that combine the memory strengths of DKVMN with the pattern‑recognition abilities of attention, as well as exploring meta‑learning, transfer learning, and federated learning to enable few‑shot or zero‑shot personalization. Incorporating auxiliary student metadata (e.g., prior knowledge assessments, learning styles) is also suggested to further mitigate cold‑start challenges.

Overall, the paper provides a valuable benchmark for evaluating KT models under realistic cold‑start conditions and highlights the need for next‑generation models that can generalize to entirely new learners with limited interaction data.