Duolingo wants to recommend stories of increasing difficulty as a student learns a language. The app has a library of short reading passages across multiple languages, each with a CEFR level (A1 to C2), and it must serve each learner a story that stretches them just past what they can already read comfortably — challenging enough to push growth but not so hard that the learner gives up. The core challenge is measuring story difficulty and learner ability jointly: a story’s difficulty depends on vocabulary frequency, grammar complexity, sentence length, and concreteness, while a learner’s ability depends on which words and grammar they have mastered so far. Deep knowledge tracing (DKT) can model a learner’s evolving knowledge state from their exercise interactions (correct/wrong, hints used, time spent), and this state can be matched against story difficulty estimates to pick the next story.
How would you design this model? Cover the candidate knowledge tracing architectures (deep knowledge tracing with recurrent networks vs transformer-based knowledge tracing vs item response theory enhanced with neural features), how you estimate story difficulty from text features and historical learner performance, how you combine the learner state with story difficulty to produce a recommendation score, how you handle cold start for new stories and new learners, and how you evaluate recommendation quality beyond simple accuracy.
The Problem: the shelf is sorted by CEFR level, but the learner is not a CEFR level. Which story is one step past what this particular learner can already read, and how would a model know?
Answer
The design is an attention-based knowledge tracer with an IRT-shaped head. A causal transformer reads the learner’s interaction history (which exercise, correct or not, hints used, response time) and emits two outputs: a scalar ability 
on a logit scale and a per-skill mastery vector over the words and grammar points the course teaches. Each story is placed on the same logit scale with a difficulty 
that starts from text features (vocabulary frequency, sentence length, clause depth, concreteness) and is then calibrated on historical learner outcomes. The recommendation score is the predicted comprehension probability 
pulled toward a target band of 0.70 to 0.85, plus a bonus for stories whose skills sit in the learner’s learning zone. The two pivotal decisions are to put ability and difficulty on one shared scale, so a brand-new story can be placed from its text alone, and to window the attention so the per-request cost stays flat as histories grow.
(1) DKT With Recurrent Networks: an LSTM consumes embeddings of (skill, correctness, hint, time) per interaction; its hidden state is the knowledge state and a sigmoid layer reads out per-skill mastery.
(2) Transformer Knowledge Tracing: SAKT, SAINT, or AKT-style causal attention in which the next item queries the interaction history; a distance-aware decay supplies recency and Rasch-style item embeddings supply a difficulty scale.
(3) IRT Enhanced With Neural Features: the Rasch logistic 
stays as the output model, while neural front-ends estimate from history and from story text (Deep-IRT, knowledge tracing machines).
Figure 1: Two estimators on one logit scale: the learner side traces ability and mastery from interactions, the story side places difficulty from text and outcomes, and the match picks the story whose predicted comprehension lands in the target zone.
Clarify Before Designing:
(1) Skill Granularity: is mastery tracked over individual words (thousands per course) or over grammar concepts and word families (hundreds)? This sets the output width and how sparse each skill’s evidence is.
(2) History Length: how many interactions does an active learner accumulate, hundreds or tens of thousands, and must the model attend to all of it or only a recent window?
(3) Inference Budget: is the learner state updated online after every exercise on a CPU budget of a few milliseconds, or recomputed in a nightly batch?
(4) Outcome Signal: do stories carry comprehension checks, or only completion and abandonment, and are hints and response times logged reliably enough to be features?
(5) Per-Language Data: how many learners does the smallest course have, since that number decides whether a transformer is even trainable there?
(6) Difficulty Ground Truth: are the editorial CEFR tags labels the model must respect, or priors it may revise from learner outcomes?
Leave a Reply