DL0202 RL Production Recommendation

The hard part is almost never the learning algorithm. A production recommender is an off-policy problem with a combinatorial action space, a corrupted reward signal, and a closed feedback loop, and each of those violates an assumption that textbook RL takes for granted. You cannot explore freely on paying users, so the only training data is what yesterday’s serving policy logged, which makes importance weighting and its variance the central engineering problem rather than a footnote. The action is not one item but a slate of K items drawn from a catalogue of order , so enumerating a Q value per action is impossible and the policy must be factorized or decomposed. Finally, the reward you actually want, long-term satisfaction, arrives days after the action, and is confounded by position bias, seasonality, and the recommendations themselves.

(1) Off-Policy By Construction: training data comes from a logging policy , not from the policy you are optimizing, so every gradient needs an importance correction whose variance grows with the mismatch.
(2) Combinatorial Slate Actions: a page of items from candidates gives actions, forcing decompositions such as top-K correction or SlateQ‘s per-item value factorization.
(3) Reward Specification And Delay: clicks are immediate and gameable, retention is the real objective and is observed days later, so credit assignment spans sessions rather than steps.
(4) Feedback Loops And Non-Stationarity: the deployed policy shapes tomorrow’s training distribution, so a small early bias compounds instead of averaging out, and the catalogue and user base drift underneath the model.
(5) Offline Evaluation Is Unreliable: off-policy estimators are high variance on large action spaces, so offline value gains routinely fail to reproduce in an A/B test.
(6) Infrastructure Cost: you must log propensities at serving time, keep the randomization budget small, and still return a ranked page inside a tens-of-milliseconds latency budget.

Figure 1: The loop is the problem. Supervised ranking treats the log as a fixed dataset, but here the policy under training generates its own future training distribution, so bias compounds across deployments and the reward attached to each logged action is both delayed and confounded by presentation.

The standard production recipe, popularized by YouTube’s top-K REINFORCE recommender, is to keep policy-gradient learning but reweight each logged trajectory by the ratio between the new policy and the logging policy. Two corrections then appear. The first is weight clipping, which caps the influence of any single log line and trades unbounded variance for a controlled downward bias. The second is the top-K correction factor, which accounts for the fact that the system shows slots rather than sampling one item, and which flattens the gradient on items the policy already places with near certainty. Both are variance-control devices, and understanding why they are needed is the difference between a candidate who has read the paper and one who has shipped the system.

Mathematical Formulation:

Where:

• is the expected discounted return of the target policy over a user trajectory , with discount and per-step reward .
• is the user state (interaction history and context), the item actually shown and logged, and the return following that action.
• is the target policy and the logging (behavior) policy; is their ratio, and it is undefined wherever .
• is the clipping threshold; is the truncated weight, which is biased but bounded.
• is the top-K correction for showing slots instead of one; it tends to for rare items and to as approaches .
• is the effective sample size over logged records, and it is the number that decides whether a 10M-row log is really 10M rows.

Figure 2: Why off-policy learning is a variance problem before it is an optimization problem. As the target policy drifts from the logging policy, the effective sample size of the log collapses exponentially, so a huge log can carry the statistical weight of a small one. Clipping restores usable sample size at the price of a systematic downward bias on exactly the actions the new policy likes most.

This is also why the choice between supervised ranking, a contextual bandit, and full sequential RL is a real design decision rather than a question of ambition. A bandit captures most of the exploration benefit with a single-step correction and no long-horizon credit assignment, and it is what Spotify’s explainable-recommendation work uses for carousel and explanation selection. Full RL earns its cost only when an action genuinely changes the future state, for example when promoting a new creator today changes what the user is willing to watch next month.