ML0076 Reinforcement Learning Loop

Written by

What is reinforcement learning? Describe the agent-environment interaction loop.

Answer

Reinforcement learning is learning what to do from trial and error: an agent acts in an environment, receives a scalar reward, and adjusts its behavior to maximize expected cumulative reward. The loop: at step $t$ the agent observes state $s_t$ , picks an action $a_t$ from its policy $\pi(a \mid s)$ , the environment transitions to a new state and returns reward $r_{t+1}$ , and the agent updates its policy or value estimates from the experience; this repeats until the episode ends. Unlike supervised learning there are no labels: the only supervision is the reward signal, which may arrive many steps late, and the agent’s own actions decide what data it sees next, which is why exploration must be built in.

(1) The MDP Tuple: states, actions, transition probabilities, a reward function, and a discount factor $\gamma$ ; the Markov property says the state summarizes everything relevant about the history.
(2) Return and Value: the agent maximizes the expected discounted return, and value functions score states or state-action pairs so credit for delayed rewards can be assigned to earlier decisions.
(3) The Same Loop Now Trains LLMs: RLHF (InstructGPT) fits a reward model from human preference rankings and optimizes the language model against it with PPO; DeepSeek-R1’s GRPO removes the value critic and baselines each prompt against its own sampled answer group, lifting AIME pass@1 from 15.6% to 71.0% with pure RL.

Agent box and environment box with an action arrow from agent to environment and a state plus reward arrow back, annotated with the policy and the experience tuple

Figure 1: The interaction loop: the agent’s policy turns the observed state into an action, the environment answers with the next state and a scalar reward, and the experience tuple $(s_t, a_t, r_{t+1}, s_{t+1})$ feeds the learning update.

Mathematical Formulation:
$G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k+1}$
$\pi^{*} = \arg\max_{\pi}\ \mathbb{E}_{\pi}\big[G_0\big]$

Where:

$G_t$ is the discounted return from step $t$ ; $r_{t+k+1}$ is the reward received $k$ steps later.
$\gamma \in [0, 1)$ is the discount factor, trading immediate reward against future reward and keeping infinite-horizon returns finite.
$\pi$ is the policy and the expectation runs over trajectories induced by $\pi$ and the environment’s transition dynamics; $\pi^{*}$ is the optimal policy.

Feature	Reinforcement Learning	Supervised Learning
Supervision Signal	Scalar reward, possibly delayed many steps	A label per example
Data	Self-generated, non-stationary	Fixed, assumed i.i.d.
Objective	Maximize expected discounted return	Minimize a per-example loss
Feedback Timing	Credit assignment over time	Immediate and exact
Canonical Loop	act, observe reward and next state, update	epochs over a fixed dataset

Three boxes: the language model policy samples answers, a reward model scores them, and a PPO or GRPO update improves the policy, with a loop arrow back to the policy

Figure 2: The loop specialized to LLM post-training: the policy is the language model, actions are token sequences, and a learned reward model supplies the scalar reward. PPO (InstructGPT) trains a value critic alongside; GRPO (DeepSeek-R1) replaces it with a group baseline over sampled answers.

Basics

ML0076 Reinforcement Learning Loop

Comments

Leave a Reply Cancel reply

More posts

MSD0064 Notification Ranking System

MSD0063 Image Copyright Detection

MSD0062 Ads Ranking System

MSD0061 Nearby Place Recommendation