Offline reinforcement learning trains a policy entirely from a fixed dataset of previously collected experience — trajectories of states, actions, and rewards — without any further interaction with the environment during training. This differs from the usual online RL loop, where an agent continually acts, observes the outcome, and updates its policy from its own fresh experience.
Learning offline is attractive wherever acting in the real environment during training is unsafe, slow, or expensive — physical robots, for instance — but it introduces its own difficulty: a policy can learn to prefer actions the dataset never actually tried, and there is no way to check, from the fixed dataset alone, whether that untested action would really work as well as the model expects.
Related papers
- Hafner, Yan, and Lillicrap, “Training Agents Inside of Scalable World Models”, arXiv:2509.24527 (2025) — proposes obtaining a diamond in Minecraft from only offline data as a challenge task, and reports the first agent to solve it without any environment interaction. Covered in 2026-w32.