Timeline of Reinforcement Learning
Learning through actions and rewards, from dynamic programming in 1957 to temporal-difference methods, game-playing systems, human feedback, and reasoning models in 2025.
- Category
- Engineering & Technology
- Tag
- computing
- Events
- 22 events 1957 – 2025
- Generated
- by gpt-6
1957: Bellman develops dynamic programming
Richard Bellman published "Dynamic Programming," organizing sequential decisions around the value of future outcomes. His recursive treatment of optimal decisions became a mathematical foundation for later reinforcement-learning methods.
https://en.wikipedia.org/wiki/Bellman_equation1959: Samuel learns evaluations from checkers experience
Arthur Samuel reported a checkers program that revised its evaluation of positions using experience. Its treatment of future consequences anticipated ideas later developed more systematically in reinforcement learning.
https://en.wikipedia.org/wiki/Arthur_Samuel_(computer_scientist)1983: Actor-critic learning balances a pole
Andrew Barto, Richard Sutton, and Charles Anderson described adaptive elements that learned to balance a simulated pole. One component selected actions while another evaluated them, demonstrating an early actor-critic architecture.
https://en.wikipedia.org/wiki/Reinforcement_learning1988: Sutton formalizes temporal-difference learning
Richard Sutton published methods that updated predictions from differences between successive estimates. Temporal-difference learning could learn before an episode ended, combining aspects of sampling experience and estimating future value.
https://en.wikipedia.org/wiki/Temporal_difference_learning1989: Watkins introduces Q-learning
Christopher Watkins presented Q-learning in his doctoral thesis. The method learned values for state-action pairs and could estimate an optimal policy while following a different exploratory policy, under the assumptions of the tabular setting.
https://en.wikipedia.org/wiki/Q-learning1992: TD-Gammon learns backgammon through self-play
Gerald Tesauro reported TD-Gammon, which trained a neural-network evaluation function by playing backgammon against itself. Temporal-difference learning produced strong play without requiring a large collection of expert-labeled positions.
https://en.wikipedia.org/wiki/TD-Gammon1992: Williams develops REINFORCE policy gradients
Ronald Williams published algorithms for adjusting stochastic policies in the direction of higher expected reward. The REINFORCE family estimated policy gradients from sampled actions and returns, providing a route to learning policies directly.
https://en.wikipedia.org/wiki/Policy_gradient_method1998: Sutton and Barto unify the field
Richard Sutton and Andrew Barto published "Reinforcement Learning: An Introduction." The textbook brought together rewards, value functions, exploration, and learning from interaction in a common framework for researchers and students.
https://en.wikipedia.org/wiki/Reinforcement_learning2002: Finite-time analysis strengthens bandit methods
Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer analyzed simple algorithms for the multi-armed bandit problem. Their upper-confidence approach balanced trying uncertain actions against choosing actions with high observed rewards.
https://en.wikipedia.org/wiki/Multi-armed_bandit2013: Deep Q-learning learns Atari from pixels
Volodymyr Mnih and colleagues released a paper on learning Atari games from screen images and rewards. A convolutional network approximated action values, while stored experience helped train the system from reused interactions.
https://en.wikipedia.org/wiki/Q-learning2015: Trust regions constrain policy updates
John Schulman and colleagues introduced trust region policy optimization. The method limited how far a policy changed in one update, addressing instability when optimizing large parameterized policies from sampled experience.
https://en.wikipedia.org/wiki/Proximal_policy_optimization#TRPO2015: DeepMind broadens its Atari evaluation
DeepMind published a deep Q-network study covering 49 Atari games. The expanded evaluation demonstrated a single learning approach across varied games, while also showing that performance remained uneven across tasks.
https://en.wikipedia.org/wiki/Deep_reinforcement_learning2016: Asynchronous learners share a neural network
Volodymyr Mnih and colleagues presented asynchronous methods for deep reinforcement learning, including the asynchronous advantage actor-critic algorithm. Parallel workers gathered different experiences and updated shared parameters, offering an alternative to replay-based training.
https://en.wikipedia.org/wiki/Deep_reinforcement_learning2016: AlphaGo defeats Lee Sedol
DeepMind’s AlphaGo defeated Lee Sedol by four games to one in a five-game Go match. It combined learning from human games, reinforcement learning through self-play, and tree search rather than relying on any one technique alone.
https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol2017: Human preferences become a learning signal
Paul Christiano and collaborators trained agents using comparisons supplied by people. A learned reward model translated preferences over short behavior clips into feedback for reinforcement learning, reducing dependence on a hand-written reward function.
https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback2017: Proximal policy optimization simplifies policy learning
John Schulman and colleagues introduced proximal policy optimization. A clipped training objective discouraged overly large policy changes while retaining a comparatively simple procedure that could reuse batches of collected experience.
https://en.wikipedia.org/wiki/Proximal_policy_optimization2017: AlphaGo Zero learns without expert games
DeepMind introduced AlphaGo Zero, which learned Go through self-play without a dataset of human games. The system combined a neural network with search and used the results of its own games to improve both move selection and position evaluation.
https://en.wikipedia.org/wiki/AlphaGo_Zero2018: Soft actor-critic encourages exploratory policies
Tuomas Haarnoja and colleagues introduced soft actor-critic for continuous control. The method optimized both expected reward and policy entropy, encouraging exploration while learning from previously collected experience.
https://en.wikipedia.org/wiki/Deep_reinforcement_learning2019: MuZero learns a model for planning
DeepMind released the MuZero preprint, describing an agent that learned a model useful for predicting rewards, values, and policies. It used that model for search without being supplied the environment’s transition rules.
https://en.wikipedia.org/wiki/MuZero2022: InstructGPT applies feedback to language models
OpenAI described InstructGPT, combining supervised demonstrations with a reward model learned from human rankings and reinforcement learning. Evaluators preferred its responses to those from the base language model on the studied prompt distribution.
https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback2023: AlphaDev discovers faster sorting routines
DeepMind used reinforcement learning to search for assembly programs implementing small sorting routines. Some resulting improvements were incorporated into the LLVM standard C++ library, connecting game-like program search with production software.
https://en.wikipedia.org/wiki/AlphaDev2025: DeepSeek studies reinforcement-trained reasoning
DeepSeek introduced R1-Zero and R1. R1-Zero explored reinforcement learning without a preliminary supervised fine-tuning stage, while R1 combined supervised data and reinforcement learning to improve reasoning and response usability.
https://en.wikipedia.org/wiki/DeepSeek_(AI)