Timeline of Large Language Models
The path from neural language modeling in 2000 to transformers, scaling laws, instruction tuning, downloadable weights, and reasoning-focused models in 2025.
- Category
- Engineering & Technology
- Tag
- computing
- Events
- 22 events 2000 – 2025
- Generated
- by gpt-6
2000: Bengio’s team learns distributed word representations
Yoshua Bengio and colleagues presented a neural probabilistic language model at NIPS that learned word representations together with next-word probabilities. Sharing information between similar words helped the model generalize beyond word sequences observed during training.
https://en.wikipedia.org/wiki/Language_model#Neural_models2013: Word2vec makes word embeddings practical
Tomas Mikolov and colleagues introduced efficient methods for learning vector representations of words from large text collections. Word2vec helped establish reusable learned representations as an alternative to treating each word as an unrelated symbol.
https://en.wikipedia.org/wiki/Word2Vec2014: Attention improves neural translation
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio released a paper that let a translation model attend to different source positions while generating text. The mechanism reduced reliance on compressing an entire input sentence into one fixed vector.
https://en.wikipedia.org/wiki/Attention_(machine_learning)2017: The Transformer centers sequence modeling on attention
Ashish Vaswani and colleagues published "Attention Is All You Need." The Transformer used attention instead of recurrent layers for translation, providing an architecture that could be trained efficiently in parallel and later adapted to language pretraining.
https://en.wikipedia.org/wiki/Transformer_(deep_learning)2018: Generative pretraining transfers to language tasks
OpenAI introduced the original Generative Pre-trained Transformer, or GPT. A transformer trained to predict text was then fine-tuned on supervised tasks, demonstrating transfer from broad language pretraining to more specialized problems.
https://en.wikipedia.org/wiki/GPT-12018: BERT learns from both sides of context
Google researchers introduced Bidirectional Encoder Representations from Transformers, known as BERT. Predicting masked words allowed the model to use context on both sides, improving transfer to tasks such as question answering and sentence classification.
https://en.wikipedia.org/wiki/BERT_(language_model)2019: GPT-2 demonstrates broader text generation
OpenAI introduced GPT-2 and initially released only smaller versions, citing concerns about misuse. The full model followed in November, making both the capabilities of larger text generators and decisions about model access part of the public debate.
https://en.wikipedia.org/wiki/GPT-22019: T5 casts language tasks as text transformations
Google researchers introduced the Text-to-Text Transfer Transformer, known as T5. A common text-in, text-out formulation let translation, summarization, and classification share a model and training framework.
https://en.wikipedia.org/wiki/T5_(language_model)2020: Scaling laws quantify training tradeoffs
Jared Kaplan and colleagues reported empirical relationships between language-model loss, parameter count, training data, and computation. The results gave researchers a framework for allocating training budgets, rather than treating model size as the only variable.
https://en.wikipedia.org/wiki/Neural_scaling_law2020: Retrieval supplies documents during generation
Patrick Lewis and colleagues introduced retrieval-augmented generation for knowledge-intensive tasks. Their system combined a trained text generator with retrieved passages, allowing external documents to contribute information beyond the model’s stored parameters.
https://en.wikipedia.org/wiki/Retrieval-augmented_generation2020: GPT-3 demonstrates learning from prompt examples
OpenAI described GPT-3, a language model with 175 billion parameters. It performed many evaluated tasks from instructions or examples placed in the prompt, without updating its weights for each task, while still showing substantial limitations.
https://en.wikipedia.org/wiki/GPT-32021: Instruction tuning improves unseen-task performance
Google researchers described the Finetuned Language Net, known as FLAN. Fine-tuning a language model on tasks expressed as instructions improved its ability to follow instructions for tasks withheld from training.
https://en.wikipedia.org/wiki/Large_language_model#Fine-tuning2022: InstructGPT learns from human preferences
OpenAI published a method combining human demonstrations, rankings of model outputs, and reinforcement learning. The resulting InstructGPT models were better aligned with evaluator preferences than their pretrained counterparts on the studied tasks.
https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback2022: Chinchilla revises compute-optimal training
DeepMind’s Chinchilla study found that many large language models had been trained on too little data for their size. A smaller model trained on more tokens outperformed larger comparison models, shifting attention toward the balance between data and parameters.
https://en.wikipedia.org/wiki/Chinchilla_(language_model)2022: ChatGPT opens conversational generation to the public
OpenAI released ChatGPT as a public research preview on November 30. The conversational interface allowed follow-up prompts and revisions, bringing instruction-tuned language models to an audience far beyond machine-learning researchers.
https://en.wikipedia.org/wiki/ChatGPT2023: GPT-4 extends language modeling to images
OpenAI introduced GPT-4, describing a model that accepted text and image inputs and generated text outputs. Image access was initially limited, so the announcement and widespread availability of visual interaction were separate milestones.
https://en.wikipedia.org/wiki/GPT-42023: Llama 2 makes model weights broadly downloadable
Meta released Llama 2 weights for research and commercial use under a custom community license. Developers could run and adapt the models themselves, although the license’s conditions distinguished the release from unrestricted open-source software.
https://en.wikipedia.org/wiki/Llama_(language_model)2023: Mixtral releases a sparse mixture of experts
Mistral AI released Mixtral 8x7B with downloadable weights under the Apache 2.0 license. Its mixture-of-experts design activated selected components for each token, separating total parameter count from the amount of computation used per token.
https://en.wikipedia.org/wiki/Mistral_AI#Mixtral_8x7B2024: Gemini previews million-token context
Google introduced Gemini 1.5 with a limited preview supporting context windows of up to one million tokens. Processing long text, audio, or video inputs expanded the material a model could consider in one interaction.
https://en.wikipedia.org/wiki/Gemini_(language_model)2024: Llama 3.1 releases a 405-billion-parameter model
Meta released Llama 3.1, including a model with 405 billion parameters and smaller models with expanded context. Downloadable weights widened options for self-hosting high-capacity models, while use remained subject to Meta’s license.
https://en.wikipedia.org/wiki/Llama_(language_model)2024: OpenAI introduces reasoning-focused o1 models
OpenAI released an o1 preview trained to spend more computation before producing an answer. The release emphasized reinforcement learning and inference-time reasoning for difficult tasks, alongside the established strategy of scaling pretraining.
https://en.wikipedia.org/wiki/OpenAI_o12025: DeepSeek releases reasoning models and distilled variants
DeepSeek released R1 and described experiments using reinforcement learning to improve reasoning. Downloadable weights and smaller distilled models made the approach accessible to developers with a range of hardware budgets.
https://en.wikipedia.org/wiki/DeepSeek_(AI)