Landmark AI Research Papers, Explained Simply

In short: a small number of research papers explain how today's AI works. The transformer ("Attention Is All You Need", 2017) is the design behind ChatGPT, Claude and Gemini; GPT-3 (2020) showed that large models can learn a task from a few examples in the prompt; InstructGPT (2022) showed how human feedback turns a text predictor into a helpful assistant; the scaling-law papers (2020 and 2022) showed how size, data and compute trade off. This guide lists the landmark papers in order, says in plain words what each one changed, and links to the original on arXiv, where all of them are free to read.

Why read AI research papers?

Papers are the original source for how AI systems work; summaries, including this one, are second-hand. You do not need them to use AI tools, but they explain why models behave as they do - why they have context windows, why examples in a prompt help, why chatbots are trained on human ratings. Almost every major AI paper is posted free on arXiv, a public archive of research papers run by Cornell University.

The landmark AI papers at a glance

Which papers started modern deep learning?

AlexNet showed deep networks could beat older methods on images; word2vec and ResNet made deep learning work for words and for very deep networks.

• AlexNet (2012). Krizhevsky, Sutskever and Hinton's ImageNet classification paper won the ImageNet image-recognition challenge by a wide margin using a deep convolutional network trained on graphics cards. See computer vision explained.
• Word2vec (2013). Efficient Estimation of Word Representations in Vector Space turned each word into a list of numbers (an embedding) so that words with similar meanings get similar numbers - the idea behind how every language model represents text.
• ResNet (2015). Deep Residual Learning for Image Recognition added "shortcut" connections that let signals skip layers, making it possible to train networks with up to 152 layers. The same trick is used inside transformers.

Which papers explain how ChatGPT, Claude and Gemini work?

Three papers cover the core: the transformer architecture, large-scale pre-training, and training on human feedback.

• Attention Is All You Need (2017). This Google paper introduced the transformer. Instead of reading text one word at a time, it uses attention to let every word look at every other word in the input at once, which trains faster and handles long-range connections better. See Attention Is All You Need explained and how LLMs work.
• BERT (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding pre-trained a transformer on unlabelled text by hiding words and predicting them from the context on both the left and the right. It was built for understanding text - classifying it, finding answers in it - rather than writing it.
• GPT-3 (2020). Language Models are Few-Shot Learners trained a model with 175 billion parameters, which its authors describe as 10 times more than any previous non-sparse language model, and showed it could do new tasks from a few examples in the prompt, with no retraining. See few-shot prompting explained.
• InstructGPT (2022). Training language models to follow instructions with human feedback fine-tuned GPT-3 using people's ratings of its answers - reinforcement learning from human feedback (RLHF). In the authors' human evaluations, outputs from a 1.3-billion-parameter InstructGPT model were preferred to those of the 175-billion-parameter GPT-3. This is the approach behind ChatGPT-style assistants.

Which papers explain how big AI models should be?

The scaling-law papers measured how a model's performance changes as you add parameters, data and computing power.

• Kaplan et al. (2020). OpenAI's Scaling Laws for Neural Language Models found that a language model's error falls as a power law with model size, dataset size and the compute used for training - smoothly and predictably.
• Chinchilla (2022). DeepMind's Training Compute-Optimal Large Language Models found that model size and training data should grow equally: for every doubling of model size, the number of training tokens should also double. Its 70-billion-parameter Chinchilla model used the same compute budget as the larger Gopher, trained on about four times more data.

See AI scaling laws explained.

Which papers are behind prompting, tools and agents?

• Chain-of-thought (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models showed that worked, step-by-step examples in a prompt improve answers on multi-step problems. See chain-of-thought prompting explained.
• RAG (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks combined a language model with a search step, so it retrieves relevant documents before answering - the pattern behind "chat with your documents" tools.
• ReAct (2022). ReAct: Synergizing Reasoning and Acting in Language Models had a model alternate between reasoning and taking actions, such as searching, and reading the results - an early form of today's AI agents. See what is MCP for how agents connect to tools now.
• LoRA (2021). LoRA: Low-Rank Adaptation of Large Language Models fine-tunes a model by training small added pieces instead of all its weights. Compared with fully fine-tuning GPT-3 175B, its authors report up to 10,000 times fewer trainable parameters and a third of the GPU memory.

Which papers are behind AI image generators?

Diffusion models learn to turn random noise into an image, step by step. Denoising Diffusion Probabilistic Models (Ho, Jain and Abbeel, 2020) showed this approach could produce high-quality images; it is the basis of the diffusion family of image generators. For understanding images rather than making them, see Vision Transformers explained and Segment Anything explained.

Which papers cover AI safety?

• Concrete Problems in AI Safety (2016). This paper grouped accident risks into five research problems: avoiding side effects, avoiding reward hacking, scalable supervision, safe exploration and distributional shift.
• Constitutional AI (2022). Anthropic's Constitutional AI: Harmlessness from AI Feedback trained a harmless assistant with no human labels identifying harmful outputs; the only human oversight was a list of rules or principles.
• GPT-4 Technical Report (2023). OpenAI's report describes GPT-4's capabilities, evaluations and safety work. Unlike the papers above, it is one company's report on its own model and does not give full details of the model's size or training.

See our AI ethics guide for how these ideas fit into wider AI ethics.

How do you read an AI research paper as a beginner?

Read the abstract, introduction and conclusion first; skim the rest.

1. Abstract - a one-paragraph summary of what was done and found.
2. Introduction - the problem and why it matters, usually in plainer language.
3. Figures and tables - the main results, often clearer than the text.
4. Conclusion and limitations - what the authors claim, and what they say it does not show.
5. Method - skim for the general idea; the equations can wait.

The abstract page on arXiv shows the submission date and version history, so you can see when a paper was first posted and whether it was revised.

Words to know

• arXiv - a free public archive where researchers post papers.
• Abstract - the short summary at the top of a paper.
• Transformer - the attention-based design behind modern language models.
• Attention - the mechanism that lets each word weigh every other word.
• Pre-training - the first, large training stage on raw text.
• Fine-tuning - further training on a smaller, specific dataset.
• RLHF - reinforcement learning from human feedback; training on people's ratings.
• Parameters - the internal numbers a model learns during training.
• Scaling law - a measured rule for how performance changes with size, data and compute.
• Diffusion model - a model that generates images by removing noise step by step.

Related guides

• How LLMs work
• Attention Is All You Need explained
• AI scaling laws explained
• Chain-of-thought prompting explained
• AI ethics guide