Research Papers
23 resources
Foundational work, in rough order of what you need first. Every entry links to the paper itself — arXiv where the paper is on arXiv — rather than to a summary of it.
Attention Is All You Need
The transformer architecture
Introduced the architecture nearly every model on this site is a descendant of. Dispensed with recurrence entirely in favour of self-attention.
BERT
Pre-training of deep bidirectional transformers
Showed that pre-training a bidirectional encoder then fine-tuning it beat task-specific architectures across the board.
Language Models are Few-Shot Learners
The GPT-3 paper
The scale argument: a big enough language model performs new tasks from a handful of examples in the prompt, without gradient updates.
LLaMA
Open and efficient foundation language models
Meta's open-weights family, and the paper that made small-but-well-trained a mainstream position.
Mixtral of Experts
A sparse mixture-of-experts language model
Routes each token through a subset of experts, so inference cost stays near a much smaller dense model.
Mamba
Linear-time sequence modelling with selective state spaces
The most credible non-transformer challenger: linear scaling in sequence length rather than quadratic.