Microsoft Phi-4: Built From Textbooks, Scoped in Writing

What Phi-4 is

Phi-4 is Microsoft's small language model, MIT-licensed, with a 16K token context window and pricing around $0.07 per million input tokens and $0.14 output through third-party providers. The Phi line's distinguishing idea has always been the training data rather than the architecture: Microsoft build these models substantially on newly created synthetic textbook-like data, written for the purpose of teaching mathematics, coding, common-sense reasoning and general knowledge of the world — science, daily activities, theory of mind.

Microsoft state the scope, including what it is not for

Microsoft name three primary cases: memory or compute constrained environments, latency-bound scenarios, and reasoning and logic. They then give an out-of-scope section stating that the model is not specifically designed or evaluated for all downstream purposes, instructing developers to evaluate and mitigate for accuracy, safety and fairness before deployment — particularly in high-risk scenarios — to observe applicable privacy and trade-compliance law, and to note the model's focus on English.

Context window

The context window is 16K tokens.

LMArena

On LMArena's text leaderboard Phi 4 has 1256 Elo (rank 323); by category, 1306 Elo in Coding (rank 310), 1265 Elo in Math (rank 290), 1268 Elo in Expert (rank 293) and 1210 Elo in Creative Writing (rank 322).

Training details

Microsoft's card describes phi-4 as a 14-billion-parameter dense decoder-only Transformer trained on 9.8 trillion tokens over 21 days on 1,920 H100-80G GPUs, between October and November 2024, and released on 12 December 2024. Training data cut off in June 2024 or earlier for publicly available data. The data mix extends Phi-3's: rigorously filtered public documents and educational data, newly created synthetic textbook-like data, acquired academic books and Q&A datasets, and chat-format supervised data; multilingual data is about 8% of the total. Alignment used supervised fine-tuning and iterative direct preference optimisation, and Microsoft's AI Red Team tested it with jailbreaks, encoding-based, multi-turn and adversarial-suffix attacks.

Microsoft’s figures

On OpenAI's simple-evals, Microsoft report MMLU 84.8, GPQA 56.1, MATH 80.4, MGSM 80.6, HumanEval 82.6, DROP 75.5 and SimpleQA 3.0, against Phi-3's 77.9, 31.2, 44.6, 53.5, 67.8, 68.3 and 7.6. The card notes the model is best suited to chat-format prompts, is primarily English and not intended for multilingual use, and that most of its code training data is Python using common packages.

Specifications and limits

Phi-4 takes text input, is best suited to chat-format prompts and has a context length of 16K tokens. It is a static model trained on an offline dataset, released under the MIT licence. Microsoft list its primary uses as memory- or compute-constrained environments, latency-bound scenarios, and reasoning and logic, primarily in English. In the same simple-evals table Microsoft give Qwen 2.5 14B Instruct MMLU 79.9 and GPQA 42.9, and GPT-4o MMLU 88.1, GPQA 50.6 and HumanEval 90.6. Microsoft note that Llama-3.3 70B's MATH and HumanEval scores in the table are lower than Meta's own reports of 77 and 88, which they attribute to simple-evals' strict formatting requirement.

Lineage

Phi-4 was developed by Microsoft Research and follows Phi-3, whose 14B model Microsoft use as the baseline in the card's table. A technical report on arXiv (2412.08905) describes the data and safety alignment in more detail.

Official Resources