Llama 3.1 Nemotron 70B Instruct: NVIDIA's Helpfulness-Tuned Llama

What this model is

Llama 3.1 Nemotron 70B Instruct is NVIDIA's helpfulness-tuned version of Meta's Llama 3.1 70B, released under the Llama 3.1 licence with a 128K token context window. NVIDIA describe its purpose with unusual directness: it is a demonstration of their techniques for improving helpfulness in general-domain instruction following, and they state explicitly that it has not been tuned for performance in specialised domains such as mathematics.

What the demonstration achieved

As of 1 October 2024, NVIDIA reported this model ranking first on all three automatic alignment benchmarks — including the verified tab of AlpacaEval 2 LC — ahead of GPT-4o and Claude 3.5 Sonnet. The technique is documented in an accompanying paper. NVIDIA's illustration: the model answers "How many r in strawberry?" correctly without specialised prompting or extra reasoning tokens.

What automatic alignment benchmarks measure, and do not

Automatic alignment benchmarks like AlpacaEval 2 LC assess how well responses match preferred-response patterns, judged by a model rather than a human panel. They measure helpfulness and presentation — being clear, complete, well-structured, appropriately hedged. They do not measure whether the mathematics is right.

LMArena and price

On LMArena's text leaderboard Llama 3.1 Nemotron 70B Instruct has 1299 Elo (rank 290); by category, 1328 Elo in Coding (rank 295), 1279 Elo in Math (rank 278), 1296 Elo in Expert (rank 272) and 1276 Elo in Creative Writing (rank 273). It is around not published per million tokens for both input and output.

Training and NVIDIA’s figures

NVIDIA trained the model with reinforcement learning from human feedback, specifically REINFORCE, using Llama-3.1-Nemotron-70B-Reward and HelpSteer2-Preference prompts, starting from Llama-3.1-70B-Instruct. The HelpSteer2 dataset holds 21,362 prompt-responses, 20,324 for training and 1,038 for validation. NVIDIA report Arena Hard 85.0, AlpacaEval 2 LC 57.6 and MT-Bench (GPT-4-Turbo) 8.98, against Llama-3.1-70B-Instruct's 55.7, 38.1 and 8.22, GPT-4o's 79.3, 57.5 and 8.74, and Claude 3.5 Sonnet's 79.2, 52.4 and 8.81. The mean response length on MT-Bench is 2,199.8 characters, against 1,728.6 for the base Llama model.

Licence and deployment

Use is governed by the NVIDIA Open Model License, with the Llama 3.1 Community License Agreement as additional information. The model accepts up to 128K tokens of input and produces up to 4K tokens of output. NVIDIA publish a Hugging Face Transformers conversion as Llama-3.1-Nemotron-70B-Instruct-HF and offer free hosted inference with an OpenAI-compatible API at build.nvidia.com. For self-hosting through the NeMo Framework with TensorRT-LLM, NVIDIA list at least four 40GB or two 80GB GPUs and 150GB of free disk space; tested hardware is H100, A100 80GB and A100 40GB. The method is described in the HelpSteer2-Preference paper, arXiv 2410.01257.

Related NVIDIA work

The card links the research the model builds on: HelpSteer, HelpSteer2 and HelpSteer2-Preference for the preference data, and the SteerLM method. NVIDIA's NeMo deployment walk-through uses the nemo:24.05.llama3.1 container, converts the checkpoint to TensorRT-LLM and serves it through Triton on port 8000. NVIDIA state that developers should work with their model team to check the model meets the requirements of their industry and use case, and publish Explainability, Bias, Safety and Security, and Privacy sub-cards.

Official Resources