NVIDIA Opens 10 Trillion‑Token Dataset to Power AI Agents

Why Agent Performance Is a Data Problem, Not a Weights Problem

NVIDIA argues that an agent that merely completes code is not a true agent; real agents need diverse, high‑quality data covering software‑engineer traces, tool‑failure samples, multi‑step reasoning, retrieval, safety, user simulation, and workflow execution. The author echoes this view, summarizing that model weights define an agent’s architecture while data determines its capability.

Nemotron‑CC‑v2: Massive Common‑Crawl‑Based Corpus

The Nemotron‑CC‑v2 collection aggregates over 6.5 trillion tokens from English Common Crawl (3.36 T), synthetic CC (1.26 T), Diverse QA (0.69 T) and its translations (0.56 T), plus mathematics, code, and various SFT subsets. It is fresher than standard Common Crawl, adding eight snapshots from 2024‑2025, and includes synthetic rewrites generated by Qwen3‑30B‑A3B to reduce over‑fitting to specific web styles.

Nemotron‑CC‑Math: A Five‑Step Pipeline for High‑Quality Math Data

The Nemotron‑CC‑Math‑v1 dataset contains 133 billion high‑quality math tokens extracted from 98 snapshots (2014‑2024) of Common Crawl, covering over 980 k domains. NVIDIA processes the data through:

Lynx rendering to convert HTML to structured text while preserving formula layout.

LLM cleaning with Phi‑4 (14B) to strip boilerplate and standardize formulas to LaTeX.

Quality classification using the FineMath classifier (1–5 scores per page).

Deduplication via MinHash + LSH (NeMo‑Curator).

Leakage detection against MATH, GSM8K, MMLU, and MMLU‑Pro test sets.

Benchmark results show a jump from 34.6 to 44.2 on the MATH benchmark (≈10 percentage‑point absolute gain), with comparable improvements on GSM8K, HumanEval+, MBPP+, MMLU, and MMLU‑STEM.

from datasets import load_dataset
ds = load_dataset("nvidia/Nemotron-CC-Math-v1", "4plus", streaming=True)

Nemotron‑Personas: 53 Million Synthetic User Profiles

This component synthesizes 53 million persona records covering 10 countries, 9 language variants, and a total population of 2.4 billion. Each persona is generated through a four‑stage pipeline:

OCEAN five‑personality sampling.

Probabilistic graphical model (PGM) sampling to respect real‑world demographic correlations.

LLM‑driven generation of structured persona attributes (culture, skills, goals, hobbies).

Synthesis of nine descriptive dimensions (career, finance, health, sport, art, travel, diet, brief, detailed).

These personas are used for long‑context construction, tool‑call rollout training, logical reasoning data, safety‑reject data (SSCR), and general dialogue/instruction following. A notable case is Nemotron‑Nano‑9B‑v2‑Japanese, which leveraged the Japan persona set to achieve top performance on Japanese LLM leaderboards.

Prompt Atlas: Visualizing 10 Trillion Tokens

NVIDIA provides an interactive Prompt Atlas where each point represents a prompt sample, clustered by semantic similarity. Users can color by dataset, pipeline stage, or domain, filter via sidebar metadata, write SQL predicates, or perform natural‑language semantic search. The underlying embedding model is nemotron-llama-embed-v2 with dimensionality reduction.

NeMo Data Designer: Building Your Own Synthetic Datasets

The open‑source NeMo Data Designer (Apache 2.0) offers multiple samplers (statistical, LLM, seed‑data), column‑level dependency control, quality validation (Python/SQL/LLM‑as‑a‑judge), preview, and an asynchronous engine that adapts concurrency per provider/model. pip install data-designer Example configuration to generate product categories and LLM‑crafted reviews:

import data_designer.config as dd
from data_designer.interface import DataDesigner

data_designer = DataDesigner()
config_builder = dd.DataDesignerConfigBuilder()

# Sample a product category
config_builder.add_column(
    dd.SamplerColumnConfig(
        name="product_category",
        sampler_type=dd.SamplerType.CATEGORY,
        params=dd.CategorySamplerParams(values=["Electronics", "Clothing", "Home & Kitchen", "Books"]),
    )
)

# Generate a personalized review using an LLM
config_builder.add_column(
    dd.LLMTextColumnConfig(
        name="review",
        model_alias="nvidia-text",
        prompt="Write a brief product review for a {{ product_category }} item you recently purchased.",
    )
)

preview = data_designer.preview(config_builder=config_builder)
preview.display_sample_record()

As of June 1, the dataset contains over 2.6 T tokens, and a Claude Code skill ( npx skills add NVIDIA-NeMo/DataDesigner) enables agents to invoke the designer directly from the Claude Code environment.

Impact and Adoption

At ICML 2026, NVIDIA’s Nemotron family was cited in 145 papers, with 74 NVIDIA‑authored papers accepted and roughly 2 000 papers referencing NVIDIA GPUs. External projects such as Sakana AI (Fugu series), KiloCode (token‑cost reduction), NAVER (Korean model training), and Together AI (public Nemotron hosting) demonstrate rapid ecosystem uptake.

One‑sentence takeaway: Nemotron has evolved from “NVIDIA’s model” into an open research stack that shares weights, data, and training recipes for the community.