What does a 2.8 trillion token corpus change for Indian language models?

The corpus in numbers

The release spans 22 languages listed in India's Constitution. Together they serve as first languages for 97 percent of Indians, or nearly 1.2 billion people at the 2011 Census. The corpus holds 2,601.0 billion native-script tokens and 224.6 billion romanized tokens as counted by the Gemma-4 tokenizer. Twelve languages appear in both native script and Latin script: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu. Ten languages appear only in native script: Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi. That yields 34 language-script configurations in total.

Token counts reflect the retained corpus after quality filtering and include repaired rewrites. They are not a training-run budget. Counts also shift with the tokenizer, so corpus inventory and training-token budgets should be read separately.

Two pipelines, one quality bar

Source material comes from 15 Nemotron-CC-v2 subsets covering web text, question-answer pairs, mathematics, code, and other specialized domains. NVIDIA's open-source NeMo Data Designer orchestrates translation, assessment, and repair while preserving every source identifier for traceability.

For the twelve dual-script languages the workflow runs in four stages. Gemma-4-31B-it translates full documents so context across paragraphs, tables, and equations stays intact. Prompts explicitly mark code, equations, URLs, and formatting as content to preserve. The same model then scores each translation against its English source using a rubric inspired by Multidimensional Quality Metrics. The rubric covers meaning, fluency, terminology, style, and formatting. Only outputs scoring at least 75 out of 100 are kept. Lower-scoring outputs receive feedback-guided repairs and are reassessed against the same threshold. This repair loop recovered 438.4 billion tokens across native-script and romanized data, already folded into the corpus total.

Accepted native-script translations become the seed for romanized data. A subset is converted to Latin script and checked against the native-script input. This two-step approach produced fewer hallucinated additions and less residual English than generating romanized text directly from English in initial human comparisons.

The other ten languages use a separate workflow built around Indic Translate, a document-level translation model for English and 22 Indian languages from Bodhan AI, a nonprofit incubated at IIT Madras. Gemma-4-31B-it still assesses each translation. Eight languages keep the whole-document approach. For longer Santali documents the pipeline translates overlapping windows and reconstructs the document. For Manipuri it translates into Bengali script before converting to Meitei Mayek, the native script. These workflows retain outputs scoring at least 75 out of 100 and produce native-script data only, without feedback-based rewriting.

What the model gained

Nemotron 3.5 Super VL added 10 billion Indic tokens during continued pretraining, drawn from a twelve-language subset of this corpus plus natural Indic text. The configured budget was 100 billion training tokens with 10 percent allocated to the Indic mixture. Scores increased on all five reported Indic knowledge and reasoning benchmarks: MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy, and Indic GSM8K. Gains range from 4.98 to 9.82 percentage points. The comparison reflects a stage that combines synthetic and natural Indic data rather than isolating the synthetic corpus alone. Language coverage differs by benchmark; these are stage-level checkpoints, not final model releases. GSM8K excludes Odia because evaluation exceeded the serving context limit.

Beyond the continued-pretraining stage, the released Nemotron 3.5 Super VL scores higher than Nemotron 3 Super on five of six task benchmarks spanning knowledge, reasoning, mathematics, code, and translation. Instruction following on IFEval 1.5 dipped 2.04 points. The model also improves across all four IndiVibe dimensions: usefulness, fluency, language and script fidelity, and appropriate verbosity. IndiVibe is Sarvam AI's open benchmark of assistant-style prompts in Indian languages.

Using the corpus and the recipe

Developers can load specific language-script configurations such as hi-Deva-IN for Hindi in Devanagari or hi-Latn-IN for romanized Hindi. Hugging Face Datasets supports streaming so records can be inspected without downloading the entire corpus. Pinning a dataset revision enables reproducible experiments. For a first training study the announcement suggests comparing mixtures at a fixed token budget while tracking natural versus synthetic and native versus romanized shares alongside English capability retention and language quality.

The open-source Nemotron Bharat translation plugin for NeMo Data Designer exposes translation and evaluation prompts, optional feedback-based rewriting, document windowing, and script-remapping support. Teams can adapt prompts to their domain, set rewrite parameters, and bring their own source documents. Generation and corpus acceptance are separate steps: the plugin returns candidates and scores while the filtering policy decides which records enter the training corpus.

Where the explanation stops

Translation broadens access to source material but does not replace text originally written in Indian languages or guarantee coverage of local knowledge and culture. Quality assessment is primarily automated with sampled human review whose coverage varies across languages. For the twelve-language workflow the translator also serves as the judge. Retained text can still contain errors. The ten-language workflow needs broader native-speaker validation. Source-ID checks remove processing duplicates; they are not semantic deduplication of similar content. Evaluating the full corpus including recovered rewrites and the other ten languages remains a next step for controlled studies.