What makes AstaBrief generate a cited report in under a minute?

A model built for the way scientists actually ask

Researchers do not type short keywords into Asta. They bring substantial context, multiple constraints, and relationships between concepts. The team analyzed hundreds of thousands of real queries and found that expert users frequently specify particular methods, populations, or settings. Many return to generated reports later, treating them as working research artifacts rather than one-off answers. AstaBrief was trained on 90K filtered research-focused queries drawn from actual Asta usage logs. Beta-tester traffic, bot traffic, overly short queries, non-English requests, non-scientific prompts, and prompts containing personal information were removed before training began.

Training data that demonstrates grounding

Supervised fine-tuning started from Qwen3-8B. The 47K usable SFT examples came from running the multi-step ScholarQA pipeline on the filtered queries. That pipeline retrieved relevant literature, organized material into sections, and used a backing model to synthesize a cited report. The backing models were a mix of proprietary systems: Claude 3.5 Sonnet, Claude 3.7 Sonnet, o3, o4-mini, and GPT-4.1.

For direct preference optimization the team needed pairs of reports where one was clearly better. They built about 6K such pairs from a separate query subset. One report per query came from the existing ScholarQA pipeline, typically backed by Claude 3.5 Sonnet or 3.7 Sonnet. The competing report came from o3, o4-mini, DeepSeek-V3, or DeepSeek-R1. Two judge models — GPT-4.1 and DeepSeek-R1 — compared each pair and picked a winner. The judges agreed with human preferences 95 percent of the time. Only pairs where both judges agreed were kept.

A single filter that moved the needle

Four statistics-based filters were tested to identify weaker synthetic training examples. Output-to-input token ratio flagged answers generating too much text from too little evidence. Citation relevance averaged the retrieval relevance scores of cited papers. Citation density measured the share of statements that had at least one citation. Citation diversity measured the share of papers cited given the set returned by the retrieval pipeline.

The strongest gains came from filtering out synthetic reports with low citation density. More aggressive filtering, filter combinations, and learning-rate sweeps did not add meaningful gains. The lesson was clear: a simple signal — whether the synthetic reports consistently cited their claims — was more useful than several more complicated combinations. Scientific specialization, in other words, is not necessarily a matter of adding more scientific text to pretraining. The composition and quality of post-training data and whether it demonstrates behaviors like grounding and attribution can materially change how the resulting model performs.

Speed that changes the workflow

Fast mode averages 51.1 seconds per report across the full Asta pipeline. Thinking mode averages 178.5 seconds. That is about 3.5 times faster.

The speed comes from a redesigned pipeline: AstaBrief writes the full report in one pass given a user query and relevant retrieved snippets. It bypasses the expensive snippet summarization and clustering stages that the Claude-based Thinking mode uses. It also does not write the answer section by section. The team found this was possible without sacrificing performance.

Among 374 Asta users who tried Fast mode, 29.1 percent used it for two or more days. Users on average generated 3.67 report threads with it. Twenty-three percent of users who tried Fast mode continued using it and never switched back to Thinking mode for future threads. An additional 18 percent switched between Fast and Thinking modes depending on their goals, using Fast mode for roughly 40 percent of their threads. Fast mode receives positive feedback at 84.2 percent versus 85.2 percent for Thinking mode.

Evaluations that separate relevance from overstatement

The main evaluation target was SQABench-CS2, a set of 200 user-written computer science research questions. Four metrics were tracked throughout development: Rubric score, Answer precision, Citation precision, and Citation recall.

A report can sound polished and complete while meandering from the question or attaching citations to claims the underlying evidence does not support. For scientific synthesis those behaviors must be measured separately. But citation support is only part of scientific faithfulness. A model can cite the right study and still make a stronger claim than the study itself supports. This happens in subtle ways: turning a finding about a particular sample into a generic claim about an entire population, shifting a result reported in the past tense into a present-tense statement that sounds more universally true, or turning a descriptive finding into a recommendation for what clinicians, policymakers, or researchers should do. Those generalizations are especially important for scientific report generation because each step can broaden the apparent scope of the evidence without introducing an obviously false statement. The development metrics focused primarily on relevance, coverage, and citation grounding. A richer evaluation of scientific report writers should also test whether they preserve the scope and strength of the claims in their sources.

Human preferences and the limits of current benchmarks

In a separate 14-question human study, three scientific researchers each contributed four to five questions and ranked reports from three systems on overall preference, completeness, relevance, organization, and citation accuracy. On overall preference DR-Tulu wins. Two of the three researchers prefer AstaBrief over other systems on citation accuracy metrics.

The LLM-judged comparison on SQABench-CS2 shows AstaBrief competitive with the Claude-powered pipeline and DR Tulu across several measures of answer and citation quality. These numbers are best read as validation of the engineering approach at the time of development rather than as a claim about where this particular base model sits relative to today's frontier. The model ecosystem moves quickly. The data construction, attribution filtering, and serving lessons are the pieces the team expects to generalize.

Open weights for local deployment

Because the model is open weights, institutions can deploy it on their own hardware, including behind their own firewall, without relying on a proprietary model API for report generation. In Asta that also means the team can study and improve this part of the report generation pipeline directly while preserving Thinking mode as an option for more compute-intensive tasks.

The model is available as allenai/AstaBrief_8B on Hugging Face. An example workflow that researchers can adapt to create reports from their own PDFs is also released, providing a starting point for local report generation. The team is exploring more fine-grained preference learning, stronger RAG-plus-RL approaches, multi-turn and multi-tool capabilities, additional scientific data sources, and query decomposition. They are also interested in evaluations that go beyond whether a claim has a supporting citation to ask whether a model preserves the evidentiary scope of its sources. That includes qualities such as concision and organization, as well as whether the model turns sample-specific findings into broad generalizations or descriptive results into recommendations. Most of the training and evaluation described was completed in 2025. The proprietary models used to generate training data and as comparison points reflect the frontier at that time. The full evaluation has not been rerun against today's frontier models.