How does AutoSynthData turn model failures into enterprise training data?

From failures to a curriculum

AutoSynthData starts by running both a target model and a stronger teacher through diagnostic tasks inside the target environment. The evaluation reveals which capabilities the target model struggles with, what tools and workflow structures are involved, and how the teacher succeeds where the target fails. These observations are distilled into sanitized capability specification cards. Each card describes the capability, the required final state properties, and the dimensions that can vary without changing the core skill. The generator receives only these cards. It does not receive the original prompts, entities, trajectories, or verifier details. It uses the cards to create new tasks with different prompts, initial states, and solution paths. The curriculum shifts as the model improves.

Building the dataset

The pipeline works in two phases. The target phase creates a core set of vetted samples. Workers generate independent tasks in parallel from the capability cards. Each candidate passes through validation, execution in the environment, solver evaluation, and a bounded repair loop before acceptance. The multiply phase then expands the dataset by creating novel variants of accepted target samples. Each variant carries its own user request, environment state, entity configuration, reference trajectory, and verifier. A multiplied sample cannot seed another multiplied sample. This anchors expansion to the vetted core and limits drift across generations.

Quality control at two levels

Individual candidates must clear a quality-control loop. Solver evaluation favors tasks the target model solves on no more than one of three trials while the stronger solver solves on at least two of three trials. Positive verification executes the reference trajectory and checks the resulting state against the candidate's verifier. Negative verification mutates parts of the expected outcome and confirms those mutated states no longer pass. Failed candidates go to a critic. The critic diagnoses inconsistent state or impossible workflows. It also flags incorrect task construction. Bad reference trajectories get flagged too. Weak verifier logic and capability mismatches are caught as well. Repairs are guided by the diagnosis and retried up to a fixed limit. A repaired task must pass the relevant checks again. Batch-level review then examines accepted and rejected samples across each batch. It asks which task families are overrepresented and which capability dimensions are missing. The controller tracks coverage in the accepted dataset, reduces generation in overrepresented regions, and directs more work toward gaps. When a region repeatedly produces poor candidates, critiques guide changes to the generation strategy. Meta-review does the same. These feedback loops improve both individual tasks and the dataset they form.