Can a local language model agent actually build working data pipelines?

What the benchmark actually measures

Fifteen tasks cover the full mobility-data lifecycle: discovery, connectors, transport-feed processing, semantic enrichment, feature engineering, validation, visualization, and reporting. Each task requires the agent to emit multiple artifact types — scripts, structured files, tables, figures, and narrative reports. Deterministic checkers then verify every artifact against ground-truth expectations. No human judgment enters the scoring loop.

Ten local model configurations run in two modes. One-shot generation produces a single answer per task. Closed-loop mode lets the agent inspect intermediate artifacts, detect errors, and repair them before final submission. Five repetitions per model, mode, and task yield 1,500 scored attempts. All runs execute on one consumer-grade GPU.

The workspace condition changes everything

For every model above two billion parameters, the closed-loop workspace lifts pass rates by 26.7 to 52.0 percentage points over one-shot generation. The mechanism is straightforward: when an agent can see a failed test or a malformed table, it revises the code and retries. Gains concentrate in tasks where intermediate artifacts expose concrete errors — a missing column, a schema mismatch, a rendering failure. Tasks without such checkpoints show smaller improvements.

The strongest configuration reaches 85.3 percent artifact-level success. A quantized nine-billion-parameter model hits 69.3 percent while occupying roughly 6.5 GB of VRAM. That footprint fits on many mid-range cards sold today. Smaller models below two billion parameters do not benefit measurably from the workspace; their base capability appears too low to exploit the feedback loop.

Model scale, quantization, and the memory ceiling

No single best model emerges from the data. The work maps a trade-off surface: larger models succeed more often but demand more memory; quantization shrinks the footprint with modest accuracy loss; architecture choices shift the curve. The nine-billion-parameter quantized variant sits at a practical sweet spot for engineers who cannot afford high-end cards. Whether a seven-billion or a fourteen-billion model works better for a specific workflow depends on the task mix — the benchmark lets teams answer that question before committing compute budget.

Runtime is reported but not emphasized as a primary axis. The paper notes that closed-loop runs take longer because of repair cycles. For batch-oriented engineering work, the extra minutes rarely matter. For interactive use, they might.

What this means for a laptop or workstation owner

If you run an open-weight model locally to build data pipelines, the benchmark gives you a reproducible way to test the whole stack — model, quantization, tool wrappers, and validation harness — before you trust it with production data. The fifteen mobility tasks are domain-specific, but the method generalizes: define your tasks, write deterministic checkers, run the same five-repetition protocol. The paper does not claim the results transfer to other domains without re-evaluation.

The dataset and evaluation code are released alongside the preprint. No proprietary API keys are required. The entire experiment runs offline.

Where the explanation stops

Hallucinated library calls that happen to pass the checkers by coincidence are not measured. The work does not compare local agents against cloud-hosted counterparts on the same tasks. It does not quantify the engineering effort needed to write deterministic checkers for a new domain. Those questions remain open for follow-up work.