What does it take for an AI agent to handle a tax engagement?

Fifty engagements, two made public

The benchmark targets long-horizon professional agents. Tax work was chosen because it demands judgment, exploration of large file sets, and reconciliation of conflicting records across years, entities, and jurisdictions. The full suite spans individual and family returns, small businesses, partnerships, property and investments, estates and trusts, international tax, corporate transactions, and tax review and disputes. Agents must investigate client records, reconcile conflicting information, identify consequential issues, and produce returns, workpapers, technical reviews, and client advice.

Only two tasks are downloadable today.

The remaining forty-eight stay held out. Access requires an email to benchmarks@surgehq.ai.

What the agent actually sees

Each task arrives in Harbor format. That means an instruction file, a configuration file, a containerized Linux environment, and automated tests. The environment comes equipped for document and spreadsheet processing. Network access is restricted to explicitly allowed model API hosts. Browsing and package downloads are blocked.

Supporting materials go far beyond tax returns. Operating agreements appear. Stock ledgers appear. Debt guarantees, dispatch records, vessel-use logs, and client correspondence all appear. A plausible answer can still miss the fact that determines the outcome. The benchmark cites errors in prior returns, unsupported assumptions in staff drafts, and obligations revealed only in operational logs.

Why the pass bar sits at ninety-five percent

Every task carries a detailed rubric. The median rubric holds thirty and a half criteria. An attempt passes only when it satisfies at least ninety-five percent of those criteria. Published results report pass@1 under that threshold. The evaluation protocol runs five attempts per task.

Results on the two public samples alone do not constitute a score on the full benchmark.

The workspace is large. The full set contains one thousand seven hundred eighty-nine total files. Average input files per task reach thirty-five point eight. The median sits at thirty. The range stretches from twelve to one hundred twenty-three. File types break down into one thousand four hundred eighty-seven PDFs, one hundred seventeen Excel workbooks, thirty-eight Word documents, plus emails, CSVs, JSON records, images, and text documents.

How the tasks were built

Tax professionals created the assignments from workflows and decisions encountered in practice. They developed the supporting records, expected outcomes, and grading criteria. Synthetic documents model the forms and records practitioners use. Each task underwent three layers of expert review for correctness, completeness, and solvability.

Contributors bring experience in corporate and private-equity taxation, partnership allocations, real estate, individual returns, trusts, and multistate compliance.

Running the evaluation yourself

Install the DAYJOB GitHub harness. Configure the model endpoint and grading credentials. From the harness directory run a single sample with the harbor command line tool. Replace the task path and model placeholders. Set the required API credentials as documented. For comparisons with published results, use the full benchmark and the documented evaluation configuration including five attempts per task.

Five attempts per task are required for published comparisons.

The dataset card carries an MIT license. The repository weighs two point three three megabytes. It holds two rows in a single test split. The canary string SURGE_DAYJOB_TAX_CANARY_PMQjWdGiSSKJSu9OmoebC2a0IFQayJpA marks the data. Intended use is held-out evaluation. Keep tasks, supporting files, rubrics, and derived materials out of training, fine-tuning, reinforcement learning, and distillation datasets. Do not use them for training-time checkpoint selection.