Why Does IBM's Agent Memory Use One-Seventh the Tokens of ACE?

An agent fails a multi-step task. It mis-paginates an API. It resolves the wrong person. It returns a value nobody asked for. The model knows the APIs. It has not internalized how to use them reliably.

That gap is learnable. ACE and ALTK-Evolve both mine the agent's own trajectories. Both turn failures into reusable lessons. Both feed those lessons back at inference time. No weight updates. No human labels.

What both systems refuse to do

Compression. ACE names the failure modes: brevity bias and context collapse. Its answer is a rich, itemized playbook with a helpful/harmful counter on every bullet. The model distills relevance at read time.

ALTK-Evolve reaches the same conclusion from the other direction. Every distinct guideline keeps a support count — how many independent episodes produced it. The store is never summarized down to a handful of rules. A lesson five different tasks discovered is a different object from one that appeared once. Both are worth keeping.

Count them. Don't collapse them. ACE's per-bullet counters and ALTK-Evolve's support counts are two spellings of the same idea.

Where the token bill diverges

Consolidation differs. ACE grows one playbook through a Generator → Reflector → Curator loop with incremental delta updates and embedding-based de-duplication. ALTK-Evolve clusters near-duplicate lessons and merges within a cluster, support-conserving — the survivor inherits the combined count. It also extracts typed guidelines — strategy, recovery, optimization — with causal attribution and provenance back to the source trajectory at subtask granularity.

Delivery is where the numbers separate. ACE injects the comprehensive playbook on every step, the same way regardless of model or task. ALTK-Evolve treats delivery as a dial. A small fixed core of high-support guidelines. Extended per task with a handful selected for the task at hand — cosine or LLM-guided, priority-weighted. Or the full consolidated set when a model has the headroom to use it.

Same lessons available to both agents. ACE always sends all of them. ALTK-Evolve sends however many a given model can actually use.

What the benchmarks actually show

AppWorld. 168 tasks. ReAct code agent. Memory mined from train/dev only. Single runs, pass@1. ACE numbers are IBM's own runs on the same splits and same base models.

DeepSeek-V3.2. ACE: 80.4 TGC, 73.2 SGC, 634K tokens per task. ALTK-Evolve: 89.3 TGC, 80.4 SGC, 263K tokens per task. Better on both metrics at roughly forty percent of ACE's inference cost.

gpt-oss-120b. ACE: 54.8 TGC, 35.7 SGC, 777K tokens per task. ALTK-Evolve: 56.0 TGC, 37.5 SGC, 116K tokens per task. Accuracy is effectively tied — a repeat run of ALTK-Evolve landed at 54.8, matching ACE almost exactly, within the benchmark's run-to-run noise. Cost is about one-seventh.

The guideline counts tell the story. DeepSeek-V3.2: ACE uses 106 guidelines, ALTK-Evolve uses 191. gpt-oss-120b: ACE uses the full playbook, ALTK-Evolve uses roughly 29 selected guidelines.

Why the weaker model tells a different story

On gpt-oss-120b, ACE leads on Easy and Medium tasks. The comprehensive prompt helps more than it distracts when the task is solved by generic instruction-following. But on Hard tasks, where the model must pick the right lesson rather than wade through all of them, curated retrieval pulls ahead — and that tier decides the aggregate.

On DeepSeek-V3.2 the story flips. The stronger model absorbs ACE's full playbook well enough to edge ALTK-Evolve on Medium. But ALTK-Evolve leads Easy, Hard, and Overall. With more capacity to spare, more lessons delivered the ALTK-Evolve way keep helping instead of crowding each other out.

Each model gets its best configuration. Full consolidated set for the strong model. Selective retrieval for the weaker one. A large context overwhelms a weaker model rather than helping it.

The by-difficulty tables confirm it. gpt-oss-120b baseline 39.9 TGC. ACE reaches 54.8. ALTK-Evolve reaches 56.0. Hard tier: baseline 19.1, ACE 23.8, ALTK-Evolve 31.8. DeepSeek-V3.2 baseline 79.8 TGC overall. ALTK-Evolve reaches 89.3. ACE reaches 80.4. Hard tier: ALTK-Evolve 77.8, ACE 61.9.

ACE's efficiency story is about building its context cheaply. ALTK-Evolve's is on a different axis — serving it. Retrieving a few guidelines per task instead of injecting the whole playbook on every step is where the tokens go. Direct consequence of the delivery difference.

Exactly how much to inject and how it scales across the capability spectrum is the subject of a next post.