What Does a Benchmark That Runs Real Apps Tell Us About Mobile AI Agents?

Most benchmarks for mobile agents fall into two camps. One camp watches whether an agent can tap the right pixels on a screen. The other checks whether an agent picks the right function signature from a menu. Both miss what happens when an app returns an error, when a permission dialog blocks the next step, or when the agent must remember a preference from three tasks ago.

MobilePA-Bench replaces both with an executable sandbox. The sandbox keeps live application databases. It returns structured feedback after every tool call. An agent that plans a multi-step trip, books a restaurant, and then adjusts the calendar because the user hates early mornings gets judged on the final state — not on whether each individual API name matched a ground-truth list.

The benchmark covers 13 functional domains. Travel, finance, messaging, and calendar sit alongside health, shopping, and smart-home control. Across those domains sit 212 tools. Each tool behaves like the real thing: it reads from a database, writes to it, enforces permissions, and can fail in ways the agent did not anticipate.

Three advanced dimensions sit on top of basic tool use. First, sub-agent collaboration. A complex request gets decomposed. Specialized sub-agents handle pieces — one searches flights, another checks hotel cancellation policies, a third syncs the result to the calendar. The central planner must hand off cleanly and stitch the results together. Second, memory usage. The agent must recall stored memories, user profiles, and past preferences to resolve implicit requests. If the user once said "never book window seats," the agent applies that without being reminded. Third, skill usage. Pre-packaged composite skills replace step-by-step planning for common patterns. The agent invokes a "book round-trip" skill instead of chaining search, select, pay, and confirm manually.

Experiments with current frontier LLMs show sharp drops when the sandbox enforces strict tool ordering. Permission limits cut success rates further. Unexpected runtime errors — a network timeout, a database lock, an expired token — cause cascading failures. The paper states that performance drops sharply under these conditions. No model tested remains reliable across the full suite.

The sandbox also serves as an interactive foundation for agentic reinforcement learning. Because every step produces a verifiable state change, reward signals can be derived from evidence rather than from a static answer key. That turns the benchmark into a training ground as well as a test set.

The authors are Yi Zhu, Xiongwei Wu, Qiyi Wang, Tingyu Qu, Jiajun Liu, Sihan Cao, Long Chen, Weigao Sun, Feida Zhu, Yiran Zhong, and Steven Hoi. The preprint appeared on arXiv on 24 August 2026.

What the benchmark does not measure is how much battery the agent drains, whether users trust its decisions, or how it behaves when the phone loses signal mid-task. Those questions live outside the sandbox.