What does Android Bench 2.0 change for developers using AI coding tools?

From bug fixes to week-long challenges

The first version of Android Bench measured how models handle incremental changes. Bug fixes. Small feature requests. That scope matched what AI could do at the time. It also matched how developers were actually using assistants. Android Bench 2.0 raises the bar. Long-horizon tasks now include upgrading dependencies across a codebase and adding new features from scratch. Building entire apps is another category. Converting a cross-platform project to Android is the fourth. These are tasks that take an engineer multiple days or even a week. The benchmark now mirrors the work developers delegate to AI today.

Continuous scoring replaces binary grades

Binary pass or fail grading hides important detail. An agent might refactor 40 screens to Jetpack Compose, set up database tables, and satisfy 90 percent of requirements. A single failing edge-case assertion would still yield a zero percent score. The new methodology calculates a completion rate. It combines functionality and visual fidelity. Regression avoidance is the third factor. Objective penalties apply for deviations from evaluation instructions or structural constraints. The leaderboard shows pass rate, completion rate, and average cost per model and per task. The highest pass rate for long-horizon tasks sits around 28 percent. The original tasks still show roughly 91 percent.

Where models succeed

Models write new code better than they refactor existing code. Refactors and migrations depend on architectural complexity rather than raw line count. Deterministic transformations work well. Converting Java to Kotlin and swapping Retrofit for Ktor are two examples. Introducing a ViewModel layer is a third. Models apply these patterns consistently across 125-plus files and 8,000-plus lines.

Where models stall

Runtime validation breaks them. Missing dependency injection graphs cause failures. Breaking framework changes cause failures. Knowledge gaps with unreleased libraries cause failures. Porting cross-platform apps remains unsolved. No model reaches a 100 percent pass rate. Frontier models top out at an 80 percent completion rate.

Agent evaluations enter the picture

Benchmarking models alone misses how developers actually work. Android Bench 2.0 adds agentic evaluation. Each model runs with an agent from its own provider. GPT 5.6 Sol runs on Codex. Gemini 3.8 Flash runs on Google Antigravity. The pairing shows how harness design improves outcomes. Prompt caching and compact tool windowing reduce token usage. Future expansions will test cross-combinations of models and agents. The goal is to help teams discover which pairings work best for their workflow.

New models on the leaderboard

The leaderboard now includes Gemini 3.8 Flash and Gemini 3.7 Flash. OpenAI's GPT-6 and Anthropic's Fable 5.1 are also present. Kimi K3 and Qwen 3.8 Max round out the additions. OpenAI's GPT-6 Astra leads with a 28 percent pass rate on long-horizon tasks. Feedback shapes future updates. Developers can share input on GitHub, X, and LinkedIn.