Code Agents and the Reality of Engineering: The Reality Check of Android Bench 2.0

Over the past two years, announcements about artificial intelligence applied to software development have been measured by one fetishized metric: the pass rate on synthetic tests. We have seen models break records on HumanEval, solve LeetCode Hard problems in seconds, and fix isolated issues on SWE-bench. The narrative was straightforward: if AI can solve complex algorithms, replacing the work of a software engineer is simply a matter of time.

However, the release of Android Bench 2.0: Pushing the Frontier with Challenging Long-Horizon Tasks by Google’s Android team has poured cold water on that perception.

When autonomous agents were confronted with everyday engineering work—migrating entire projects, dealing with broken dependency injection, upgrading SDKs, and converting cross-platform applications—the success rate plummeted: only a 28% pass rate for the most advanced frontier models.

1. The End of Binary “Pass/Fail” and the Rise of LHTs

Classic benchmarks operated under an extreme simplification: either a code patch passes all unit tests (pass), or it is considered a complete failure (fail). That metric works when the goal is to fix a typo in a mathematical function, but it is inadequate for measuring real-world engineering projects.

What Are Long-Horizon Tasks (LHTs)?

In Android Bench 2.0, Google introduced LHTs (Long-Horizon Tasks): architectural challenges that take a human developer anywhere from several days to an entire week to complete. Some of the evaluated examples include:

Structural refactoring: Migrating 40 traditional XML-based screens to Jetpack Compose.
Deep dependency upgrades: Replacing entire networking layers (for example, switching from Retrofit to Ktor) and reconfiguring serialization.
Cross-platform migrations: Converting cross-platform mobile applications to a native Kotlin architecture.
Persistence integration: Designing relational database schemas (Room/SQL) while maintaining consistency across domain models.

The Need for “Continuous Scoring”

Under the previous binary approach, if an agent refactored 40 screens, created the repositories, and met 95% of the requirements, but failed a single edge-case assertion in the visual interface, the result would be recorded as 0% success.

Android Bench 2.0 replaces this with a continuous score composed of:

Verified functionality: Actual execution on devices and emulators.
Visual fidelity: Comparison of layouts and design consistency.
Absence of regressions: Ensuring that modified screens do not break unrelated modules.
Deviation penalties: Automatic penalties if the agent ignores structural constraints or directives from the initial prompt.

2. Anatomy of the 28%: Why Does AI Fail on Real-World Projects?

The jump in complexity exposed a gap that synthetic benchmarks systematically concealed:


┌──────────────────────────────────────┐       ┌──────────────────────────────────────┐
│       Isolated Tests (LeetCode)      │       │       Real Engineering (LHTs)        │
├──────────────────────────────────────┤       ├──────────────────────────────────────┤
│ • Self-contained state in memory     │       │ • Runtime side effects               │
│ • Predictable input and output       │  VS   │ • Broken dependency graphs (DI)      │
│ • Context < 2,000 tokens             │       │ • Millions of accumulated tokens     │
│ • No interactions between modules    │       │ • Friction between undeclared SDKs   │
└──────────────────────────────────────┘       └──────────────────────────────────────┘

A. The Runtime Validation Trap

Writing syntactically valid code is straightforward for an LLM. What becomes critical in Android Bench 2.0 is the compilation and dependency-injection cycle (for example, with Dagger/Hilt or Koin). An agent can generate flawless service classes, but if it forgets to register a module in the dependency graph or introduces a circular dependency, the application may compile but crash at startup with an IllegalStateException.

B. Synthesis vs. Maintenance

The comparison results confirmed a fundamental principle: models are much better at writing code from scratch than refactoring existing code.

When creating new code, the model imposes its own assumptions and patterns.
When refactoring, the agent must infer design decisions made years earlier by other humans, respect implicit hierarchies, and avoid mutating shared state.

C. Memory Loss in Long Tool Chains (Harness Drift)

To complete an 8,000-line migration across 125 files, an agent must execute dozens of terminal commands, read Gradle error traces, inspect files, and apply successive patches. As the conversation exceeds hundreds of thousands of tokens, the agent begins to forget initial instructions, reintroduce previously fixed errors, or enter trial-and-error loops that drive up inference costs without making progress on the task.

3. Video Analysis: Comparing Code Agents in Real-World Environments

To understand how execution environments and tool harnesses (agent harnesses) behave under stress in complex coding projects, the following practical analysis breaks down the differences between some of the industry’s most widely used agents:

Comparative analysis of autonomous development agents (Codex, Claude Code, GLM) in practical programming scenarios.

4. Demonstrated Strengths: Where Does AI Actually Shine?

Despite the 28% overall pass rate demonstrating that full autonomy is still a long way off, Android Bench 2.0 did not deliver only bad news. It revealed areas where agents already demonstrate remarkable accuracy:

Large-scale deterministic transformations: Agents consistently handled the conversion of legacy Java classes to Kotlin, as well as the systematic replacement of HTTP calls between well-known libraries.
Disciplined repetition: When the migration rule is clearly defined in the specification, models were able to apply the same architectural pattern consistently across repositories containing more than one hundred files.

The bottleneck is not the ability to type repetitive code, but strategic decision-making in the face of ambiguity.

Conclusion: The Developer as Architect and Context Referee

Android Bench 2.0 marks the beginning of a more mature era for AI tools aimed at developers. The 28% metric challenges the idea that agents can receive a vague product specification and deliver a production-ready migration without supervision.

The real competitive advantage for engineering teams will not be handing over an entire application to a blind agent, but using AI for heavy, deterministic work, while reserving human judgment for architectural consistency, contract design, and runtime validation.

Technical References

Android Developers Blog: Android Bench 2.0: Pushing the Frontier with Challenging Long-Horizon Tasks (September 2026).
Developer Tech News: Google’s Android Bench 2.0 Tests AI Models on Complex Tasks.

Coding Agents and Android Bench 2.0