AI Agents Struggle with Complex Mobile Tasks: New GMA Benchmark

TL;DR: The new GMA benchmark reveals that current AI agents fail at complex mobile workflows due to poor context retention. Success rates drop sharply as task difficulty increases, proving existing models lack the robustness needed for reliable autonomy.

Key facts

• GMA (General Mobile Assistant) benchmark launched August 21, 2026, evaluating eight frontier models across seven open-source apps and 300 tasks spanning four difficulty tiers.
• MobileWorld benchmark features 201 tasks across 20 applications with an average of 27.8 completion steps, nearly double AndroidWorld’s 14.3 steps.
• 62.2% of MobileWorld tasks are multi-app workflows compared to just 9.5% in AndroidWorld, highlighting a shift toward complex cross-application testing.
• MVISU-Bench (October 2024) showed that adding an Aider plug-and-play module improved agent success rates by 19.55%, indicating significant gaps in standard designs.
• OmniaBench (May 2024) reported Overall Pass@1 scores of only 58.54% for Claude-Sonnet-5 and 57.14% for GPT-5.6-Sol, revealing persistent planning limitations.
• GAIA benchmark results from November 2023 showed a massive performance gap, with humans scoring 92% while GPT-4 with plugins achieved only 15%.

The Gap Between Simulation and Reality

New research highlights significant limitations in current AI agents’ ability to handle complex, real-world mobile tasks. A benchmark called GMA (General Mobile Assistant), introduced by researchers on August 21, 2026, evaluates general mobile assistants across seven open-source applications and 300 tasks spanning four difficulty tiers [1][2]. The study found that performance declines substantially as task complexity increases, with current agents struggling to reliably meet realistic user requirements.

The GMA benchmark was developed to address critical gaps in existing evaluation frameworks like AndroidWorld and MobileWorld. While these earlier benchmarks established strong foundations for mobile agent testing, their application coverage and task design do not fully capture the diversity and complexity of real-world usage [1].

Limitations of Previous Benchmarks

AndroidWorld has been a staple in the field, offering reproducible environments for testing autonomous agents. However, recent studies suggest it is now saturated, with many agents achieving success rates exceeding 90% on its specific tasks [7]. This high performance masks underlying weaknesses because AndroidWorld lacks key application categories found in daily life, such as e-commerce platforms and enterprise communication tools [7].

To bridge this gap, the MobileWorld benchmark was introduced. It features 201 tasks across 20 applications, emphasizing long-horizon, cross-application workflows that better reflect how users interact with their devices [7]. MobileWorld is significantly more demanding than its predecessor: it features nearly twice as many completion steps on average (27.8 vs. 14.3) and a much higher proportion of multi-app tasks (62.2% vs. 9.5%) compared to AndroidWorld [7].

MobileWorld also utilizes open-source alternatives like Mattermost for Slack to enable a fully observable and controlled environment, ensuring that researchers can accurately track agent actions without relying on proprietary APIs [7]. Despite these improvements, even MobileWorld does not fully capture the nuance of unpredictable user instructions or the need for robust context retention over extended sessions.

Introducing GMA: A Harder Test

The new GMA benchmark expands on these efforts by evaluating eight frontier models across a wider range of scenarios [1][2]. It focuses on four difficulty tiers, ranging from simple single-step commands to complex, multi-stage workflows that require the agent to maintain context and adapt to changes.

Controlled ablation studies within GMA demonstrated that harness design choices—such as context retention and explicit state tracking—can meaningfully improve performance on demanding workflows [1][2]. However, the effectiveness of these improvements varies significantly across different foundation models, suggesting that no single architecture currently dominates in all real-world scenarios.

The study found that as task complexity increases, agent success rates drop sharply. This decline underscores a fundamental challenge: while LLMs are powerful at generating text or code, they often struggle to maintain an accurate mental model of the device state across multiple applications and interactions [1].

Broader Context from Other Benchmarks

The struggles observed in GMA align with findings from other recent benchmarks, illustrating a consistent pattern of difficulty for AI agents.

MVISU-Bench, released in October 2024, focuses on multi-app, vague, interactive, single-app, and unethical instructions across 137 mobile applications [6]. It identified a substantial gap between existing agent capabilities and user expectations. In this benchmark, a plug-and-play module called Aider improved success rates by 19.55% overall, highlighting how much room for improvement remains in standard agent designs [6].

MobilePA-Bench, published in January 2026, targets tool-calling and planning abilities in an interactive sandbox [4]. It revealed that even frontier LLMs remain unreliable under strict tool ordering, permission limits, and runtime errors [4]. This suggests that reliability issues are not just about understanding natural language, but also about handling the technical constraints of software interfaces.

Beyond mobile-specific interfaces, broader agent benchmarks underscore similar difficulties. OmniaBench, released in May 2024, evaluates general agents across diverse scenarios with explicit state spaces [3]. It found that even top models like Claude-Sonnet-5 and GPT-5.6-Sol achieved Overall Pass@1 scores of only 58.54% and 57.14%, respectively [3]. These results highlight persistent limitations in planning and adaptive correction, which are essential for autonomous operation.

Similarly, GAIA, introduced in November 2023, posed real-world questions requiring reasoning and tool use [5]. In that benchmark, human respondents scored 92% compared to just 15% for GPT-4 with plugins [5]. This massive gap between human performance and AI capabilities remains a central challenge in the field.

What It Means for Mobile AI

These collective findings suggest that while AI agents are advancing, they remain far from the robustness required for reliable autonomous operation in complex, heterogeneous environments. The introduction of GMA provides a more rigorous test bed, forcing developers to confront the reality that simple command-and-response models are insufficient for true autonomy.

As benchmarks like GMA and MobileWorld push the boundaries of what is expected of AI assistants, the industry must focus on improving context retention, error handling, and cross-application coordination. Until these core capabilities are strengthened, AI agents will likely continue to struggle with the very tasks they are designed to simplify.

Sources

1. Benchmarking General Mobile Assistants in Challenging Real-World Scenarios (arxiv.org) — 2026-08-21
2. Benchmarking General Mobile Assistants in Challenging Real-World Scenarios (papers.cool) — 2026-08-21
3. MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments (arxiv.org) — 2024-12-01
4. MVISU-Bench: Benchmarking Mobile Agents for Real-World Tasks by Multi-App, Vague, Interactive, Single-App and Unethical Instructions (arxiv.org) — 2024-10-22
5. MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks (arxiv.org) — 2026-01-31
6. OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios (arxiv.org)
7. GAIA: a benchmark for General AI Assistants (arxiv.org) — 2023-11-21