MemU-Bench tests whether AI phone assistants can remember what users leave unsaid

An AI assistant may know how to tap through a calendar yet still have no idea which meeting its user wants changed. That distinction is the focus of MemU-Bench, a benchmark introduced by researchers from Zhejiang University and Ant Group. Their preprint examines smartphone tasks whose instructions leave out details that appeared earlier in the user’s activity.

The researchers also introduce AppMem, a memory framework that organizes past interactions by application. In their experiments, it substantially improves agents’ ability to complete these ambiguous requests without taking long detours through unrelated apps.

Testing the context behind a command

According to the authors, existing graphical user interface benchmarks generally give agents precise, self-contained instructions. MemU-Bench instead asks them to recover objective facts from history: a phone number previously viewed, an expense amount, or the timing needed to set an alarm. This is different from learning someone’s tastes or preferences.

The benchmark contains 118 tasks grounded in 29 interaction histories covering more than 30 applications. Histories run from roughly 100 to 800 steps and include frequent application switches; one trajectory contains 179 switches.

These are constructed scenarios, not recordings of unrestricted everyday phone use. The team manually designed tasks and high-level activity sketches, then had operators carry them out in an Android emulator. Operators could choose applications and routes while introducing interruptions, accidental touches and redundant switching.

The researchers saved screenshots, actions and timestamps, then preserved application databases and device state. Before an agent attempts a task, the environment is restored to the point where the operator left it.

Why finishing is not enough

An agent can sometimes compensate for missing memory by searching through the phone until it finds a clue. To distinguish that behavior from efficient use of history, the benchmark measures more than task completion.

Success Rate records the share of tasks completed. Interaction Efficiency compares the agent’s action count with a human reference, considering successful tasks only. Success weighted by Path Length, or SPL, combines completion with that step-efficiency comparison: failures contribute zero, while successful but circuitous attempts receive less credit.

These efficiency measures are proxies for useful recall, not direct proof that an agent remembered correctly. The paper itself reports cases where agents succeeded despite retrieving insufficient information because they found additional clues during execution.

Without user history, the strongest reported SPL was 16.4%, achieved by Gemini-3.1-Pro. The three tested end-to-end GUI agents all scored below 5%. Those findings establish the difficulty of these particular ambiguous tasks, rather than a general inability to operate smartphone interfaces.

A memory organized around apps

AppMem processes history in windows of 10 screenshot-action pairs. It creates entries recording the application, the event, specific visible details, timestamps and whether the activity is ongoing or finished.

When a user returns to an unfinished activity, the framework merges the new information into the earlier entry. Completed events remain separate. The aim is to preserve a coherent account of interrupted activity without repeatedly duplicating its details.

When a vague request arrives, AppMem infers which applications’ memories might help, retrieves their entries and extracts relevant information. It then checks whether critical details are still missing, allowing up to three retrieval rounds before execution. The framework requires no additional model training.

For GPT-5.5, the authors report that AppMem raised SPL from 15.7% without history to 66.9%, a gain of 51.2 percentage points. A simpler history-summary baseline reached 57.6%, putting AppMem ahead by 9.3 points. GPT-5.5’s task success rate with AppMem was 73.7%.

AppMem improved SPL across all nine evaluated models. However, its benefits did not eliminate substantial differences between them: the best end-to-end GUI agent with AppMem, MAI-UI-8B, completed 22.0% of tasks.

What remains unresolved

Outcome checking combines database rules with model-based assessment. The authors report that 44.9% of tasks use rules alone, while 55.1% require a language-model judge for open-ended text. A manual check of judge-dependent outcomes for one evaluated model found agreement above 95%.

Most reported evaluations use one complete run per model because processing long histories is costly. The team repeated one model’s evaluation three times and reported small variation, but did not present equivalent repeated-run checks for every model.

Memory is also not the only remaining obstacle. In an AppMem error analysis using Qwen3.5-397B-A17B, 33.1% of tasks failed with insufficient retrieved information, while 14.4% failed despite sufficient information because of navigation limitations. The results support structured memory as a useful component of phone agents—not a complete solution to reliable assistance.