LOLPlay tests whether visual AI agents can keep up with League of Legends

A screenshot captures a moment; a live game does not preserve it. While an AI agent decides where to click, an enemy can move, a fight can end, or the controlled character can die. The challenge is not simply understanding an image, but turning that understanding into useful action before circumstances change.

In the preprint LOLPlay: Benchmarking Real-Time Visual Agents in League of Legends, Zhaojian Yu introduces a benchmark built around that mismatch. The reported results show agents completing some character-development objectives while struggling to combine requirements into sustained success. They also challenge the assumption that increasing reasoning effort necessarily improves real-time control.

A live game, but narrowly defined goals

LOLPlay contains 36 tasks, each assigned to a different champion, the character an agent controls. Budgets range from five to 20 minutes. Objectives span resource collection, ability development, survival, combat, cooperation and map objectives.

These are not tests of winning full competitive matches. Twenty-six tasks contain no bot champions, although minions, monsters, turrets and the game clock remain active. Ten include beginner-difficulty enemy bots; three also include an allied bot. Cooperation means participating in allied-bot kills, not coordinating through communication with another learned agent.

Agents receive screenshots, task instructions and recent interaction history, then issue mouse and keyboard commands. They do not receive the structured game-state data used for scoring. Gameplay continues during model requests and action execution.

Separate automated checks determine success. Across the suite, 82 subgoal checks distinguish partial progress from full completion. A task can require several outcomes together, sometimes with deadlines or a specified event order. Declaring completion does not make an attempt successful.

For example, Warwick must personally kill a Voidgrub and then destroy the enemy top outer turret. Completing only the first objective is insufficient. Two experienced players validated the suite, with all 36 tasks successfully completed under the benchmark conditions. That establishes feasibility, not a population-wide human performance estimate.

Progress is not completion

The study evaluates GPT-6 Astra, GPT-6.1 Sol and DeepSeek-v4.1-Flash at low, medium and high reasoning effort. Each configuration attempts all 36 tasks, producing 324 runs. Episodes have both a task-specific time budget and a ceiling of 100 model responses; one response can contain multiple actions.

Astra achieves the highest task success rate: nine of 36 tasks, or 25%, at both low and medium effort. Medium effort also produces the highest subgoal success rate, 40.3%. That metric averages the fraction of completed requirements within each task, giving every task equal weight rather than pooling all checks.

Across all configurations, 40 runs succeed. Economy and development tasks account for 38 successes. The remaining two concern a combat-category task requiring a level threshold and limited deaths, rather than a kill. No run satisfies a personal champion-kill or allied-assist check, and no sustain, cooperation or map-objective task is fully completed.

Meanwhile, 108 failed runs satisfy some requirements. That distinction matters because preserving health, avoiding deaths or leaving an ability unlearned can earn subgoal credit without demonstrating effective progression toward the remaining objectives.

More reasoning does not guarantee better control

High effort improves neither task success nor subgoal success over low effort for any tested model. Astra’s task success falls from 25% at low effort to 16.7% at high effort. Its average request latency rises from 7.47 to 16.67 seconds, while average responses per episode fall from 43.9 to 27.1.

Those observations do not isolate delay as the cause of failure. Request latency includes service, network, retry and parsing overhead, not just reasoning. Sol’s latency also changes non-monotonically across effort settings. Conversely, DeepSeek’s faster low-effort responses do not translate into better scores than Astra’s. Speed alone is not sufficient.

A Warwick case illustrates the timing problem. A screenshot shows combat with Voidgrubs; a request then takes roughly 145 seconds. During that interval, recorded events show a Voidgrub kill followed by the champion’s death. The next screenshot shows the character at base behind an inactivity warning, yet the returned action batch includes a click chosen from the earlier scene.

The record demonstrates changed context, not which individual inputs the warning intercepted or whether latency alone caused failure.

LOLPlay’s scope remains limited to one objective per champion in custom matches. Its outcome checks do not establish the quality of every action, and some requirements permit passive progress. Within those boundaries, the preprint makes a useful distinction: local achievements, frequent inputs and additional reasoning are not substitutes for completing the requested objective while the world keeps moving.