GTAGym turns GTA V missions into measurable tests for AI agents

From screenshots to successful actions

An AI agent that says “turn left” is not necessarily able to steer around a corner. Testing interactive agents requires tasks where success can be checked reliably across attempts. That can be difficult in open-world games, where objectives may involve several stages, moving characters or vehicles, and conditions that are hard to judge from a screenshot alone.

In an arXiv preprint, Jiajun Chen and colleagues introduce GTAGym, a framework for evaluating agents and generating reinforcement-learning training environments based on Grand Theft Auto V (GTA V). The researchers combine tasks derived from the game’s story missions with automated checks of whether an agent meets each task’s conditions.

The benchmark, GTAVBench, contains 150 tasks. Its online 3D track has 103 tasks across nine categories, including walking, driving, flying, following, interception and evasion. Another 47 image-based tasks use GTA V imagery to test shooting and sniping. The team manually decomposed 75 story missions into stages, specifying goals, interaction order and completion rules.

These are reconstructed tasks, not the original missions running unmodified. The 3D environments use Babylon.js, with adapted layouts and task-specific behaviors. Agents interact through screenshots and keyboard-and-mouse actions. For online tasks, the system checks internal environment state—for example, whether an agent reaches a destination in the required movement mode or maintains proximity to a moving target. In image-based aiming tasks, it checks hits against annotated target regions.

What did the agents manage?

The researchers evaluated six language-model-based agents. In GTAVBench-AGI, the goal appears only in the first screenshot, as an in-game objective, and a software harness automatically compresses the interaction history. The best reported configuration reached 70.9% overall accuracy. For comparison, three experienced players each attempted the benchmark once; their mean accuracy was 89.8%.

Performance varied by task. Some models did well on certain on-foot, indoor and vertical-navigation tasks, while the stronger models also performed well on shooting and sniping. Aerial navigation, evasion and interception remained difficult. These tasks require agents to adjust as speed, obstacles or a target’s position change, rather than simply recognize a goal. The authors’ review of gameplay traces identified problems such as mistimed or oversized inputs, difficulty sustaining a maneuver and failure to change strategy after getting stuck.

The team also used low-rank adaptation and reinforcement learning to train Qwen3.8-27B on generated on-foot navigation tasks. On held-out on-foot tasks, its mean accuracy increased from 54.2% to 72.9%; overall benchmark accuracy increased from 34.2% to 38.9%. In a separate evaluation on 100 tasks in native GTA V, recorded success rose from 15% to 23%. These results apply to this model, training setup and task selection; they do not establish that reinforcement learning will produce similar gains in other settings.

What the results do—and do not—mean

GTAGym’s automatic scoring can make task outcomes easier to verify consistently. Its generator samples parameters within fixed templates—for example, route length or the speed of moving actors—and supplies success and failure signals for training. This allows researchers to create batches of related environments instead of relying only on a fixed set of examples.

Benchmark accuracy is not a measure of general game-playing ability. GTAVBench covers selected reconstructed tasks, and its rules define precisely what counts as success. The results therefore describe performance on those tasks under the reported evaluation conditions; they do not establish reliable performance across the full GTA V game or in real-world settings.

The timing protocol also matters. By default, the online simulation pauses while the model decides what to do. In an experiment where it continued running during inference, one model’s accuracy fell by 22.3 percentage points, with particularly severe effects on tasks involving moving targets and evasion. A paused environment can therefore make dynamic tasks less demanding than a setting where the world keeps changing while the agent thinks.

These findings come from an arXiv preprint, which may not have undergone peer review. The study includes several evaluation protocols, a human comparison involving three players, and one reinforcement-learning training study. Results should be interpreted within those limits. GTAGym is best understood as a proposed testing and training framework: its experiments show how the evaluated agents performed in this particular setup, and where controlling a changing virtual world remained challenging.