AceAction maps the steps between game instructions and player actions

Following a game instruction involves more than choosing the next button to press. An agent might need to approach a building, inspect its entrance, engage an opponent and then retreat to recover. Performing each action competently is not enough if the agent keeps fighting when the objective has changed.

AceAction-5M, described in a preprint submitted on November 3, 2026, is a dataset intended to make those changes of purpose explicit. Researchers from Tencent, East China Normal University and Jilin University connect human gameplay recordings and keyboard-and-mouse actions with three levels of language annotation: overall task instructions, intermediate subgoals and local action descriptions.

The contribution is a data collection and annotation method. The paper does not report experiments demonstrating that training on AceAction improves an agent’s task completion or ability to generalize across games.

Organizing play around objectives

The researchers argue that dividing gameplay into fixed-duration segments does not necessarily reveal its task structure. A sustained objective can cross a time boundary, while several short objectives can occur within one interval.

Their alternative groups behavior according to continuity of purpose. A segment continues while the player pursues the same objective; a new segment begins when the primary objective or interaction target changes. The paper defines 43 skill categories spanning navigation, reconnaissance, combat, interaction and recovery.

A vision-language model processes observations and recorded actions to assign skill categories, descriptions and segment boundaries. Every four consecutive actions form an annotation unit, and processing windows contain up to 32 units. To preserve an objective across windows, the next window receives information about the preceding window’s final observation and subgoal. Segments are merged only when a predicted continuation flag, matching skill categories and adjacent sequence indices support the connection.

These labels are generated interpretations of recorded behavior, rather than direct access to a player’s intentions. Their usefulness therefore depends on whether the descriptions remain grounded in the visible scene and controls.

Giving navigation a recognizable destination

Navigation receives an additional waypoint-based treatment. The model examines a sliding window of recent observations to identify arrival at a visually recognizable location. It assigns that location a descriptive name and associates it with the arrival frame, reusing an accepted name when it recognizes the same location again.

The pipeline then samples clips of varying lengths that end at that waypoint. This produces navigation examples with a shared destination but different starting points and intervening actions. The model describes the observed route using sampled keyframes, the complete action sequence, skill context and waypoint name. The resulting example retains the complete image–action sequence.

Adjacent subgoals from the same trajectory are subsequently combined, in execution order, into an overall instruction. Their individual descriptions and temporal boundaries remain attached. The intended result is a record of both what a task requires and which observations and actions belong to each stage.

Broad coverage, with inconsistent headline counts

The dataset combines directly recorded gameplay with publicly sourced material. Direct recordings include Game for Peace and Valorant; other sources include Counter-Strike recordings, Open-P2P and Minecraft material from VPT. The authors say the directly collected gameplay came from consenting in-house staff and was transcribed in anonymized form.

The preprint reports about 5.48 million episodes in its contribution statement. However, its coverage totals are inconsistent: the overview lists 41 games and approximately 5,180 hours, while the contribution statement gives 42 games and 5,027.87 hours. The paper does not reconcile those figures, so a single precise coverage total would overstate its clarity.

Actions share a representation covering keyboard states, mouse-button states and horizontal and vertical mouse displacement. Sampling rates are not uniform: directly collected and Counter-Strike actions use 16 Hz, while Open-P2P and VPT retain 20 Hz.

What the quality checks establish

Two models review each sample from local action descriptions through subgoals to the overall instruction. Both must approve every annotation level; rejection by either excludes the sample.

Eight human reviewers then inspect 500 trajectories, with each assessed by at least two reviewers. The reported 95.8% pass rate is calculated across individual reviewer assessments, not the percentage of trajectories unanimously approved.

That audit provides evidence about consistency between annotations, observations and actions in the inspected sample. It is not a gameplay success rate or proof of better learned policies. AceAction’s central proposal is structured supervision for studying skill composition and transitions; its practical benefits for trained agents remain untested in the reported work.