Steam review prompts brought more praise—and quieter rating labels

A review prompt can bring many more voices into a rating system while making its warning signals quieter. That is the central finding of a preprint by Zihan Chen, Lei Nico Zheng and Di Zhu examining Steam’s introduction of in-client review requests in October 2019.

Their analysis distinguishes a rating’s ability to separate meaningful variation from noise from the movement of its displayed percentage and label. More positive, less responsive reviews can shrink score movements, while their greater number reduces sampling noise. Those effects need not cancel in the same way for the underlying information and the display.

A change around three hours of play

The main dataset contains 21.6 million reviews of 315 popular games, with playtime recorded when each review was written. Steam does not publish the prompt’s precise trigger, and individual reviews are not marked as prompted. The researchers infer an eligibility threshold at three hours from sharp changes in review counts after the new library launched on October 30, 2019.

They compare the discontinuity around that threshold before and after launch—a method called difference in discontinuities. Within 30 minutes on either side, reviews above the threshold outnumbered those below by 4.25 to one after launch, compared with 0.97 before. The jump in the positive-review share increased by 6.6 percentage points.

This comparison concerns changes in the review population, not an experiment assigning otherwise identical players to receive a prompt. Its interpretation depends on assumptions about how reviewing would have evolved around the threshold without the prompt.

Less movement does not necessarily mean less information

To measure responsiveness, the researchers examine how monthly scores in different playtime groups move with the score from reviews written within the first two hours. They call this relationship “pass-through.”

For reviews written at three to four hours, pass-through fell from 0.73 before launch to 0.18 afterward. In the neighboring two-to-three-hour group, it fell less, from 0.81 to 0.64. The estimates measure co-movement, not proof that short-playtime reviews capture changes in product quality.

The authors identify two mechanisms. First, lifting an already high positive share toward 100% compresses percentage-point movements. Second, the added reviews respond less to changes in short-playtime evaluations. Their estimates center on roughly half the response of voluntary reviews when measured in log odds, a scale used to separate behavioral differences from ceiling compression.

But responsiveness is not the same as sensitivity: how well a review stream distinguishes signal from ordinary variation. With equal numbers of reviews, sensitivity deteriorated more above the threshold. At actual review volumes, however, the extra observations offset much of that disadvantage. The threshold-specific change in correlation was −0.03, with a 95% confidence interval from −0.13 to 0.06. That is no detectable relative loss, not proof that no loss occurred.

Labels became harder to move

Fixed rating categories can become less responsive even when the stream retains useful information. Applying Steam’s percentage boundaries to the adjacent playtime groups, the researchers find that the three-to-four-hour tier fell in 42% of downturn months before launch and 12% afterward. Below the threshold, the corresponding rates were 47% and 41%.

These are diagnostic tiers constructed from subsets of reviews, not the labels buyers actually saw. The estimated difference between groups was imprecise: its confidence interval included zero.

A separate reconstruction of Steam’s public 30-day score also showed quieter labels. They fell in 38% of downturn months before launch and 9% afterward, while false alarms fell from 10% to 3%. This public-score comparison has no adjacent control group, so it cannot attribute the entire change to prompting.

Recalibration is therefore a trade-off, not a free improvement. On the reconstructed public score, none of the alternative rules tested outperformed the label at its own low false-alarm rate. Catching more downturns required accepting more false alarms.

Failures still showed up

Crucially, short-playtime evaluations were not a general warning system for product problems. They barely registered failures identified from developers’ own announcements. Scores across all reviews did: relative to quiet windows in the same games, they fell by 1.0 percentage point before launch and 2.9 afterward. The post-launch estimate rests on 25 usable failure episodes.

Agreement with critic rankings changed little near launch at modest, equal review counts, but preservation was not established at larger counts. Agreement weakened later, without a design attributing that decline to prompting.

The study’s limits include inferred prompt exposure, snapshot data containing edited reviews, and detailed playtime coverage ending in January 2021. It does not measure purchases, refunds or consumer welfare. Its practical implication is narrower than “stop asking”: platforms should reassess rating thresholds when solicitation changes the scale. Separating displays of overall satisfaction from signals of change remains a proposal to test, not a validated solution.