Q-learning firms can lock in high prices by never discovering that undercutting pays

A preprint by Cristian Chica, Yinglong Guo and Gilad Lerman provides a theoretical explanation for how Q-learning pricing algorithms can settle permanently on supracompetitive prices. Its central mechanism is incomplete learning: algorithms that stop experimenting may never discover that undercutting their rivals would be profitable.

The paper establishes conditions for this outcome and explicit positive lower bounds on its probability. These are mathematical results within a repeated pricing game, not measurements of commercial algorithms or estimates of how often high-price lock-in happens in actual markets.

Learning from prices actually tried

The main pricing results concern firms competing over an infinite sequence of periods. Each chooses from a finite set of prices, and its pricing decision can depend on the previous period’s complete price profile. The environment is fixed for these results, rather than subject to changing demand or cost states.

The model assumes a common competitive price that is a Nash equilibrium of the one-period game: no firm can improve its profit by changing its price alone when rivals charge that price. It also assumes another common price that gives every firm greater profit, which the authors call the “collusive-enabling” price.

Each firm maintains a Q-table containing estimated values for its possible pricing actions, conditional on the previous price profile. Only the entry corresponding to the action actually taken in the state actually visited is updated. The update combines current profit with an estimate of future returns.

Initially, firms experiment using a softmax rule, which assigns higher probabilities to actions with higher estimated values while still allowing alternatives. At a finite deadline, experimentation stops. Firms then choose actions with the highest estimated values, although they continue updating the entries they visit.

When high prices become absorbing

The first main result identifies two sufficient conditions at the experimentation deadline. The collusive-enabling price must rank strictly above alternatives in each firm’s table at the states relevant to entering and maintaining the high-price outcome. In addition, estimated values for alternative prices at the collusive state must not exceed the discounted value of charging the collusive-enabling price forever.

When these conditions hold, all firms choose that price in every subsequent period. The outcome is absorbing: continued learning along the realized path does not dislodge it.

Crucially, these conditions concern what the algorithms have learned, not whether a deviation would actually pay. Once firms repeatedly choose the same price, estimates for untried alternatives at that state remain unchanged.

The authors also show that sufficiently strong rankings can survive all remaining experimentation. Under specified restrictions, randomly initialized tables therefore produce lock-in with an explicitly bounded positive probability, regardless of subsequent exploratory choices.

High prices from an initially blank table

A separate result starts every Q-value at zero, so no price initially has an advantage. The proof follows one possible exploration path: every firm chooses the collusive-enabling price in every experimentation period.

Along that path, positive profits reinforce the chosen price. Alternative-price entries remain zero because those actions are never tried. By the deadline, the conditions for permanent high-price play hold. No communication, instruction to collude or executed punishment is needed.

The probability calculation counts only this particular path, making it a lower bound rather than an estimate of total lock-in frequency. Lower exploration temperatures, faster learning and larger profits at the chosen price raise this bound; that comparison does not establish equivalent changes in the overall probability of lock-in.

The mechanism is not exclusive to supracompetitive prices. Any common price giving every firm positive profit can become absorbing through the same argument. Competitive lock-in therefore also has positive probability when competitive profits are positive. Although the high-price lower bound is larger under the paper’s stated conditions, this does not prove that high-price lock-in is more likely. When competitive profits are zero, the argument fails because the blank table retains ties.

Persistent prices are not necessarily equilibrium behavior

The paper separately identifies Q-table conditions supporting three strategies: always charging the collusive-enabling price, reverting to competitive pricing after a deviation, or climbing a price ladder after a competitive reset.

These strategy conditions are distinct from the blank-table emergence proof. Always charging the high price is an equilibrium only if that price is itself a one-period Nash equilibrium. Grim-trigger punishment can instead be subgame perfect when firms value future profits sufficiently highly.

The distinction matters for interpreting algorithmic pricing: persistent high prices need not demonstrate learned retaliation. But the analysis does not characterize the long-run distribution across absorbing outcomes or explain nonabsorbing price cycles. It establishes a possible lock-in mechanism, not a general prediction of market behavior.