When a Learning Agent Becomes the Wrong Kind of Firm

A reinforcement-learning agent improves its decisions by interacting with an environment and pursuing rewards. But successful optimization does not establish that the environment represents the intended economic problem. In a study by Ruxin Chen and Zeqiang Zhang, a simulated firm learns a profitable hiring strategy that departs sharply from the competitive equilibrium the researchers want to reproduce.

The supplied manuscript, available on arXiv, states that this version has been accepted after peer review for publication in Computational Economics. It is not the final Version of Record. Its central contribution is a proposed correction to the translation between an economic model and a learning environment—not evidence about how actual businesses use artificial intelligence.

A labor market with costly matching

The researchers use a search-and-matching model in which firms post vacancies, workers seek jobs, and existing employment relationships can dissolve. Production increases with employment but has diminishing marginal returns: additional workers contribute progressively less extra output.

A key variable is market tightness, the ratio of vacancies to unemployed workers. It affects vacancy-filling probabilities and wage determination. In the intended competitive setting, an individual firm treats this aggregate condition as given rather than something it can change through its own hiring decision.

Using parameters calibrated to Japanese labor-market evidence, the authors calculate a theoretical steady-state tightness of approximately 0.78 and unemployment of approximately 0.033, or 3.3%. These values provide a benchmark against which to compare the learning agent. The tightness value is itself a calibration target, so matching it tests reproduction of the specified model rather than an independent prediction of labor-market conditions.

Learning market power instead

In the naive simulation, the agent chooses vacancy postings and receives production revenue minus wages and vacancy costs. Crucially, the environment computes market tightness directly from the agent’s vacancies and unemployment. The agent therefore controls a variable that the competitive model assumes an individual firm takes as given.

The reported result is market tightness around 0.1, substantially below the theoretical benchmark, alongside rewards above the benchmark firm’s flow profit. The authors interpret this as monopsonistic behavior: the agent restricts hiring to influence wages in its favor.

On this reading, the agent has not simply failed to learn. It has learned to exploit the control granted by its environment. The authors call this a structural bias. The finding concerns their naive single-agent formulation; it does not establish that reinforcement learning necessarily produces market power in every economic simulation.

Correcting both the environment and the reward

Their first correction holds aggregate market tightness fixed while the agent learns its hiring policy. Once that policy is obtained, simulated population behavior supplies an updated aggregate value. Repeating these steps seeks a fixed point where individual decisions and market conditions agree. This mean-field construction separates an individual firm’s optimization from the determination of the aggregate market.

The second correction concerns vacancy costs. The authors argue that their naive reward formulation fails to reproduce the economic benchmark’s hiring-cost trade-off, which incorporates both the interest rate and job separation. They propose replacing the original vacancy cost c with an effective cost, c_eff = (1 + r/λ)c, where r is the interest rate and λ is the separation rate.

This is the paper’s proposed calibration for aligning the simulation with its benchmark. It should not be read as a universal rule that reinforcement-learning discount factors cannot represent economic time preferences.

What the experiments establish

The experiments use Deep Deterministic Policy Gradient, an actor–critic reinforcement-learning algorithm. With both corrections applied, the authors report market tightness fluctuating near the theoretical value of 0.78. Their figures show variability across five independent runs.

Experiments that remove one correction help distinguish their roles. With only the mean-field structure, the authors report excessive hiring and tightness above the benchmark. With only the cost adjustment, tightness remains below equilibrium because the agent still influences aggregate conditions.

The convergence argument is conditional. It requires sufficiently well-behaved policy and aggregate-response mappings whose combined sensitivity is less than one. The authors acknowledge that they do not rigorously verify the necessary global bounds for the neural-network implementation and describe approximate convergence in practice.

The demonstrated scope is a homogeneous-firm model with a unique benchmark equilibrium. Heterogeneous agents and broader multiple-equilibrium applications remain future work. Within that scope, the study illustrates a practical distinction: training an agent to maximize rewards is not the same as ensuring that it solves the economic problem intended.