Deep Double Q-learning: A Two-Function Approach to Overestimation

Reinforcement-learning systems learn by taking actions and receiving rewards. To decide what to do, an agent estimates the future value of possible actions in a situation. These estimates are called action values, or Q-values. A potential problem is overestimation: when an agent chooses the action with the highest estimated value, errors in those estimates can make the chosen value too optimistic.

Classical Double Q-learning addresses this by maintaining two action-value functions. One is used to select an action, while the other evaluates it when calculating a learning target. This separation is intended to reduce the bias that can arise when the same estimates both select and evaluate an action.

The paper, Deep Double Q-learning, asks how to apply this idea in deep reinforcement learning. The authors argue that Double DQN does not fully decouple its estimators: it directly trains only one action-value function, leaving the estimators correlated and allowing overestimation to persist.

Their proposed method, Deep Double Q-learning (DDQL), explicitly trains two Q-functions using the Double Q-learning objective. The abstract says the method stabilizes training through a combination of lower replay ratios, longer intervals between target-network updates, and shared layers. Replay ratio concerns how often stored experience is used for learning; target networks provide a reference that is updated periodically. Shared layers mean the networks share some components, rather than being wholly separate.

The authors evaluate DDQL across 57 Atari 2600 games. They report improved aggregate performance over Double DQN, say DDQL outperforms it on 47 games, and report further reductions in overestimation. The abstract does not provide scores, uncertainty estimates, or details of the comparison procedure. The 47-game result therefore conveys the reported direction of performance, but does not tell readers how large or statistically reliable the differences were.

The authors also study design choices involving network architecture, replay ratio and minibatch sampling strategies. The abstract does not give the results of those investigations, so it is not possible from the available evidence to say which choices mattered most or how sensitive the headline findings are to them.

These results are specific to the reported Atari evaluation. The abstract does not establish how DDQL would perform in other environments, or how its results vary with random seeds, implementation details or evaluation settings. Nor does a reported reduction in overestimation by itself establish that an agent will make safer or more useful decisions in practical applications.

This account is based on the abstract rather than a full examination of the paper’s methods, results tables or supplementary material. The work is an arXiv preprint; the supplied evidence does not establish whether it has undergone peer review. Its findings should be read as the authors’ reported results, pending assessment of the full paper and further evidence. The central proposal is to make the two-function principle explicit in deep Q-learning and evaluate its effects on performance and overestimation.