Algorithmic Collusion in Q-Learning Pricing Agents May Not Rest on Punishment
ORIGINAL / Algorithmic Collusion Without Deterrence: Auditing the Punishment Mechanism in Q-Learning Pricing Agents
After replicating a classic Bertrand duopoly Q-learning pricing experiment, the study audits the punishment mechanism on frozen policies and finds supracompetitive prices can be explained by a single attractor basin rather than selective retaliation. Its value lies in revising how the standard test of algorithmic collusion is interpreted.
01 ABSTRACT
Independent Q-learning agents in a simulated Bertrand duopoly converge to supracompetitive prices, classically attributed to a learned retaliatory punishment scheme. Auditing frozen policies with learning off, the authors find that in nine of ten pairs (914 of 1,024 states) the entire state space is the basin of a single attractor, so every perturbation returns to its origin and a forced price increase triggers the same pattern as an undercut. Collusion survives because deviation is never tried, not because it would not pay, they argue.
02 KEY FINDINGS
- Frozen-policy audits show that in 9 of 10 converged pairs a single attractor basin covers the whole state space, so the retaliation pattern is not selective.
- A true optimal deviation computed by dynamic programming profits by 17.5% on average, while the one-period undercut in the classic test is unprofitable in 92% of pairs.
- Re-matching frozen agents trained independently collapses coordination by 80%, indicating coordination depends on the specific pairing rather than a transferable strategy.
AI GENERATED SUMMARY / DISCOVERED BY ARXIV Q-FIN