When self-play Q-learning looks robust but remains exploitable

A self-play Q-learning agent can look strong against sampled opponents while remaining systematically exploitable by an adversarial best response.

Read the canonical research note or see the project evidence map.

Published distribution: DEV cross-post · X thread

This repository asks:

What causes apparently competent self-play Q-policies to remain exploitable, and which controlled interventions actually remove those vulnerabilities?

Tic-Tac-Toe is deliberately small: its policy can be evaluated exhaustively, so aggregate reward and sampled win rates do not have to stand in for actual worst-case behavior. The project is about ML evaluation, agent failure analysis, observability, and reproducible experimental reasoning—not game development.

The final checkpoints select exact-optimal greedy actions on 99.93–99.96% of all 4,520 reachable current-player states. More importantly, an exact best response finds zero losing states in the reachable regions of all six final checkpoint-seat policies. Sampled W/D/L against heuristic opponents is secondary.

Baseline seed 211, playing O from ..X/.../..., chose action 3 instead of the safe center. The state had been visited 1,006 times, and the losing action led the center by 0.07697 in learned Q-value. This was not simply an unseen-state failure: sampled self-play had confidently learned the wrong continuation order.

Opponent-mixture training corrected that local move and eliminated losses to the sampled randomized and fork-aware heuristics, but all 6/6 frozen policies were still force-loss exploitable. Opponent breadth helped without solving the exact robustness problem.

The original update observes one sampled opponent response b:

target = r_b + γ max_a Q(s_b, a)

The adversarial opponent-response backup explicitly simulates legal responses:

target = min_b [r_b + γ max_a Q(s_b, a)]

Successor states are encoded from the original learner's perspective; the code does not incorrectly minimize opponent-perspective Q-values. Terminal wins, losses, and draws are handled directly. The exact solver is an evaluation oracle, never a training oracle. This intervention reduced exploitability from 6/6 to 4/6, while remaining failures shifted to sparsely updated state-action pairs and unresolved zero-value ties.

D4 canonicalization applies all rotations and reflections after constructing the current-player-relative state. State and action coordinates are transformed together and reversibly.

For the sparse state OX./..X/..., the pre-symmetry exact orientation had only 29 visits and its critical action only four updates. Canonicalization pooled 169 visits across eight orientations; the winning action received 78 equivalent updates and the ranking became Win > Draw > Loss.

Reachable states compressed from 4,520 to 627, and state-actions from 16,167 to 2,270. Under the matched update budget, all three seeds and both seats became non-losing against an exact best response.

See docs/reproducibility.md for environment setup, expected runtimes, artifact provenance, and the complete command sequence.

python -m venv .venv
# Windows: .venv\Scripts\python -m pip install -e .
# Linux/macOS: .venv/bin/python -m pip install -e .
python -m unittest discover -s tests -v

Experiment entry points:

python -m src.q_learning
python -m src.robust_eval
python -m src.mixture_experiment
python -m src.adversarial_backup_experiment
python -m src.symmetry_experiment

• Methodology: contracts, targets, and claim boundaries.
• Experiment summary: controlled progression.
• Failure analysis: trajectory-level mechanisms.
• experiments/: pre-registered change and verdict for each stage.
• artifacts/: selected machine-readable metrics and trajectories.
• checkpoints/: small frozen Q-tables for all three seeds at every stage.
• tests/: perspective, terminal, legal-action, best-response, and D4 tests.

This controlled study demonstrates that, in this implementation, sampled self-play evaluation hid exact exploitability; opponent diversity alone did not eliminate it; adversarial response backups materially reduced it; and symmetry- aware evidence pooling removed the remaining reachable forced-loss vulnerabilities under a matched budget. It is not a new RL algorithm or a general theorem about self-play, multi-agent RL, or adversarial backups. Larger stochastic, continuous, or partially observable environments may behave differently.

Original project code and documentation are released under the MIT License. The environment is provided by PettingZoo, maintained by the Farama Foundation. Dependency licenses and citation information are documented in THIRD_PARTY_NOTICES.md.