Reinforcement Learning, Pong
Two DQN agents for Atari Pong, a plain baseline and a tuned version, built to show which change actually makes an agent learn.
The tuned DQN training run: reward, smoothed performance, win rate, loss, Q-value evolution, episode length and epsilon decay over 2,500 episodes.
How it works
- Approach 1 is a basic DQN: a 1.68M-parameter CNN, one Q-network, MSE loss, a batch of 32, epsilon-greedy exploration. A clean reference point.
- Approach 2 adds a target network, prioritized replay, 8 parallel environments, reward shaping, Huber loss and a batch of 512.
- Comparing the two isolates what each change buys. Parallel environments and prioritized replay do most of the work; the target network mostly adds stability rather than score.
- Reward is sparse, one point at a time, and games run 800 to 1000 steps, so a random policy sits near minus 20.
Results
- The tuned DQN reached minus 11.23 average reward over 30 evaluation episodes, with a best episode of plus 6, its first win against the built-in opponent.
- It sustains 1,406-step rallies, six times the random baseline, and reaches minus 18 reward in a quarter of the episodes the baseline needs.
- Training loss dropped 27 times, and Q-value overestimation went from plus 3.5 to minus 0.5.
| Metric | Baseline | Approach 1 | Approach 2 |
|---|---|---|---|
| Eval avg reward | −20 | −12.9 | −11.2 |
| Best episode | −21 | −11 | +6 |
| Avg episode length | 250 | 871 | 1,406 |
| Win rate | 0% | 0% | 3.5% |
| Training loss | — | 0.012 | 0.00045 |
| Training time | — | 2h 53m | 5h 39m |
From the project deck. Baseline is a random policy.
Slides
The full project deck. You can read it right here.