Reinforcement Learning
DeepRLAgentsforSuperMarioBros
DQN · Double DQN · PPO Benchmark
Comparing value-based and policy-gradient agents in the same environment.
Observation preprocessing
Setup
A comparative reinforcement-learning project implementing and evaluating value-based and policy-gradient agents in the same environment.
Raw 240x256x3 RGB frames are frame-skipped, max-pooled over final frames, resized and grayscaled to 84x84, converted to channel-first and normalized, then stacked across 4 consecutive frames for a final (4, 84, 84) observation carrying temporal information. The action space uses SIMPLE_MOVEMENT with 7 actions.
Observation pipeline
Frames are frame-skipped and max-pooled to reduce redundancy, resized and grayscaled to 84x84, converted to channel-first and normalized, then stacked across 4 consecutive frames so the (4, 84, 84) observation carries short-term temporal information such as motion direction and velocity.
Algorithms
DQN and Double DQN share a CNN encoder (Conv2d 4 to 32 k8 s4, 32 to 64 k4 s2, 64 to 64 k3 s1, Linear 3136 to 512) feeding action values, trained with experience replay, a target network, and Huber loss under the Adam optimizer. DDQN reduces the Q-value overestimation bias inherent to vanilla DQN by decoupling action selection from evaluation. PPO adds a clipped actor-critic objective for direct policy optimization.
DQN / DDQN
DQN and Double DQN share a CNN encoder (Conv2d 4 to 32 k8 s4, 32 to 64 k4 s2, 64 to 64 k3 s1, Linear 3136 to 512) feeding action values, trained with experience replay, a target network, and Huber loss under the Adam optimizer. DDQN reduces the Q-value overestimation bias inherent to vanilla DQN by decoupling action selection from evaluation.
PPO
PPO adds a clipped actor-critic objective for direct policy optimization, evaluated in the same environment and under the same frame-preprocessing pipeline as the value-based DQN and Double DQN agents.
Training mechanics
Training configuration: 1,000 episodes, replay buffer of 30,000, batch size 32, learning rate 0.00025, gamma 0.90, and a target network update every 5,000 steps.
Demo
Agent in action
Agent in action
A recorded run from the project environment, included to show the trained agent interacting with the game rather than only the training architecture.
Comparison
Algorithm comparison
| DQN | Double DQN | PPO | |
|---|---|---|---|
| Learning type | Value-based (Q-learning) | Value-based, decoupled action selection/evaluation | Policy-gradient |
| Loss / objective | Huber loss via Adam | Huber loss via Adam | Clipped actor-critic objective |
| Key mechanism | Experience replay, target network | Experience replay, target network, decoupled action eval | Direct policy optimization |
Experiment configuration
Config
3
algorithms compared: DQN, DDQN, PPO
Config
7
actions, SIMPLE_MOVEMENT space
Config
1,000
episode training configuration
Takeaways
Alongside LLM-centric work, this project demonstrates first-principles reinforcement learning: reward-driven optimization, exploration/exploitation, and policy-gradient methods implemented and compared directly rather than through a wrapped API.