jaintle/visual-rl-locomotion: v1.0 — 20k Multi-Seed State vs Pixel PPO Benchmark

Extends the 20k baseline (v1.0-20k-benchmark) to 1 000 000 training steps across all three conditions, revealing how the state–pixel gap evolves at scale under identical PPO hyperparameters. Results (1m_eval10 · 3 seeds · 5 eval episodes · eval every 10k steps) | Mode | Final Eval Return (mean ± std) | Final Step | |:-----|:-------------------------------|----------:| | state | 1915.1 ± 992.0 | 1,001,472 | | pixels (no stack) | 121.8 ± 40.2 | 1,001,472 | | pixels (stack=4) | 97.7 ± 36.0 | 1,001,472 | Key findings Gap widens with scale. State–pixel gap grows from 4.4× (20k) to 15.7× (1M) for no-stack pixels and 19.6× for stack=4. The CNN encoder does not close the representational gap within a 1M-step budget under vanilla PPO. Frame stacking reversal. stack=4 outperformed no-stack at 20k (+31%); at 1M it finishes 20% below no-stack (97.7 vs 121.8). The larger 12-channel input appears to hinder optimisation at scale without augmentation or a tuned learning rate. State reaches meaningful locomotion. 1915 ± 992 reflects real hopper locomotion in favourable seeds, though the high σ (52%) reflects Hopper-v4's characteristic instability — once the hopper falls, returns collapse. What's new since v1.0 reproduce_hopper_v4_1m.sh — 1M benchmark end-to-end script results_hopper_v4_1m.md — aggregated 1M results train_ppo.py / run_compare.py — mps (Apple Silicon) device support README.md — 1M benchmark section, updated findings and limitations