Does momentum help or hurt when the objective keeps moving?
Accelerated PPO Variants · 2026
Standard PPO uses a clipped surrogate objective to constrain policy updates, but what happens when you add momentum to that? Acceleration mechanisms that work well in supervised learning interact non-trivially with a non-stationary value function and a trust-region constraint. I am implementing and benchmarking accelerated PPO variants against a clean PPO baseline across MuJoCo environments with ten seeds each, tracking not just reward but gradient norms, entropy and policy ratios to understand the optimization dynamics.
Outcome: Ongoing.
