Clip-Low and Clip-High Asymmetry Drives Opposing Entropy Effects in RLVR Training of Large Language Models

Understanding how objective clipping drives policy entropy collapse allows technical teams to prevent premature model determinism during post-training without altering their reward functions.

* Researchers uncovered opposing entropy biases: A study from UCLA, Seoul National University, KRAFTON, Stanford, and Santa Clara University proves that in PPO and GRPO algorithms, clip-low increases policy entropy while clip-high decreases it. * Standard default settings accelerate collapse: Because pretrained language models exhibit heavy-head token distributions, symmetric clipping at standard settings like 0.2 allows clip-high effects to dominate, shrinking entropy regardless of the underlying reward signal. * Asymmetric clipping preserves exploration: Isolating and strengthening the clip-low threshold counters policy collapse, giving operators a direct mechanical control to sustain exploration across extended reasoning runs.

* Reinforcement learning with verifiable rewards (RLVR) has become standard: Post-training workflows for complex domains like mathematics rely on verifiable feedback, but models frequently suffer from entropy collapse where responses turn rigid and deterministic. * The field historically blamed reward signals: Previous diagnostics assumed that aggressive policy collapse stemmed entirely from how rewards were shaped, overlooking the numerical constraints built directly into optimization objectives. * Clipping serves as a mathematical boundary: In Proximal Policy Optimization (PPO) and Generalized Reward Policy Optimization (GRPO), clipping limits surrogate objective updates, but its asymmetric downstream impact on probability distributions was uncharacterized until now.

• Clip-low increases entropy: When advantage values are negative, the clip-low operator restricts updates on low-probability tokens. This restriction effectively cushions negative updates across the tail of the distribution, dispersing probability mass and increasing overall policy entropy.
• Clip-high decreases entropy: When advantage values are positive, the clip-high operator caps updates on high-probability tokens. However, because pretrained language models feature a "heavy-head" distribution—where a small subset of candidate tokens holds the overwhelming majority of probability mass—the upward pressure concentrated on these primary tokens consistently outweighs the dispersion from the tail.

* LLM post-training engineers: Teams running PPO or GRPO on reasoning pipelines will need to review default symmetric hyperparameter baselines to prevent silent degradation of token exploration. * AI research labs focused on RLVR: Researchers developing math and logic models will gain a structural knob to diagnose whether training plateauing is caused by reward noise or mathematical objective clipping. * Compute infrastructure leads: Training runs that previously failed due to early entropy collapse can be stabilized by adjusting clipping ratios, reducing wasted compute cycles spent restarting collapsed runs.

* Asymmetric clipping hyperparameter adoption: Track whether open-source fine-tuning libraries implement separate clip-low and clip-high flags instead of a unified epsilon parameter. * Validation on broader reasoning benchmarks: Watch for follow-up empirical studies testing asymmetric clipping beyond mathematics across general code generation and multi-step agentic tasks.