Stable critic training recipe, summarized in 6 steps

Background

In RLHF pipelines, unstable critic training is a major cause of later-stage collapse. Even slight value drift can make policy gradients explode, and performance cliffs are common. Practical stabilization methods therefore matter for production-scale alignment work.

This post organizes that problem into a 6-step recipe. Authors Qi, Zhou, and Lee describe what regulation is needed at each step and experimentally verify why it works, in arXiv 2608.23566. The goal is not just incremental improvement, but a stable training curve from initialization to the final iteration.

6-step recipe

Their recommended recipe is:

1. Start with DPPO.
2. Apply value clipping.
3. Add MC rollouts.
4. Check regularization removal.
5. Use privileged baseline.
6. Finish with variance-based GAE.

This ordering was confirmed via ablation study. Removing any step causes validation reward or policy consistency to drop.

Key hyperparameters

Authors use lambda_pi=0.99 and lambda_V=1 in the critic loss. The critic head uses arctan activation clipped to the [0,1] range, keeping stable learning even without reward scaling.

Sanity test results

The sanity test used a 1.5B model on 1,460 topics for 1,500 iterations. Under the same settings, both reward and policy gradients remained stable.

Larger scale

The dataset size is 40.3K, and the same trend holds in 30B-A3B MoE. This part is observational rather than ablative, so replication is needed.

Code and references

Authors released QPHutu/golden_critic. You can directly inspect architectures and learning curves. The paper is available at arXiv 2608.23566, and a full HTML version is also provided.