Can You Read Step Importance in CoT Text? Legibility ≠ Interpretability

Why This Paper Matters Now

The idea that chain-of-thought (CoT) text makes a model's reasoning "readable" is deeply ingrained. Tools like LLM judges, process reward models, and generative critics all analyze CoT steps to score outputs or provide step-by-step feedback. But does the text actually reflect what the model is thinking? This COLM 2026 paper, posted to arXiv in September 2026, gives a rigorous answer.

Core Concept: Measuring Step Importance as Advantage

The authors define the importance of each reasoning step as advantage. In RL, advantage measures "how much better this action is than the baseline in the current state." Here, for each reasoning step, they remove it and resample; the drop in final accuracy indicates importance. They measure this with Monte Carlo rollouts. Larger drops mark the step as consequential; smaller or no change marks it as uninformative. Importantly, this labeling does not rely on looking at the text content; it is grounded in actual model behavior.

Experimental Results: The Limits of Judges and Fine-tuned Critics

They tested how well LLM judges can recover this "true importance." They evaluated judges ranging from Qwen3-1.7B up to Qwen3-32B and Qwen3-6-27B zero-shot, plus fine-tuned critics with a regression head on each backbone. Metrics are PR-AUC and precision@k%.

The results are clear. Out-of-the-box judges perform only slightly above chance and cannot approach the noise ceiling. Fine-tuned critics reach PR-AUC 0.28–0.30 on internal splits and 0.18–0.32 on external splits, near the noise ceiling on incorrect responses. But on correct responses, they score only 0.065–0.10, roughly 10–20% of the noise ceiling. So when the model is right, simply reading the text and judging which steps were decisive is very hard.

Additionally, they conducted a cue-based faithfulness test. When given the cue "the answer is B," the model often follows that cue in reasoning steps even when it was not cued. In the base setting, 58% of responses include the consequential step; in the cue setting, that drops to 15%. Even without an explicit cue in the text, the model's internal search can become "decision-biased" toward the cue. This is an important caveat.

Practical Implications for CoT Interpretation

This research carries a stronger message than simply "judges are weak." When step importance is defined as an RL concept, text alone tends to compress that importance. Process reward models or generative critics need additional signals beyond text to provide step-level supervision orpidar feedback. In short, the fact that the thinking model or larger models perform better does not mean they "find the correct answer in the reasoning process." Rather, it may reflect a strong prior toward correct answers from the very first step. CoT may be better understood as a strong expression of prior confidence than as an explanation of how the answer was reached.

Research code and data are publicly available. The repository is at https://github.com/kdu4108/importance-advantage, and the dataset is on Hugging Face.

Figure 17. Confusion matrices for base/cue settings of the Qwen3-1.7B thinking model Source: arXiv:2609.04194