A Two-Stage Framework That Makes LLM Reasoning 5,000x Faster

Why trade-up recommendation matters

In online stores, the ability to recommend a better product instead of product A is directly linked to revenue and customer satisfaction. But this task is not as simple as a basic recommendation — it requires handling diversity, coverage, and cold-start at the same time. Even in the same category, different attribute levels like grade, color, and type can lead to very different trade-up candidates, so we need to deeply understand the user's purchase history and how product features change.

This paper addresses both issues together. It proposes a two-stage framework that distills LLM reasoning into a student model and specializes reasoning at the category level.

Core structure of the two-stage framework

Source: arXiv 2609.05363

Level 1 distills structured relation labels and free-text reasoning from a search-augmented few-shot LLM teacher. It then encodes the teacher's reasoning text and aligns it precisely using an embedding-space classifier and a regression target. At the reasoning checkpoint, the model uses only two 768-dim product embeddings, so LLM inference and text generation are unnecessary. The full pipeline runs at 15.5M parameters but performs 4-way relation classification.

Level 2 is a test-time specialization step called Product-Type Test-Time Training, or PT-TTT. For each product type, it uses a small number of labeled examples to optimize category-level anchors within the student model. Here, it learns only the LoRA-style low-rank module while keeping the base model frozen. If we route different test-time criteria per category during inference, the overall classification efficiency remains unchanged.

Experimental results and performance analysis

The team evaluated the approach on the Amazon-Walmart dataset with 29 product types, 1,019,241 training pairs, and 8,352 human-annotated gold standard matches. Looking at the main results, the 15.5M parameter reasoning-distilled student model achieved AUC 0.924 in the gold standard. A model using only the same structure's label reached 0.912, showing roughly a 0.012 point improvement over the baseline without reasoning.

When applying PT-TTT, AUC improved from 0.924 to 0.941, and average precision improved from 0.920 to 0.940. The most notable point is that the smaller reasoning-distilled model outperformed the larger vanilla student model in the gold standard. This appears to be because, while the smaller model trains faster, the teacher's free-text reasoning precisely refines and simplifies decision boundaries.

Practical deployment implications

The biggest strength of this research is inference cost and speed. In 100K-step pretraining on a standard 8-GPU machine, the distilled student model trains about 5,000 times faster than direct LLM inference, and inference cost is reduced by over 10,000 times. Because category-specific anchors are set after training, all queries can be reused, yielding an operational efficiency improvement of over 3,448 times compared to query-level customization.

Limitations and next steps

Because the current approach writes category-specific anchors per product type, it requires additional support costs as the number of categories increases and new items are added. Also, since the teacher LLM is black-box, the method relies on logit-level distillation rather than direct representation alignment; if the teacher's reasoning path changes, the quality of real-world recommendations can drop. While this paper validates experiments across 29 product types, distribution stability in actual production deployments with stricter budgets requires additional verification.

References

• Paper: Distill Globally, Adapt Locally: Reasoning Distillation and Product-Type Test-Time Training for Scalable Trade-Up Recommendation
• arXiv HTML: Distill Globally, Adapt Locally