Why Self-Improving Agents Are Fragile

In the age of agents, reliability is still unfinished

As coding agents enter production and model gateways combine with large payment infrastructure these days (today's news of the OpenRouter–Stripe combination and Codex's full usage reset are examples), the trust that agents actually "work well" still stands on shallow sand. The three papers we look at today each address, at a different level, the common denominator of agentic AI: the difficulty of measurement. The fragility of "self-improving agents" that learn and improve online, the absence of tokenizer evaluation, and the two-way acceptance when a person delegates a conversation to an agent. All three converge on a single message: before trusting an agent, first check whether you're measuring its performance properly.

The fragility of memory-based self-improving agents

The paper On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification confronts head-on the reliability of "memory-based self-improving agents" that learn on an online task stream, maintain a text memory bank, and incrementally improve. The researchers re-validated existing evaluation by broadening it along two axes.

As a result, two observations stand out. First, amplification of evaluation noise. Agent evaluation is noisy to begin with, and overlaying a self-improving loop makes that noise larger. Second, task-order dependence. The default order commonly used in prior work has an implicit curriculum embedded in it, effectively acting as a hidden precondition for success.

The illusion the hidden curriculum creates

If the conclusion of "an agent that automatically gets better when left alone" was in fact due to inertia created by task order, that result is an illusion untransferable to other environments and other starting points. The researchers point to the underspecification (lack of concreteness) of tasks and environments as the root of this fragility and tested the hypothesis by injecting better specification information — detailed rubrics and environment feedback — into the memory composition. Adding that information partly narrowed the performance drop from earlier experiments, but a significant gap remained, suggesting additional causes exist.

The engineering implication is clear. When adopting a self-improving agent, don't trust a single run's result; repeat many times and examine variance, change task order and initial state to check robustness, and specify evaluation metrics and task specifications concretely enough.

Tokenizers too, a swamp of non-evaluation: TokEval

The second paper, TokEval: A Tokenizer Evaluation Suite, points out the problem that the tokenizer, which governs the LM performance underlying an agent, is chosen with almost no evaluation. Beyond the standard metrics of fertility and compression rate, it proposes evaluation metrics that capture linguistic and structural meaning, such as UTF-8 character-boundary integrity and digit (place-value) boundary alignment in math.

In controlled pretraining experiments varying only the tokenizer's training-data mix, pre-tokenization strategy, and learning algorithm, the results are interesting.

That tokenizer design choices act differently on model ability means a quantitative evaluation is possible — pre-filtering with intrinsic metrics instead of heavy pretraining sweeps. TokEval provides a basis for substituting intrinsic measurement when the two approaches diverge.

Agent-delegation asymmetry: delegating is easy, but responding is hard

The third paper, Delegation Asymmetry in Agentic Recommender Systems, measured the two-way condition that for autonomous LLM agents conversing on a user's behalf on a matching platform to work, a customer must not only "delegate" to an agent but the other side must "accept" the agent conversation being sent. It analyzed a large-scale survey of a real dating platform (generated-profile trait N=2,894 / autonomous conversation agent N=2,617, in two languages) with a latent-variable measurement model.

As a result, willingness to send and to receive separated into distinct constructs (correlation rho=0.92 but separable), and a systematic delegation asymmetry was quantified. The acceptance threshold needed to deploy one's own agent (-0.38) was far lower than the value for interacting with the other party's agent (+0.32, full participation +1.39), and the deployment propensity was roughly 3× higher than the participation propensity. In a random-pairing counterfactual calculation, directional pairs where agent deployment and recipient participation combine amount to only 4–13%.

That is, a "world where agents speak on my behalf" won't open on technology alone. Without reflecting two-sided acceptance (consent to receive) in the design, an imbalance arises where there are delegators but no one to receive them.

Wrap-up: agent reliability starts with "measurement"

The three papers, though at different levels, point in one direction. Before believing agentic AI actually works, you must first answer whether you're measuring that behavior properly. That's why self-improving agents must remember run-repeat variance and task order, tokenizers must remember intrinsic evaluation metrics, and agent-adopting products must remember two-way acceptance.

Today's industry news (the combining of large model gateways, the full usage reset of coding agents) shows agents are being commercialized in earnest. The more so, the more the measurement gaps these papers point out could lead to accidents. Before you ship an agent, first check whether your evaluation metrics are robust to noise.