Blog

A benchmark that puts TypeSafe's System One model Jev on the same Python code-review task as Gemini 3.8 Flash and Claude Fable 5.1, comparing cost, speed, accuracy, and consistency. Jev saves 45–274× on cost and responds 4–5× faster, but takes a small hit at 98% accuracy.

To cut the per-comment AI billing of a hobby scraper, a developer had Gemini label the data just once and ported it to the open-source NER model GLiNER. Total cost was $9 for labeling plus $2.5 of GPU, and the result was 0.83 F1 on the validation set. A popular Hacker News post that honestly lays out the real costs and pitfalls of a small fine-tuning job.

We break down an arXiv paper that adds a memory module (AMM) and a self-reflection module (SRM) to a dual-process agent and runs ablations in ScienceWorld. The key point: the performance bottleneck is execution-time control, not memory — and unlimited reflection is actually harmful.

We summarize a September 2026 arXiv paper that proves, in theory and experiment, that off-policy evaluation (OPE) can become exponentially hard under partial observability even with seemingly healthy logging data. It covers a lower bound where the required samples explode as (3/2)^H, along with practical implications.

DeepSeek V4.1 Flash scored a perfect run on the AI hacking benchmark 'Enclave,' breaching all 11 vulnerable targets. An attack-path audit revealed that 5 of those runs succeeded via unexpected routes the designers hadn't anticipated, prompting a benchmark overhaul. The credited execution cost was just $4.65 thanks to cached input tokens.

Heterogeneous asynchronous distributed training, where each worker uses a different data distribution, is bound to far slower time complexity than the homogeneous setting. A September 2026 arXiv paper proves this gap cannot be mathematically closed with common function-similarity assumptions, and shows homogeneous-level convergence guarantees are possible only when strong interpolation and a local Polyak-Łojasiewicz condition are required together.

A study addressing the threat of wireless devices whose logical identity persists even when hardware is secretly swapped. FreqSpaNet, a dual-branch network that separately learns frequency and directional information, scored an average AUROC of 96.31% on open-set hardware anomaly detection.

Navigation AI (VLN) is vulnerable to error accumulation, where a single wrong step keeps compounding. The new paper ENCP proposes uncertainty estimation with probabilistic guarantees here. By calibrating just one maximum value per episode, it guarantees coverage across the whole path, met all step-coverage targets in experiments, and uses prediction-set size to decide when to ask a human for help.

While auditing inference provider Baseten, the autonomous hacking agent Strix pulled an image from an exposed Harbor registry and, in just 25 minutes, found a GitHub token left behind in the Docker build history. That token had been valid for over three years and carried admin privileges. Here's the exposed path and how to defend against it.

A multi-agent harness for mathematics and theoretical computer science that explores several strategies before writing a proof, passes a readiness gate, then decomposes. It solved 71.0% on TCS-Bench with cross-model selection and 218 of 222 Codeforces problems, and was integrated into Google Antigravity as a Long Proof pattern.

The chronic ailment of AI coding agents whose memory resets when you close a session. The open-source 'agent-comms' proposes the AHRP protocol, bundling work context into capsules to hand off to other sessions and other computers, and even supporting real-time collaboration over a WebSocket mesh. We dug into this fledgling project that made waves on Hacker News.

Open-source YuE2 makes the melody and chord scores it produces via symbolic planning editable. It posts scores that compete with Suno on WildSongBench, while its center of gravity stays open — code under Apache 2.0 and the model under CC BY-NC 4.0.

An AI code review CLI Alibaba used internally for years is now open source. Built as a hybrid that mixes a deterministic pipeline with agents, it published benchmark results showing higher accuracy while cutting tokens to roughly one-ninth using the same model.

MAxBench is a benchmark that evaluates how to recover and steer multinomial concepts with many subcategories — like "animal," "country," "plant" — in LLMs. Comparing 10 methods across 4 models, affine subspaces worked better than single directions, and that advantage came largely from the concept-centered offset rather than the basis.

ECE, which measures how well a model's stated probability matches reality (calibration error), suffers the difficulty that it can't be estimated exactly on finite samples. This paper introduces rankECE, which obtains a value close to ECE using only rank comparisons with no tuning parameter, and shows theoretically and empirically that it's more accurate and far faster than existing bin approximations.

vernLLM on Show HN embeds rate limiting, circuit breaking, and backoff into application code to replace an external LLM gateway. We assess its operational impact in API-heavy services.

Angel Y. He and David Parker present the first PAC learning framework for handling transition uncertainty in general-sum concurrent stochastic games. Combining data-driven confidence intervals with a robust MDP exploration mechanism, it computes an approximate Nash equilibrium or soundly certifies that no equilibrium exists. Sample complexity was empirically shown to match the theoretical bound.

An introduction to 'Compile by Training,' which turns a spec into a local neural function with a single compile instead of calling a large model API on every input. In benchmarks it raised semantic accuracy from 22.4% to 83.6%, and after compilation it runs without any teacher model.

ponytail is an open source tool that teaches AI coding agents YAGNI-based minimalism. Benchmarks show 54% fewer LOC, 22% fewer tokens, and maintained safety, with install guides and tunable commands.

On the ARC-AGI-3 benchmark, GPT-6 Astra scored 62.7% on Standard and 99.9% on the Provider Adapter. As high as the score is, you have to look at the differences in cost and evaluation conditions too. This piece organizes the official results, cost comparison, action efficiency, algebraic shorthand, and why 99.9% is not evidence of AGI but merely benchmark saturation.

arXiv 2609.02881 shows that replacing 75% of a Transformer's dense layers with a Graph Machine preserves quality almost entirely. It is a new architecture that uses edges like pointers for dynamic routing, achieving O(1) access while keeping O(n) state.

Synthetic data alone doesn't guarantee better inference. arXiv 2608.28576 proposes a size-weight frontier that keeps target coverage when mixing real and LLM responses.

Agent Behavior standardizes repeated AI agent actions. This framework makes long-running agent behavior observable and measurable across recognition, judgment, execution, and recovery.

Trail of Bits' QEMU/KVM escape experiment with GPT 5.6-Cyber linked 3 zero-days into a host path after ~12 hours of autonomous search. Firecracker resisted, showing VM isolation is insufficient alone.