67 Cents, 1.5 Hours to Train a Tiny Transformer Beating LLMs on ARC-AGI-1

67 Cents and 1.5 Hours to Score 44% on ARC-AGI-1

According to a blog post published by Mithil Vakde on July 25, 2026, he trained a small Transformer from scratch on a single RTX 5090 and achieved 44% on the official ARC-AGI-1 evaluation and 7% on ARC-AGI-2. The total cost from training to inference was about 67 cents, and the elapsed time was 1.5 hours. The result came from a single transformer on a single GPU, without a giant model like an LLM or massive pretraining, so it is a fairly strong level for that scale.

Why Standardized Efficiency Matters

The core goal of this experiment is standardized efficiency. Recent AI research often raises performance by increasing compute and parameter counts, but that approach quickly drives up costs. Improving standardized efficiency makes it possible to achieve meaningful performance with less data and fewer compute resources, so researchers can repeat experiments much more easily. The ARC benchmark is well suited for testing this goal. The total puzzle set is about 1,000 items, and each puzzle is a meta-learning task that shares rules with others, allowing direct verification of generalization ability.

How Is the Model Trained?

The basic structure is relatively simple. Input/output tile patterns are made into a token descriptor, and the small Transformer is trained to autoregressively predict the next token. Here, instead of training on input tiles, the author chose a supervised learning method that uses output-tile-only extraction during inference. This change alone raised the score from about 40% to 44%, and further testing showed additional gains. Test-time augmentation easily breaks weak small models, but this work shows a different failure mode from existing methods that target minimal test-time augmentation.

Position information is encoded with 3D RoPE, and each puzzle is given a separate learnable task-specific embedding. The system is strengthened through color permutation and duality transformation, and during inference, the test input is transformed, inverse transformation is applied after output, and the two most frequent outputs are submitted using the AAIVR method.

Key Architecture Changes That Boost Performance

The author concretely identified the design elements that lifted the score from about 40% to 44%. The performance improvements came mainly from 3D RoPE, task-specific embeddings, SwiGLU, RMSNorm, and 8 adapters. For cost reduction, the switch to the NorMuon optimizer and variable-length FlashAttention, along with the inference FlexAttention controller, were key.

Ablation results show that replacing 3D RoPE with 1D RoPE drops the score to about 24%, and removing task-specific embeddings also cuts it by roughly the same amount. These numbers show how important both components are for expressiveness. Training on input tiles instead slightly reduces the score to about 39%, and the aggregated success rate across multiple runs is 55%, suggesting that reaching 65% under the same conditions is still possible.

The 67 Cent Controversy and Test-Set Training Criticism

This result has faced criticism from some quarters about whether it constitutes 'test-set training.' The author clearly states that the test output labels for the evaluation puzzles were hidden and were not used for training. Because ARC is a meta-learning benchmark, the author also emphasizes that it is permissible to learn the prediction check of evaluation puzzles. Moreover, the 67 cents is the end-to-end cost from model initialization through full task interpretation, excluding pretraining costs, based on the price of using an RTX 5090 for 2 hours on Vast.ai.

Why Does This Result Matter?

Achieving 44% on ARC-AGI-1 with just a single transformer shows that there is a path to improving standardized efficiency at low cost, even without synthetic data or huge models. By releasing the code and detailed implementation changes together, other researchers can directly run experiments to further reduce costs or improve scores under the same conditions. However, the criticism that training on all evaluation puzzles at once differs from real-world problems is also valid, so it is worth analyzing under which conditions this result holds.

Reference Links

• Original blog: 44% on ARC-AGI-1 in 67 cents - Mithil Vakde
• Public code: mdlARC GitHub
• GeekNews thread: 67 cents, 1.5-hour training of a tiny Transformer