What is RealtimeBench and what does it test?

RealtimeBench is a new benchmark designed to test how well large language model agents perform in real-time environments where gameplay continues running live during inference. It includes 180 tasks across six distinct game types, such as puzzle-solving and navigating dynamic mazes, to simulate continuously changing conditions.

Why do faster AI models sometimes beat smarter ones in real-time tasks?

The benchmark reveals that in live, real-time settings, execution speed often outweighs reasoning depth because even correct decisions arrive too late if the model is slow. Paired evaluations show that a faster but less capable model like Jev can outperform more advanced agents when timing is tight.

What is TORSI and how does it work?

TORSI stands for Two-Timescale Offline Recursive Self-Improvement, a method that uses a 'slow teacher' and a 'fast student' model working at different speeds. The slow teacher analyzes failures in paused environments to generate reusable instructions, while the fast student executes actions during live gameplay using those pre-generated plans.

How much did TORSI improve AI agent scores in the RealtimeBench tests?

Applying two rounds of TORSI boosted Jev’s average score from 13.51 to 24.19 on a 0–100 progress scale across all six game categories in the benchmark. This improvement was consistent and broad, demonstrating that the method effectively enhances performance without requiring retraining.

Can the improvements from TORSI be applied to other AI models?

The instructions generated by the teacher model improved performance for three other models without any need for retraining or fine-tuning. This suggests that TORSI creates generalizable strategies that can benefit multiple systems beyond just the specific agent it was trained on.

Jev vs GPT-6.1-sol: What RealtimeBench Reveals About AI Agent Speed and Smarts

TL;DR: RealtimeBench reveals that speed often trumps reasoning depth in live environments, with the faster Jev model outperforming slower alternatives. Applying the TORSI method boosted Jev’s score from 13.51 to 24.19 by decoupling deep reflection from rapid execution.

Key facts

RealtimeBench is a new benchmark featuring 180 tasks across six game types to test LLM agents in continuously running environments without pausing for deliberation.
The study reveals that faster models like Jev often outperform slower, more advanced agents such as GPT-6.1-sol in real-time settings where execution speed outweighs reasoning depth.
Researchers introduced Two-Timescale Offline Recursive Self-Improvement (TORSI), a method using a slow teacher to analyze failures and generate instructions for a fast student agent.
Applying two rounds of TORSI boosted Jev’s average score on RealtimeBench from 13.51 to 24.19 on a 0–100 progress scale across all six game categories.
Instructions generated by the teacher model improved performance for three other models without requiring any additional retraining or fine-tuning, demonstrating generalizable strategies.

The Race Is On: When AI Agents Must Think Fast

In video games, a split-second delay can mean the difference between victory and defeat. Now, researchers have built a benchmark to test how well large language model (LLM) agents handle real-time challenges — not in paused simulations, but while games continue running live during inference [1]. This setup exposes a critical flaw: even if an AI agent has deep reasoning skills, it can fail if its response arrives too late.

The new benchmark, called RealtimeBench, includes 180 tasks across six distinct game types designed to simulate continuously changing environments [1][2]. These tasks range from puzzle-solving under time pressure to navigating dynamic mazes and managing resources in evolving scenarios. Unlike traditional evaluations that pause the environment for deliberation, RealtimeBench keeps gameplay active — mimicking real-world conditions where delays are costly.

The Speed vs. Depth Paradox

When games are paused, allowing agents time to think deeply, more powerful models tend to win. But in live play, speed becomes the dominant factor. Paired evaluations show that a faster but less capable model, referred to as Jev, can outperform a more advanced agent when timing is tight [1][2].

This isn’t because the slower model is smarter — it’s because it acts quicker. In real-time settings, even correct decisions arrive too late to matter. The core insight from RealtimeBench is clear: in dynamic environments, execution speed often outweighs reasoning depth.

Teaching AI to Learn Between Moves

To bridge this gap, researchers introduced a new method called Two-Timescale Offline Recursive Self-Improvement (TORSI) [1][2]. The approach uses two models working at different speeds: a ‘slow teacher’ and a ‘fast student’.

The slow teacher doesn’t act during gameplay. Instead, it analyzes failures after each attempt, runs simulations in paused environments to test fixes, and generates reusable instructions for the student model [1][2]. These instructions are designed to improve decision-making without requiring real-time input.

Meanwhile, the fast student executes actions during live gameplay, relying entirely on pre-generated guidance. Updates are only adopted if they lead to higher performance in validation tests [1][2].

Real Results: From 13.51 to 24.19

In evaluations using RealtimeBench’s 180 tasks, applying two rounds of TORSI boosted Jev’s average score from 13.51 to 24.19 on a 0–100 progress scale [1][2]. This improvement was consistent across all six game categories, showing that the method works broadly, not just in niche scenarios.

Even more striking, the instructions generated by the teacher improved performance for three other models without requiring any retraining or fine-tuning [1][2]. This suggests that TORSI doesn’t just help one agent — it can create generalizable strategies that benefit multiple systems.

What This Means for AI Development

The findings challenge a common assumption in AI research: that better reasoning always leads to better outcomes. In real-world applications — from autonomous vehicles to financial trading bots — the environment never pauses. Agents must act quickly, and delays can be catastrophic.

TORSI offers a practical solution: separate thinking from doing. Let the teacher reflect deeply during downtime, then deliver optimized plans for the student to execute in real time. This two-timescale approach mirrors how humans learn — by reflecting on mistakes and improving future actions without constant supervision.

The success of TORSI suggests that future AI systems won’t just be smarter or faster — they’ll be better organized. By decoupling deliberation, learning, and action across different timescales, developers can build agents that are both thoughtful and timely.

Sources

The World Won’t Wait: Evaluating and Teaching Agents Under a Live Clock (arxiv.org) — 2026-10-14
The World Won’t Wait: Evaluating and Teaching Agents Under a Live Clock (ai-daily-beacon.up.railway.app) — 2026-10-16