SpireBench vs Human Experts: How AI Agents Struggle with Long-Term Strategy in Slay the Spire

TL;DR: SpireBench reveals that top LLMs fail at sustained strategy in Slay the Spire, lagging behind expert humans as games progress despite strong early performance. This highlights a critical ‘horizon gap’ where agents prioritize short-term safety over long-term deck building, proving current models lack genuine strategic reasoning.

Key facts

• Researchers introduced SpireBench, a benchmark using the full commercial version of Slay the Spire to test LLMs on extended strategic planning and the ‘horizon gap’.
• The study evaluated 20 LLMs at the game’s highest difficulty, finding that while frontier models matched average humans early on, they lagged behind expert players in later acts.
• Stronger models succeeded through better tactical execution in individual combat encounters rather than superior high-level long-term vision or planning capabilities.
• Renaming core game terms (e.g., ‘ascension level’ to ‘tower floor’) drastically reduced model performance, indicating reliance on pre-existing memorized knowledge over real-time inference.
• Self-generated memory logs failed to improve agent performance due to inaccuracies and an overly cautious approach, suggesting a lack of meaningful learning from experience.
• A February 2026 paper introduced SAGE for generating physically valid 3D scenes, while a June 2025 position paper suggests Small Language Models may be more efficient for repetitive agentic tasks.

Evaluating Long-Horizon Strategy in AI Agents

Researchers have introduced SpireBench, a benchmark that uses the commercial deck-building game Slay the Spire (StS) to test how well agentic large language models (LLMs) handle extended strategic planning [1]. Unlike prior benchmarks focused on short, self-contained tasks, SpireBench evaluates performance over multiple in-game acts—where early decisions shape long-term outcomes. This setup directly addresses the ‘horizon gap’: the tendency of models to perform well on immediate steps but fail when required to maintain coherent strategy across many sequential choices [2].

Model Performance Across Game Stages

The study evaluated 20 LLMs at StS’s highest difficulty level, comparing their results to both heuristic baselines and human benchmarks. Frontier models outperformed rule-based agents and matched average human players in early stages [1]. However, the gap between these models and expert humans grew significantly as gameplay progressed. While strong agents won the first act at rates comparable to experts, they consistently underperformed in later acts, indicating a breakdown in sustained planning.

Tactical Execution Over High-Level Planning

What separated stronger from weaker models was not superior long-term vision but better execution during individual combat encounters. Success in each fight—such as managing health and resource use—had a compounding effect on deck strength over time. Yet all tested models showed a consistent misjudgment of risk: they prioritized preserving health points instead of strategically spending them to build stronger decks, a behavior that ultimately hindered long-term victory [1].

Reliance on Prior Knowledge and Memory Failures

To assess whether models were reasoning or relying on memorized knowledge, researchers renamed core game terms (e.g., ‘ascension level’ became ‘tower floor’). This renaming drastically reduced model performance, indicating that success depended heavily on pre-existing exposure to StS mechanics rather than real-time strategic inference [1].

Additionally, models did not improve from self-generated memory logs across runs. These logs were often inaccurate and reflected an overly cautious approach, suggesting agents lack the ability to learn from experience in a meaningful way [1].

Why SpireBench Stands Out

SpireBench is distinguished by being the first benchmark to grant models access to the complete, unmodified version of Slay the Spire. This closed-loop environment enables a more rigorous evaluation than previous efforts that used simplified or abstracted versions of gameplay [1], positioning it as an unsaturated benchmark capable of diagnosing exactly where and why agents fail in complex, evolving scenarios.

Broader Context in Agentic AI Research

While SpireBench focuses on strategic depth, other recent work explores different dimensions of agentic systems. A February 2026 paper introduced SAGE, a framework for generating physically valid 3D scenes using critics to assess realism and stability [3]. Meanwhile, a June 2025 position paper argues that Small Language Models (SLMs) may be better suited than LLMs for specialized, repetitive agentic tasks due to their economic efficiency [4]. These trends suggest future systems may rely on hybrid architectures rather than monolithic models.

earlier research found a ~15% disparity between sequence and graph planning performance even in GPT-4—a gap that highlights ongoing challenges in structured reasoning [5]. SpireBench adds to this body of work by showing that long-horizon strategy remains a critical weakness across current agentic models.

Sources

1. SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire (arxiv.org) — 2026-10-08
2. SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire (arxiv.org)
3. SAGE: Scalable Agentic 3D Scene Generation for Embodied AI (arxiv.org) — 2026-02-10
4. Small Language Models are the Future of Agentic AI (arxiv.org) — 2025-06-02
5. Benchmarking Agentic Workflow Generation (arxiv.org) — 2024-10-10