Skip to content
KoishiAI
ไทย
← Back to all articles
ai-research ai-agents open-source benchmark

AI Agents in Scientific ML: Engineering vs Insight

New research reveals AI agents excel at ML engineering but lack deep domain insight in scientific discovery, as shown by the Agent4Sci benchmark study.

KoishiAI · Editor: เกียรติดำรง ตรีครุธพันธ์ · · 3 min read
AI-drafted from cited sources, fact-checked against them by an automated pipeline, and read by a human editor for clarity. How we work · Standards · Report an error
Two scientists wearing lab coats and goggles analyze data on a computer in a modern laboratory.
Photo by Pavel Danilyuk on Pexels

TL;DR: Agent4Sci reveals that while AI agents like Claude Opus improve ML baselines by 10.4%, they rely on engineering rather than insight. Adding structured skills boosts this gain to 20.1%, proving human-guided frameworks are essential for genuine scientific innovation.

Key facts

  • Agent4Sci is a long-horizon testbed evaluating AI agents across 20 benchmarks in 11 domains under ‘in-the-wild’ conditions.
  • Claude Opus achieved a mean relative improvement of 10.4% over published baselines without any flagged academic misconduct.
  • Agents primarily rely on routine ML-engineering tricks, struggling with novelty miscalibration and rapid collapse into local engineering sweeps.
  • Lightweight plug-in skills for research ideation boosted Claude Opus’s performance from 10.4% to 20.1% mean relative improvement.
  • The SciResearcher-8B model achieved a state-of-the-art score of 19.46% on the HLE-Bio/Chem-Gold benchmark at its parameter scale.
  • SciResearcher-8B demonstrated 13-15% absolute gains on SuperGPQA-Hard-Biology and TRQA-Literature benchmarks, surpassing larger proprietary agents.

Frontier coding agents are being positioned as end-to-end automated researchers, promising to accelerate scientific discovery by autonomously improving machine learning models. A new study suggests these systems are powerful engineers but lack the deep domain insight required for genuine breakthroughs.

The research introduces Agent4Sci, a long-horizon testbed designed to measure how far current AI agents can improve upon published best models in scientific machine learning contexts [1][2]. Unlike previous evaluations that rank performance on static leaderboards, this framework operates under “in-the-wild” conditions across 20 benchmarks spanning 11 diverse domains [1][4]. The goal is to assess practical progress and limitations when agents are tasked with real-world scientific improvement rather than just benchmark matching.

Empirical results show that today’s frontier agents successfully advance the state of the art on every benchmark tested. Claude Opus, one of the evaluated models, achieved a mean relative improvement of 10.4% over published baselines [1][2]. Notably, no runs were flagged for academic misconduct during the evaluation period [1][2]. However, trajectory analysis reveals that these gains stem almost entirely from routine ML-engineering tricks rather than research-level insight or domain-specific understanding [1][3].

The study identifies three recurring limitations in agent behavior: novelty miscalibration, resource under-utilization, and a rapid collapse into local engineering sweeps [1][4]. Essentially, the agents are highly effective at optimizing existing code structures but struggle to conceptualize fundamentally new approaches. This suggests that while AI can automate the labor-intensive aspects of model refinement, it has yet to master the creative leap required for scientific innovation.

To address these gaps, researchers implemented lightweight plug-in skills focused on research ideation, experiment management, and scheduled checklists [1][2]. These interventions significantly boosted performance, raising Claude Opus’s mean relative improvement from 10.4% to 20.1% and Codex’s from 4.4% to 8.1% [1][2]. Furthermore, these skills yielded more diverse and domain-relevant interventions, suggesting a pathway for pushing agents from basic engineering toward genuine scientific insight.

In a related development, another study titled SciResearcher explores scaling deep research agents for frontier scientific reasoning [5]. This framework addresses the challenge of scattered domain knowledge by synthesizing conceptual and computational tasks grounded in academic evidence [5]. Using this curated data for supervised fine-tuning and agentic reinforcement learning, the authors developed SciResearcher-8B [5]. This model achieved a new state-of-the-art at its parameter scale on the HLE-Bio/Chem-Gold benchmark (19.46%) and demonstrated 13-15% absolute gains on SuperGPQA-Hard-Biology and TRQA-Literature benchmarks, surpassing several larger proprietary agents [5].

Together, these studies highlight both the current capabilities of AI in automating scientific ML workflows and the distinct challenges remaining in achieving true domain insight. While Agent4Sci emphasizes the need for structural support (skills/checklists) to enhance existing frontier models, SciResearcher focuses on data construction and training paradigms to build specialized foundation models for scientific reasoning.

Sources

  1. Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild? (arxiv.org) — 2026-10-20
  2. Paper page - Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild? (huggingface.co) — 2026-10-22
  3. Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild? (arxiv.org) — 2026-04-24
  4. HyperAI (hyper.ai) — 2026-10-23
  5. SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoning (arxiv.org) — 2026-05-02

Frequently asked questions

What is Agent4Sci and how does it evaluate AI agents? ▼
Agent4Sci is a long-horizon testbed designed to measure how far current AI agents can improve upon published best models in real-world scientific machine learning contexts. It operates under "in-the-wild" conditions across 20 benchmarks spanning 11 diverse domains, focusing on practical progress rather than static leaderboard rankings.
How well do AI agents perform in scientific machine learning according to the study? ▼
The study found that frontier coding agents successfully advanced the state of the art on every benchmark tested, with Claude Opus achieving a mean relative improvement of 10.4% over published baselines. However, these gains stemmed almost entirely from routine ML-engineering tricks rather than deep domain insight or research-level creativity.
What are the key limitations of AI agents in scientific research? ▼
The researchers identified three main limitations: novelty miscalibration, resource under-utilization, and a rapid collapse into local engineering sweeps. Essentially, while agents are effective at optimizing existing code structures, they struggle to conceptualize fundamentally new approaches without structured interventions.
What is SciResearcher-8B and how does it differ from general AI agents? ▼
SciResearcher-8B is a specialized model developed using curated data for supervised fine-tuning and agentic reinforcement learning. It achieved a state-of-the-art score on the HLE-Bio/Chem-Gold benchmark and demonstrated significant gains on other scientific benchmarks, surpassing several larger proprietary agents.
How can AI agents be improved for better scientific reasoning? ▼
Implementing lightweight plug-in skills focused on research ideation, experiment management, and scheduled checklists significantly boosted agent performance. For example, these interventions raised Claude Opus's mean relative improvement from 10.4% to 20.1%, suggesting a pathway toward genuine scientific insight.