Panorama of Agent V2

Posted on Edited on In Views:
This work describes the panorama of Agent area. It is suggested to review this article frequently.

1. What is Agent & Harness?

First, the Large Language Model(LLM) is very mature now. Its essence is a prediction model:

  • You input a string of token, and it outputs a string of token.

The LLM nowadays can answer our questions but it can not finish a job independently. For example, you can ask LLM how to create a new folder on PC and it can give you an answer. However, the LLM cannot create the folder directly because the only thing it can do is output tokens.

Now we have LLM to describe the workflow of a job. But we need to finish the workflow ourselves. ==Agent== is the software that can resolve problems independently.

==Agent== is usually composed two parts: ==LLM== and ==Harness==.

Agent ≈ LLM-based Policy + Harness

The LLM provides the main reasoning/policy capability, while the Harness provides orchestration, state/context management, tools, environment interaction, execution, and runtime controls. e.g. LLM thought it needs to read a file, then the Harness part will read the file and return the result to LLM.

In Agent, LLM is more likely a role of “Brain”, and the Harness is the “Limb”.

2. Agent’s Pursuit

Agent is designed to solve problem independently. Everything we do now is to improve Agent’s ability of solving problems.

We hope Agent can solve particular problems in daily life and complex problems in specific area like software engineering.

3. Agent Area Panorama

Agent is composed of LLM and Harness.

So it is obviously that we should enhance LLM’s performance in Agent area. And optimize the Harness part. Then we need benchmarks to judge the Agent’s performance.

For this research map, the main research areas are divided into four parts:

  • LLM Tuning
  • Harness Structure Optimization
  • Benchmark-related work
  • Multi-agent work

3.1 LLM Tuning

In Agent area, the most popular post-training methods are:

  • Imitation Learning (SFT)
  • Online RL
  • Offline RL
  • Preference Learning (DPO)

3.1.1 Imitation Learning (SFT)

ℒSFT = −∑tlog Pθ(tokent ∣ token < t) #### 3.1.1.1 Trajectory SFT

Core Process:

1
2
3
4
5
6
7
Expert Agent Trajectory
        ↓
Teacher Forcing
        ↓
Cross Entropy
        ↓
LLM

Trajectory (ReAct) :

1
2
3
4
5
6
7
task
→ thought
→ action
→ observation
→ action
→ ...
→ final answer

Relative papers:

  • SWE-Gym (ICML 2025), Rejection Sampling / Self-Improvement
  • SWE-Lego (arXiv:2601.01426, 2026)

3.1.1.2 Critical-Step SFT

A trajectory includes many steps, but not all steps are necessary. There’s one way only to SFT the critical steps in trajectories.

  • ATLaS — (Findings ACL 2025), only train critical steps

3.1.1.3 Synthetic Trajectory

One problem of SFT is that high-quality training data is few, so there’s a research direction is to compose Synthetic Trajectory.

  • SWE-smith, Synthetic Trajectory

3.1.1.4 Distillation

Distillation is to extract ==reasoning trajectories== form different teachers and distill into a student model.

1
2
3
4
5
6
7
8
9
10
11
12
13
Teacher A
  │
  ├── Tool-based reasoning
  │
Teacher B
  │
  └── Text reasoning
       │
       ▼
     Distill
       │
       ▼
Student Agent
  • Agentic-R1 (EMNLP 2025)

3.1.2 Online RL

3.1.2.1 Reward Design

The reward is a criterion to judge how good a trajectory is.

There are many ways to calculate reward:

1
2
3
4
5
6
7
            Reward
              │
  ┌───────────┼────────────┐
  │           │            │
Human      Verifier    Process Reward
  │           │            │
 RLHF        RLVR       PRM/PR

RLVR (Reinforcement Learning Verifiable Reward) is an important type, because many engineering problems are verifiable.

  • SWE-RL (NeurIPS 2025)
  • WebAgent-R1 (EMNLP 2025)

Process Reward Model(PRM) is another research roadmap; There are many steps in a trajectory. The traditional reward is trajectory-level, which means that LLM cannot distinguish if one certain step in a trajectory is important.

1
2
3
4
5
6
7
8
9
Step 1
Step 2
Step 3
...
Step 50
 ↓
Success
 ↓
Reward = 1
  • LLM cannot know if step 3 is really important.
1
2
3
4
5
Outcome Reward
       ↓
Process Reward
       ↓
Step-level Credit Assignment
  • Rewarding Progress (ICLR 2025)
  • PURE (NeurIPS 2025)
  • ReasonFlux-PRM (NeurIPS 2025)

3.1.2.2 RL Optimization Algorithm

After calculating the reward of a trajectory, we need to update the weights of LLM.

There are three ways to update weights, but the PMD is not commonly used.

1
2
3
4
5
6
7
8
9
10
11
12
         Reward
           │
           ▼
   Policy Optimization
           │
 ┌─────────┼─────────┐
 │         │         │
PPO       GRPO      PMD
 │         │         │
 └─────────┼─────────┘
           ▼
      Update LLM
3.1.2.2.1 GRPO

GRPO has become one of the most widely used policy-optimization methods for reasoning-oriented LLM post-training. A key difference from standard PPO is that GRPO estimates relative advantages from a group of sampled responses instead of maintaining a separate value/critic model.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Question x
   │
   ├── rollout 1 → R1
   ├── rollout 2 → R2
   ├── rollout 3 → R3
   ├── ...
   └── rollout G → RG
                 │
                 ▼
      Group Relative Advantage
                 │
                 ▼
                GRPO
                 │
                 ▼
                LLM
  • SWE-RL
3.1.2.2.2 PPO

The core thought of PPO (Proximal Policy Optimization) is:

  • Need to make LLM more inclined to generate high-reward responses, while ensuring that a single update does not deviate too far from the original model.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
Question x
    │
    ▼
Policy πold
    │
    ▼
rollout → trajectory
    │
    ├── Reward R
    │
    └── States / Actions
              │
              ▼
        Value / Critic V(s)
              │
              ▼
        Advantage A
              │
              ▼
        PPO Objective
              │
        ┌─────┴─────┐
        │           │
   Probability    Clip
      Ratio
        │           │
        └─────┬─────┘
              ▼
             Loss
              │
              ▼
       Update Policy πθ

3.1.3 Offline RL

  • OREO (ACL 2025): Offline Reinforcement Learning for LLM Multi-step Reasoning
  • Reward-Weighted Fine-Tuning (NeurIPS 2025): Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization

3.1.4 Preference Learning (DPO)

DPO (Direct Preference Optimization) is widely used in LLM post-training, but in Agent-specific post-training it is generally less dominant than SFT and online RL.

SFT makes LLM realize what is right answer while Preference Learning just tells LLM: A is better than B.

1
2
3
4
Task

Trajectory A -> good
Trajectory B -> bad

Train:

1
2
πθ(A|x) ↑
πθ(B|x) ↓

Basic DPO process:

1
2
3
4
5
6
7
Prompt
 ├── Chosen
 └── Rejected
       ↓
      DPO
       ↓
      LLM

The traditional DPO compare different trajectories. Nowadays the tendency is to research which certain step makes trajectory better.

1
2
3
4
5
6
7
Trajectory-level
       ↓
Step-level
       ↓
Action-group level
       ↓
Critical-step preference
  • HPL (ICLR 2026): Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents
  • Online DPO (ICLR 2025)
  • SDPO (ACL 2025)

3.2 Harness Structure

For this research map, the Harness is organized into seven engineering-oriented layers. This is a practical taxonomy rather than a universally accepted academic standard, and the layers can overlap in real systems.

Part Abstruct Description
Orchestration
Loop / Planning / ReAct / Reflection / Verification		 Control How should the agent decide what to do?
Context & Memory
Context Management / Retrieval / Compression / Memory		 State What should the agent remember?
Tools & Actions
Tool Calling / Discovery / Selection / Code / Browser / Computer		 Action What can the agent do?
ACI & Environment
State / Action Space / Observation / Error / Feedback		 Interface What state, observations, actions, errors, and feedback does the environment expose to the agent?
Skills & Composition
Skills / Workflow / Subagents / Agent-as-a-Tool		 Capability How can reusable capabilities and workflows be represented, retrieved, composed, and reused?
Protocol & Interoperability
MCP / A2A / Agent API / Session Protocol		 Interoperability How do agents/tools/services communicate?
Runtime & Safety
Sandbox / Execution / Permission / Guardrail / Reliability / Observability / Cost		 Runtime & Governance Where and how are actions executed?
How do we keep agents safe, reliable, observable, recoverable, and economical?		 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
Agent
│
├── LLM
│
└── Harness
    │
    ├── 1. Orchestration
    │      Loop / Planning / ReAct / Reflection / Verification
    │
    ├── 2. Context & Memory
    │      Context Management / Retrieval / Compression / Memory
    │
    ├── 3. Tools & Actions
    │      Tool Calling / Discovery / Selection / Code / Browser / Computer
    │
    ├── 4. ACI & Environment
    │      State / Action Space / Observation / Error / Feedback
    │
    ├── 5. Skills & Composition
    │      Skills / Workflow / Subagents / Agent-as-a-Tool
    │
    ├── 6. Protocol & Interoperability
    │      MCP / A2A / Agent API / Session Protocol
    │
    └── 7. Runtime & Safety
           Sandbox / Execution / Permission / Guardrail /
           Reliability / Observability / Cost

3.2.1 Orchestration / Control

Q: How should the Agent decide what to do?

1
2
3
4
5
6
Orchestration
├── Loop
├── Planning
├── ReAct
├── Reflection
└── Verification

Earlier Agent(Classic ReAct):

1
LLM → Tool → LLM → Tool

Nowadays Agent:

1
Plan → Act → Verify → Reflect → Re-plan → Act

From simple ReAct Loop to a Loop with planning / verification / recovery.

  • PlanGEN (EMNLP 2025): This work combines planning and iterative verification

  • ReflAct (EMNLP 2025): The traditional ReAct sturcure will accumulate little errors which will make agent drift from correct path.(ALFWorld + 27.7% than ReAct)

  • 1
    2
    3
    4
    5
    6
    7
    Current State
          ↕
    Goal State
          ↓
    Reflection
          ↓
    Action

  • Metagent-P (ACL 2025)

3.2.2 Context & Memory / State

Q: What should the Agent know at the current moment, and what past information should be retained ?

1
2
3
4
5
6
7
Orchestration
    ↓
“What should I do?”

Context & Memory
    ↓
“What should I know?”

The Context & Memory is not just ‘Chat Log’. It store the Agent State.

1
2
3
4
5
6
7
8
9
Current Context
+
Working Memory
+
Long-term Memory
+
Retrieved Experience
+
Compressed History

The ==tradition memory system==:

1
2
3
Memory
=
Vector DB + Embedding + Retrieval

Nowadays:

1
2
3
4
5
6
Agent decides:
    store?
    update?
    delete?
    retrieve?
    summarize?

Memory Management will be considered into Agent Policy.

  • Memory OS of AI Agent, (EMNLP 2025)

    This work design the Memory System like operating system hierarchy.

    1
    2
    3
    4
    5
    Short-term
        ↓
    Mid-term
        ↓
    Long-term

  • Coarse-to-Fine Grounded Memory, (EMNLP 2025)

    This work use memory to replan. Memory is used to store useful things.

    1
    2
    3
    4
    5
    6
    7
    Past Experience
          ↓
    Grounded Memory
          ↓
    Current Situation
          ↓
    Planning

  • Agentic Memory, ACL 2026

  • Memory-R1，ACL 2026

One problem:

How Memory Management Impacts LLM Agents, ACL 2026:

The agent will clearly exhibit “experience-following”: after retrieving historical experience from similar tasks, it is easy to reproduce similar behaviors. So if there exists wrong memory, the Agent are inclined to repeat the wrong operations.

3.2.3 Tools & Actions / Action

What can Agent do ?

The tradition actions system is simply call tools like:

1
2
3
4
5
6
7
8
Search
Code
Database
Browser
Filesystem
Shell
Computer
API

The new tendency of actions area is:

1
2
3
4
5
Tool Discovery
Tool Selection
Tool Composition
Tool Creation
Computer Use

  • LLM Agents Making Agent Tools，ACL 2025

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    Paper + GitHub Code
            ↓
    Agent
            ↓
    Install dependencies
            ↓
    Generate tool
            ↓
    Execute
            ↓
    Debug / Self-correct

    This paper makes Agent generate create tools itself.

  • Adaptive Tool Use，ACL 2025

    More tools do not necessarily mean a better Agent. If one tool frequently returns errors, the Agent should realize that.

  • ToolScope，ACL 2026

    In large tool ecosystem, there exists a problem:

    1
    2
    3
    4
    5
    6
    7
    Too Many Tools 
    + 
    Repeated Tools 
    + 
    Tedious Schema
    +
    Trouble Choosing

    Solution:

    1
    2
    3
    4
    5
    Tool Merging
    +
    Tool Retrieval
    +
    Context-aware Filtering

3.2.4 ACI & Environment / Interface

This part is to solve the problem: What form should the external world take for the agent, and how should the agent act upon it?

A common API-agent interface takes a form like:

1
search(query)

A computer-use agent instead exposes a human-like interaction loop:

1
2
3
4
5
6
7
8
9
Screenshot
  ↓
LLM
  ↓
click(x,y)
  ↓
Screenshot
  ↓
...

Advantage: Developer need no more to design tools for each job.

  • OS Agents Survey，ACL 2025:

    The paper defines OS Agent that interact with:

    1
    2
    3
    4
    5
    GUI
    CLI
    Web
    Mobile
    Desktop

  • OSWorld-Human, ICML 2025:

    To finish a certain task, the Agent takes more steps/operations than human. This work

    1
    2
    3
    4
    5
    Agent Benchmark
         ↓
    Can you finish?
         +
    Can you finish efficiently?

3.2.5 Skills & Composition / Capability

How can we extract complex, repeatable capabilities from a specific agent and turn them into reusable, composable capabilities?

1
2
Tool = an executable capability or interface for an action
Skill = reusable procedural knowledge or capability that can guide or execute a class of tasks.
1
2
Tools → Atomic Capability
Skills → procedural capability

Tendency: Skill are changing to software artifact from prompt.

The traditional Skill is a text:

1
2
3
4
5
To finish ... task, you should:
1. ...
2. ...
3. ...
4. ...

It’s a stable text describing a fixed workflow.

  • Agent Skills，Anthropic 2025

    Skill is no more a huge text file. It’s new structure:

    1
    2
    3
    4
    Skill/
    ├── instructions
    ├── scripts
    └── resources

    e.g:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    pdf-to-excel/
    ├── SKILL.md
    ├── scripts/
    │   ├── extract_table.py
    │   ├── clean_data.py
    │   └── validate.py
    └── resources/
        ├── excel_template.xlsx
        └── examples/

    There’s a small Skill metadata like tool definitions will be read by Agent when system starts:

    1
    2
    3
    4
    5
    6
    7
    Available Skill:

    Name:
    pdf-to-excel

    Description:
    Extract structured tables from PDFs and produce validated Excel files.

    When handling a certain task:

    1
    2
    3
    4
    5
    Task
    ↓
    Skill Retrieval
    ↓
    Load Relative Skill, e.g. pdf-to-excel/SKILL.md

    In Skill.md, there may exists content like:

    1
    2
    3
    4
    5
    6
    7
    ...

    Run:

    python scripts/extract_table.py report.pdf

    ...

    In fact the scripts are tools for skills.

    GitHub - anthropics/skills: Public repository for Agent Skills · GitHub

  • SkillWeaver, arXiv:2504.07079

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    Web Task
       ↓
    Agent Exploration
       ↓
    Successful Trajectory
       ↓
    Skill Discovery
       ↓
    Skill Synthesis
       ↓
    Skill Practice / Debugging
       ↓
    Reusable Skill API
       ↓
    Skill Library
       ↓
    Future Tasks

  • SkillRL (arXiv:2602.08234)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    Experience
         ↓
    Skill Distillation
         ↓
    Hierarchical SkillBank
         ↓
    Retrieval
         ↓
    RL Training
         ↓
    Policy Improvement
         ↓
    New Experience
         ↓
    SkillBank Evolution

  • SkillCraft, (arXiv:2603.00718)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    Task A
     ↓
    Learn Skill
     ↓
    Task B
     ↓
    Reuse Skill
     ↓
    Task C
     ↓
    Reuse / Compose Skill

  • Generative Skill Composition for LLM Agents, (arXiv:2606.32025)

    How to compose skills.

  • Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward, (arXiv:2602.12430)

3.2.6 Protocol & Interoperability / Interoperability

Solve the problems of communication between Agents, Tools, Data Source and Services.

  • MCP (Model Context Protocol):

    1
    2
    3
    Agent
     ↕
    Tools / Data / Services

  • A2A (Agent to Agent):

    Different Agents should have the capability to work together.

    1
    Agent A ↔ Agent B

3.2.7 Runtime & Safety —— Runtime & Governance

Make the Agent actually run in a controlled, secure, observable, and recoverable environment.

Traditional safety check:

1
2
3
4
5
Prompt
 ↓
Output
 ↓
Safety Check

Agent workflow:

1
2
3
4
5
6
7
8
9
Prompt
 ↓
Agent
 ↓
Tool Call
 ↓
External Data
 ↓
Side Effect

So we should check safety of not only prompt and output but also Action, Tool Call, State Transition and Permission.

  • Task Shield，ACL 2025

    Does this Action is really relative to User’s task ?

  • IPIGuard，EMNLP 2025

    Create a Tool Dependency Graph, and limit the tool call by the dependency graph.

3.2.8 Conclusion

Field 2025 Tendency in 2026
Orchestration ReAct、Planning、Reflection Adaptive / verified / long-horizon control
Context & Memory RAG、hierarchical memory、context compression Agentic memory、learned memory management
Tools & Actions Tool calling、tool selection Tool discovery、tool management、tool creation、computer use
ACI & Environment Browser / GUI agents Efficient action spaces、long-horizon computer interaction、安全 ACI
Skills & Composition Skills、workflows、multi-agent Reusable capability、skill lifecycle、agent composition
Protocol & Interoperability MCP、A2A large ecosystem、discovery、identity、security、task protocols
Runtime & Safety Sandbox、guardrails、tracing Durable runtime、containment、permission、agent security、cost/reliability

3.3 Benchmark

3.3.1 What Does an Agent Benchmark Evaluate?

Agent benchmarks evaluate more than final-answer correctness. Depending on the benchmark, they may measure task success, long-horizon execution, tool use, environment interaction, efficiency, reliability, safety, and generalization.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Agent Evaluation
│
├── Effectiveness
│   └── Did the task succeed?
│
├── Efficiency
│   ├── Steps / Actions
│   ├── Tokens
│   ├── Time
│   └── Cost
│
├── Reliability
│   └── Can the agent succeed repeatedly?
│
├── Safety
│   └── Did it violate constraints or permissions?
│
└── Generalization
    └── Can it handle unseen tasks and environments?
Category What it Evaluates Representative Benchmarks Typical Agent Capability
Generalist Agent General-purpose agents that combine reasoning, search, tools, files, and multimodal capabilities GAIA General reasoning, tool use, information gathering, multi-step problem solving
Long-Horizon / Real-World Agent Agents performing complex, multi-step tasks in realistic environments TheAgentCompany, AgencyBench Planning, sustained interaction, task decomposition, long-horizon execution
Computer-Use Agent Agents operating computers through GUI, applications, filesystem, and terminal OSWorld, OSWorld-Human GUI grounding, computer control, application usage, visual interaction
Web / Browser Agent Agents interacting with websites and browser-based environments WebArena, Mind2Web Web navigation, search, clicking, form filling, browser interaction
Software Engineering Agent Agents solving real-world software engineering tasks in repositories SWE-bench, SWE-bench Verified Code understanding, debugging, code editing, testing, repository-level reasoning
Tool-Using Agent Agents that select, invoke, and coordinate external tools/APIs BFCL, τ-bench / τ²-bench Tool selection, function calling, argument generation, multi-step tool use
Deep Research / Research Agent Agents conducting open-ended research over external information sources BrowseComp, BrowseComp-Plus, MLR-Bench Web search, information synthesis, source verification, long-horizon research
Multi-Agent Multiple agents coordinating to solve tasks collaboratively MultiAgentBench, AgentVerse-related benchmarks Communication, coordination, delegation, negotiation, collaboration
Agent Safety / Security Agents operating under safety, permission, security, and adversarial constraints Agent safety / tool-use security benchmarks Policy compliance, permission management, robustness, safe tool use
Classic Agent Environments Foundational controlled environments for interactive agent research ALFWorld, WebShop, Mind2Web Planning, instruction following, environment interaction, action execution
Benchmark Methodology Reliability and validity of agent evaluation itself BrowseComp-Plus and related benchmark methodology work Contamination resistance, reproducibility, verifier quality, efficiency, cost, generalization

3.3.2.1 Generalist Agent Benchmark

Generalist benchmarks evaluate whether an agent can combine reasoning, tool use, information gathering, and multi-step execution across heterogeneous tasks. ==GAIA== is a representative benchmark in this category.

The agent workflow nowadays can look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Goal
 ↓
Planning
 ↓
Tool calls
 ↓
Environment interaction
 ↓
State changes
 ↓
Observation
 ↓
Recovery
 ↓
More tool calls
 ↓
Final artifact

Traditional benchmarks often judge if the final answer is correct.

This research direction is about:

1
Can the Agent still complete the task after taking dozens, hundreds, or even thousands of actions?

3.3.2.2 Long-Horizon / Real-World Agent Benchmark

These benchmarks emphasize sustained execution over many interacting steps, often across realistic workplace or application environments.

  • TheAgentCompany, NeurIPS 2025

    This benchmark builds a company system:

    1
    2
    3
    4
    5
    6
    7
    8
    Company Environment
    │
    ├── Browser
    ├── Code repository
    ├── Terminal
    ├── Internal websites
    ├── Communication
    └── Coworkers

  • AgencyBench, ACL 2026

3.3.2.3 Computer-Use / GUI Agent Benchmark

1
Can an Agent really operate a computer instead of merely describing the operations?

  • OSWorld, NeurIPS 2024.

    The most important Computer-Use Benchmark.

    1
    2
    3
    4
    5
    6
    7
    8
    Agent
      │
      ├── Desktop OS Environment
      ├── Browser
      ├── Office Applications
      ├── File System
      ├── Terminal
      └── Desktop Applications

  • OSWorld-Human, ICML 2025 Workshop on Computer-Use Agents

    Calculate efficiency of Agent.

    Even when they successfully complete the task, top agents still require 1.4–2.7× as many steps as humans.

  • OSWorld 2.0

  • CRAB, ACL 2025

    Cross-Environment Agent Benchmark for Multimodal Language Model

    Thought: Benchmark should not be bound by certain environment.

3.3.2.4 Tool Use / Function Calling / MCP Benchmark

Tools Evaluation Process:

1
2
3
4
5
6
7
8
9
Function Calling
      ↓
Tool Use
      ↓
Stateful Tool Use
      ↓
Agentic Tool Use
      ↓
MCP / Interoperability

  • BFCL, ICML 2025

    BFCL v4

    It judges:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    Single-turn
       ↓
    Multiple functions
       ↓
    Parallel calls
       ↓
    Multi-turn
       ↓
    Memory
       ↓
    Dynamic decision making
       ↓
    Long-horizon agentic behavior

  • ToolSandbox, NAACL 2025 Findings

  • τ-bench, ICLR 2025

  • MCP-Bench, ICLR 2026

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    Tool Discovery
          ↓
    Tool Selection
          ↓
    Parameter Control
          ↓
    Multi-hop Planning
          ↓
    Cross-tool Coordination
          ↓
    Task Completion

3.3.2.5 Software Engineering Agent Benchmark

  • SWE-bench, ICLR 2024 Oral

  • SWE-bench Multimodal, ICLR 2025

  • SWE-Bench Pro, ICLR 2026 submission.

    For Long-Horizon software engineering.

There’s a problem: ==Contamination==.

The paper finds that some models’ performance on SWE-bench Verified may be affected by training data contamination:

Does SWE-Bench-Verified Test Agent Ability or Model Memory?

3.3.2.6 Web / Browser Agent Benchmark

The core of Web Agent is not ‘Search’ but: The Agent observes webpages, plans, clicks, types, navigates, calls APIs, and ultimately completes tasks in real-world web environments.

Classic workflow:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
User Goal
   ↓
Browser
   ↓
Observe webpage
   ↓
Plan
   ↓
Click / Type / Search
   ↓
New webpage
   ↓
Reason
   ↓
...
   ↓
Task Success
  • WebArena, ACL 2024
  • TurkingBench, NAACL 2025
  • X-WebAgentBench, ACL 2025 Findings

3.3.2.7 Deep Research / Research Agent Benchmark

Normal Web Research:

1
2
3
4
5
6
7
8
9
Web Agent

Goal
 ↓
Browse
 ↓
Action
 ↓
Done

Deep Research:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Research Question
 ↓
Decompose
 ↓
Search
 ↓
Read
 ↓
Cross-check
 ↓
Identify information gaps
 ↓
Search again
 ↓
Synthesize
 ↓
Cite sources
 ↓
Report

  • BrowseComp

  • BrowseComp-Plus, ACL 2026

  • MLR-Bench, NeurIPS 2025:

    1
    2
    3
    4
    5
    Web Research
          ↓
    Deep Research
          ↓
    Scientific Research Agent

    Can AI do research itself?

3.3.2.8 Multi-Agent Benchmark

Multi-agent benchmarks evaluate coordination, communication, task decomposition, role assignment, negotiation, and collaborative problem solving among multiple agents. This is a distinct evaluation axis from single-agent capability.

3.3.2.9 Agent Safety / Security Benchmark

Agent safety benchmarks evaluate whether agents remain within task, permission, and security constraints while interacting with tools, external data, and stateful environments. Important dimensions include prompt injection, unsafe tool use, excessive permissions, policy violations, and recovery from adversarial conditions.

3.3.3 Benchmark Methodology

Benchmark design itself is an important research direction. A useful benchmark should measure the intended capability without being dominated by contamination, brittle verifiers, uncontrolled environment changes, or hidden implementation details.

Important questions include:

1
2
3
4
5
6
7
8
9
10
11
Benchmark Quality
│
├── Contamination / Data Leakage
├── Reproducibility
├── Dynamic vs. Fixed Environment
├── Verifier Reliability
├── Difficulty Calibration
├── Human Baselines
├── Efficiency / Cost
├── Robustness
└── Agent-vs-Model Disentanglement

BrowseComp-Plus is a representative example of this direction, emphasizing controlled and reproducible evaluation for deep-research agents.

3.3.4.1 SWE-bench / SWE-bench Verified

3.3.4.2 GAIA

3.3.4.3 WebArena

3.3.4.4 OSWorld Verified / OSWorld 2.0

3.3.4.5 AgentBench

AgentBench is a multi-environment benchmark suite, rather than a single environment-specific benchmark. It includes:

1
2
3
4
5
6
7
8
9
10
AgentBench
│
├── OS
├── Database
├── Knowledge Graph
├── Digital Card Game
├── Lateral Thinking Puzzle
├── ALFWorld
├── WebShop
└── Mind2Web

Classic Agent Environments

ALFWorld, WebShop, and Mind2Web have an important historical role in LLM-agent research and should not be treated as merely implementation details of AgentBench. They are independently used environments/benchmarks that also appear within the AgentBench suite.

  • ALFWorld — interactive household tasks in a text-based embodied environment.
  • WebShop — simulated e-commerce tasks involving search, browsing, product selection, and purchase.
  • Mind2Web — web-agent tasks grounded in real-world websites and interaction trajectories.

This distinction avoids mixing a benchmark suite (AgentBench) with individual environments (ALFWorld/WebShop/Mind2Web) at the same conceptual level.