Paper Summary - Agent S2

1. Introduction

Current Agent mainly has 3 core limitations:

1. Struggle to accurately ground textual descriptions of GUI elements to precise pixel-level coordinates.
2. Have difficulty handling long-horizon tasks.
3. Third, most methods rely solely on generalist models. While generalist models offer broad capabilities, they often underperform compared to specialist models in domain-specific subtasks.

Contributions:

1. Mixture of Grounding
2. Proactive Hierarchical Planning

2. Background

2.1 Computer Use Tasks and Benchmarks.

CUA benchmarks:

• OS Tasks:

  • OSWorld: for Linux (Xie et al., 2024).

  • WindowsAgentArena: extended OSWorld to Windows (Bonatti et al., 2024).

  • AndroidWorld: (Rawles et al., 2024a)

• Focus on Visual Grounding in UI environments. :

  • ScreenSpot (ByteDance, 2024)
  • VisualWebBench (Liu et al., 2024)

Recent works:

1. Training monolithic, generalist models on UI interactions and grounding data.
2. Composing multiple models into hierarchical frameworks and creating a pipeline of planning, execution, and grounding. (Agent Structure).

2.2 Monolithic Methods for Computer Use Agents.

Monolithic methods employ a single model to handle all aspects of computer use: planning, execution, and grounding.

Model Requirements:

1. System-2 reasoning, which allows the generation of long-term plans and short-term actions.
2. UI Grounding, which requires locating interactable elements by picking a precise coordinate from screenshots or a specific element from accessibility trees.

Several monolithic models post-trained on GUI tasks have emerged as native agents:

1. Aguvis: Unified pure vision agents for autonomous GUI interaction.
2. UI-TARS: pioneering automated GUI interaction with native agents.
3. Anthropic. Claude computer use.2024. URL https://www.anthropic.com/news/3-5-models-and-computer-use.
4. Cogagent: A visual language model for GUI agents.
5. Anthropic. Claude’s extended thinking. 2025. URL https://www.anthropic.com/research/visible-extended-thinking.

Drawbacks:

1. Fine-tuning for specialization often diminishes broader capabilities.
2. Assembling diverse, large-scale datasets for reasoning and visual grounding is expensive and time-intensive.
3. Relying on a single model may not optimally address the distinct needs of planning, execution, and grounding since these tasks can benefit from different model strengths.

2.3 Hierarchical Methods for Computer Use Agents.

Hierarchical methods decouples cognitive processes of planning, execution, and grounding.

The prior work Agent S:

• High-level:

  • Manager M generates a plan for instruction I.

  • Breaking it into coherent subgoals: I = g₀, g₁, ..., g_N. Each subgoal specifies an intermediate objective of the task.

• At the low-level:

  • A Worker W sequentially performs atomic actions (click, type, etc.) to iteratively complete each subgoal g_i : g_i → a₀, a₁, …, a_t.

Recent Research:

• Knowledge augmentation and continual learning.
  • Agent S: an open agentic framework that uses computers like a human.
  • Os-copilot: Towards generalist computer agents with self-improvement.
• Verification with hierarchical planning.
  • OSCAR: operating system control via state-aware reasoning and re-planning.
• Multi-agent framework for distributing responsibilities to multiple models:
  • Cradle: Empowering foundation agents towards general computer control.

Challenge:

• Grouding
• Planning

3. Agent S2

3.1 Mixture of Grounding

Agent S2 introduces Mixture of Grounding (MoG), which forms the Specialist part of our framework. The Worker W in our framework acts as a gating mechanism and routes each generated action to the correct grounding expert. The grounding expert then generates the pixel-level coordinates.

The grounding workflow:

There are three main types grounding expert models:

• Visual Grounding Expert:
  • This is used to locate a certain pixel area.
• Textual Grounding Expert:
  • Using OCR technique to locate characters
• Structural Grounding Expert:
  • For tabular data, since spreadsheet cells can be stretched and squeezed to arbitrary sizes, and translating the table can change the starting position of the rows and columns, the Agent S2 us takes a dictionary of ⟨ “cell”: “value” ⟩ mapping and programmatically updates the content of the corresponding cells.

3.2 Proactive Hierarchical Planning

Problem:

• The real OS environment always contains many noise coming from background apps and pop-ups. And during the long-horizon tasks, too many apps, windows and observations are involved.

Solution:

• Replan: The task will be replanned at the end of each subgoals.

Comparison:

• Reactive Planning approaches: only replan when subtask fails.
• Proactive Planning: proactively replan.

3.3 Action Space

4. Examples

4.1 One Case using Structural Grounding Expert

• Task:

  • I think the last paragraph is redundant so I want to add strike-through on words in the last paragraph. Can you do this for me?

• Turn/Round 1:

  • Create 3 subgoals.

  • Start subgoal 1:

    • Step 1 Action:

      Triple-click (Call Visual Expert)

      agent.click("The last paragraph starting with 'Each student will be responsible for one discussion leading in class...'", 3, "left")

    • Step 1 Observation: Fail, only one sentence is selected.

    • Step 2 Action: Triple-click (Call Text Expert):

      agent.highlight_text_span(
          "Each student will be responsible for one discussion leading in class during the semester.",
          "The essay writing and presenting comprises 30% of final grade."
      )

    • Step 2 Observation:

      Success.

• Turn/Round 2:

  • Create 2 subgoals. (In fact, they are nearly the same as the ones in round 1).

5. Conclusion

The main work:

• Adding three ways for grounding:
  • Visual Grounding Expert:
    • This is used to locate a certain pixel area.
  • Textual Grounding Expert:
    • Using OCR technique to locate characters
  • Structural Grounding Expert:
    • To select certain area by programmatical methods.
• Replan in time:
  • Replan when subgoal finishes.