Paper Summary – Agent S
1. Introduction
1.1 Agent in OSWorld
The agent provided by OSWorld is a simplified Re-Act agent with:
- Observation:
ScreenshotorA11y Tree - Action:
computer_13orpyatuogui - Default context window:
3 turn information
The simple agent has no external task-plan model, skill model, memory model or tool model..
Reference:
OSWorld
1.2 Agent S
This work introduces a brand new multimodal agent framework comparing with the basic one provided by OSWorld.
This work mainly focus on task-planning and relative area.
They propose an Experience-Augmented Hierarchical Planning method.
- To decompose a complex task, the framework uses Online Web Knowledge and past experiences Narrative Memory.
- During the procession of subtask, the agent will retrieve detailed information related to the subtask from Episodic Memory. ( Note: I feel the Episodic Memory likes the skill for specific task nowadays. )
The input:
- Visual Input (Image)
- Accessibility Tree (A11y Tree)
Action Space:
- language-based primitives (Some Atom Action)
2. Agent S
Agent S integrates three main strategies:
- experience-augmented hierarchical planning
- continual update of narrative and episodic memory
- an Agent-Computer Interface (ACI) for precise perception and action on GUIs
