Paper Summary – OSWorld-Human
1. Introduction
Core Problem:
- The agent can complete a real work, but the it takes too much time. The action latency is too high.
Quantify the problem:
- This work proposes new metrics to benchmark CUA’s work efficiency.
Most time-cost step:
- The planning step accounts for more than half, sometimes close to 75% of the total task latency.
- The judging (GTA1) and reflection (S2) steps account for the second biggest time sink, accounting for 22.5% and 33.6% of total task time, respectively.
Agent Mainly Used:
- Agent S2
- GTA1
2. Methodology
Collect detailed timing and token traces.
| Agent | Max Step | VLM | Test Set | Grounding Model | Inference Engine |
|---|---|---|---|---|---|
| Agent S2 | 50 | GPT-4.1 | 39 Tasks of OSWorld | UI-TARS-7B-DPO | NVIDIA A6000 |
| GTA1 | 100 | o3 | 39 Tasks of OSWorld | GTA1-7B | NVIDIA A6000 |