Paper Summary – VISTA

1. Introduction

The VISTA is a brand new visual harness for multimodal LLM.

The current multimodal LLM has an observation limitation. e.g. In OSWorld, the simple agent retains only the last three turns in its context window because of the high cost of image.

In multimodal scenes, if only use one screenshot as the Observation, many information will lose. Even if you compress the pictures, it is still not enough.

The VISTA stores all the image information. Instead of putting them all into the context window. All of them will be stored in Visual Memory. When the Agent need to analyze what happened just now, it will extract relative screenshots and analyze them.

e.g. :

2. VISTA

2.1 Components

There three main components of VISTA:

• Visual observation

  These observations preserve the appearance and spatial relation of objects, allowing the model to use its learned visual priors to interpret and reason about the environment.

• Lossless visual memory

  The harness stores every frame returned by the environment, including intermediate animation frames, in its original form. Each frame is indexed by turn and frame number and remains available throughout the interaction, independently of what is retained in the model’s current context.

• Visual inspection

  Building on the visual memory, we provide the agent with tools to actively inspect visual information in space and across time.

  • In space, it can request an enlarged view of any rectangular region from the original frame, allowing it to examine fine-grained details that may be difficult to perceive in the full image.
  • Across time, it can retrieve and inspect multiple frames from arbitrary earlier turns to revisit past visual evidence, compare states, and replay how an action unfolded through its intermediate animation frames.

Supplements:

1. ARC-AGI-3 is a grid game. The origin version is a text game. The environment is a 64 x 64 grid world. So the textual observation is about 4000 tokens (64 x 64 pixels). But the ViT use the following method to tokenization:

   ① 64×64 original image ──8× nearest-neighbor upsampling──▶ 512×512 pixel image 
   ② 512×512 ──split into 14×14 patches──▶ ≈37×37 ≈ 1369 patches
   ③ each patch ──flatten into a vector + linear projection──▶ 1 token 
   ④ then pooling / resampling (e.g. 2×2) to compress ──▶ ~308 tokens

   However, this tokenization will lose some information, so the original image is stored in the Visual Memory. The agent can inspect full original image at anytime.

   

2. Every frame observation is stored in the visual memory.

   

3. Visual Inspection

2.2 Pipeline

At the start of a turn, the agent receives the current visual state and available actions.

It can also consult two notes that it maintains:

• GUIDE.md, which records its understanding of the game and information that may remain useful across levels
• WORKING.md, which serves as a scratchpad for the current state.

Through inspect, the agent can revisit earlier observations, including intermediate animation frames produced by its previous actions.

It can also use read_pixels to examine pixel values within a selected region.

When its context approaches the limit, the agent writes a concise handoff summary and resumes from the current state in a fresh context.

2.3 Tools

2.3.1 Action

Execute exactly one game action. Make progress in the game.

play(action, x=None, y=None) -> (frame, n_frames, available_actions)
# Execute exactly one game action.
# Returns the final frame. All intermediate frames are archived.

Action Space is defined by the Environment themselves:

ARC-AGI-3: RESET, Action 1, Action 2, ...

Return Value:

Return Item	Meaning	Description
frame	last frame	This frame will be loaded in the context automatically.
n_frames	number of frames	Let LLM know how many frames were generated.
available_actions	available actions set of next turn	Let LLM know the current available actions.

2.3.2 Visual Memory

view = {label, turn, frame=final, region=full}