Paper Summary – Sample-Efficient Learning from Agent Experience

Posted on Edited on In Views:
Paper Summary of "Sample-Efficient Learning from Agent Experience".

1. In-Context Learning (ICL)

In-Context Learning means that:

When agent is working for solving a job, the environment will generate lots of information including error / success info. Nowadays agent is a multi-turn type tool, it will collect the environment information, package them will other prompts and send to LLM together.

It will give LLM extra interaction info, and LLM could perform better.

But it has a problem:

  • The environment information is not persistent: if you start a new task, the previous environment will be cleaned and the improvement disappears once the context is removed.

So if there is a way to persist the context into model weights?

2. SFT with Context

Use the environment information to do SFT is barely effective because most information is useless which shows the error information instead of correct information.

3. Context Distillation

Context Distillation provides a natural mechanism for consolidating these gains into model weights.

3.1 Context Distillation Goal

Loss Function:

LCD(θ) = DKL(pθ0(⋅|x, c) ∥ pθ(⋅|x))

3.2 Interactive Agent Tasks

Definition of the trajectory of Agent at timestep t (O for Observation, a for Action):

ht = (o0, a0, …, ot)

ht is input of Agent’s decision at timestep ht + 1 , e.g.

h₀ = (o₀)

h₁ = (o₀, a₀, o₁)

h₂ = (o₀, a₀, o₁, a₁, o₂)

h₃ = (o₀, a₀, o₁, a₁, o₂, a₂, o₃)

τ is the full trajectory of a Agent:

τ = (o0, a0, …, aT − 1, oT)

The Agent’s trajectory probability distribution with parameters θ and environment M :

$p_\theta^M(\tau) = p_M(o_0) \prod_{t=0}^{T-1} \pi_\theta(a_t \mid h_t) \cdot p_M(o_{t+1} \mid h_t, a_t)$

pθM(τ) = probability of acquiring trajectory τ with strategy θ and environment M .

pM(o0) = probability of the initial environment

πθ(at ∣ ht) = probability of choosing action at after seeing the history ht (Decided by LLM)

pM(ot + 1 ∣ ht, at) = probability of returning ot + 1 by Environment after Agent taking at (Decided by Env)

$\prod_{t=0}^{T-1}$ = accumulative multiplication

3.3 Experience Distillation

3.3.1 Naive Context Distillation

Definition Experience Distillation as a Trajectory-Level Context Distillation:

LCD(θ; M, τexp) = DKL(pθ0M(τ′|τexp) ∥ pθM(τ′))

  • pθ0M(τ′ ∣ τexp) = teacher’s probability distribution of acquiring trajectory τ′ with strategy θ0, environment M and experience τexp
  • pθM(τ′) = student’s probability distribution of acquiring trajectory τ′ with strategy θ, environment M

Simplification :

$$ \begin{aligned} L_{CD}(\theta; M, \tau^{\text{exp}}) = D_{KL}\big(p_{\theta_0}^M(\tau' \mid \tau^{\text{exp}}) \parallel p_\theta^M(\tau')\big) \\ = \mathbb{E}{\tau' \sim p{\theta_0}^M(\cdot \mid \tau^{\text{exp}})}\left[\sum_t D_{KL}\big(\pi_{\theta_0}(\cdot \mid h_t, \tau^{\text{exp}}) \parallel \pi_\theta(\cdot \mid h_t)\big)\right] \end{aligned} $$

Problem:

τ′ needs teacher model run another time ➡ which is not sample-efficient

3.3.2 Improvement

  1. Use World Model to replace the real environment (Aborted because of the error accumulation)

  2. Branched Rollouts:

    • Real Traj:

      o0 → a0 → o1 → a1 → o2 → a2 → o3 → a3 → ...

      ​ ⬆️

      ​ branch point

      ​ ⬇️

            Simulate $k$ step : $a'_2 → o'_3 → a'_3 ...$

    • Not fabricate a new traj

    • Simulate k step

    • The smaller k is, the smaller the accumulated model error will be

  3. k = 1

    If k = 1, at htexp , teacher only make a Action decision at′ and stop

    o0 → a0 → o1 → a1 → o2 → a2 → o3 → a3 → ...

    ​ ⬆️

    ​ teacher make one decision:

    ​ ⬇️

    ​ a2′

    ​

  4. Advantages of k = 1

    • only need teacher at′
    • do not need World Model to predict ot′
    • htexp is existing, τexp is also existing

    World Model is no more needed and no environment interaction added.

  5. Simplification

    $$ L_{EPD}(\theta; \tau^{\text{exp}}) = -\sum_{t=0}^{T-1} \mathbb{E}{a't \sim \pi{\theta_0}(\cdot \mid h_t^{\text{exp}}, \tau^{\text{exp}})}\big[\log \pi\theta(a'_t \mid h_t^{\text{exp}})\big] $$

3.4 Teacher & Student

Teacher and Student are the same model:

Teacher: θ0(frozen) + context

Student: θ

4. Experience Distillation (EPD)

4.1 Example

trajectory(Base-0):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
t=0: o₀ = "请修复 Geometry Filter 不刷新的 bug"

t=1: a₀ = 打开 FilterComponent.js
      o₁ = 看到 handleClick 的代码

t=2: a₁ = 修改了 if(value) 这一行
      o₂ = 跑测试 → 失败，marker 消失

t=3: a₂ = 打开 SliderControl.jsx
      o₃ = 看到 x 按钮的 disabled 逻辑

t=4: a₃ = 修改了按钮重置
      o₄ = 跑测试 → 通过一部分，地图刷新了但位置不对

t=5: a₄ = 打开 QueryBuilder.js
      o₅ = 看到 where 子句过滤了 falsy 值

t=6: a₅ = 修改查询管道
      o₆ = 跑测试 → ✅ 全部通过

t=7: a₆ = 打开另一个不相关的文件
      o₇ = 翻了翻，没啥用

t=8: a₇ = 写了一行多余的 log
      o₈ = 终端输出一堆废话

t=9: a₈ = 删掉刚才的 log
      o₉ = 代码干净了

t=10: o₁₀ = 最终状态

Pre-process(g(τexp)):

Excerpt:

1
"这个任务的 bug 涉及三个组件：FilterComponent 的 handleClick、SliderControl 的 clearFilter 按钮、QueryBuilder 的 where 子句。单独改任何一个都不行，必须三个一起改。t=2 测试失败就是因为只改了一个。t=4 位置不对是因为还缺查询管道的修复。t=7-9 的操作与 bug 无关。

Teacher label 9 at′ :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
t=0: 输入 h₀ᵉˣᵖ = (o₀: "请修复...")
     教师 → a'₀ = "打开 FilterComponent.js，先定位入口"

t=1: 输入 h₁ᵉˣᵖ = (o₀, a₀: 打开 FilterComponent, o₁: 看到代码)
     教师 → a'₁ = "修改 handleClick 的条件判断，并添加 null 的显式重置"
            （比原始 a₁ 多了一步 null 处理）

t=2: 输入 h₂ᵉˣᵖ = (... a₁: 原始修改, o₂: 测试失败)
     教师 → a'₂ = "不要去翻别的文件，直接打开 SliderControl.jsx"
            （原始 a₂ 是看测试失败后才去的；
              教师提前就知道，而且跳过了"懵了乱翻"的步骤）

t=3: 输入 h₃ᵉˣᵖ = (... a₂: 打开 SliderControl, o₃: 看到按钮逻辑)
     教师 → a'₃ = "修复 clearFilter 后按钮的 disabled 状态重置"
            （和原始 a₃ 差不多，但推理链更完整）

t=4: 输入 h₄ᵉˣᵖ = (... a₃: 改按钮, o₄: 测试部分通过)
     教师 → a'₄ = "直接打开 QueryBuilder.js，还剩最后一个组件"
            （跳过原始 a₄ 中间的犹豫和乱翻）

t=5: 输入 h₅ᵉˣᵖ = (... a₄: 打开 QueryBuilder, o₅: 看到 where 子句)
     教师 → a'₅ = "修改 where 子句让 falsy 值也能触发刷新"
            （和原始 a₅ 类似）

t=6: 输入 h₆ᵉˣᵖ = (... a₅: 改完查询管道, o₆: ✅ 全过)
     教师 → a'₆ = "任务已完成，不需要再做任何修改"
            （原始 a₆ 又去翻了一个不相关的文件——多余动作）

t=7: 输入 h₇ᵉˣᵖ = (... a₆: 乱翻文件, o₇: 没啥用)
     教师 → a'₇ = "这个文件跟 bug 无关，不需要在这里浪费时间"

t=8: 输入 h₈ᵉˣᵖ = (... a₇: 加 log, o₈: 终端输出)
     教师 → a'₈ = "不需要加 log，之前的修复已经完整了"

Train Dataset (9 samples):

(htexp, at′)

1
2
3
4
5
6
(o₀): "a'"
(o₀, a₀, o₁): "a'₁"
(o₀, a₀, o₁, a₁, o₂): "a'2"
(o₀, a₀, o₁, a₁, o₂, a₂, o₃): "a'3"
...
(............full history...........): "a'₈"

749 SWE tasks ➡ 45,301 Samples

5. EPD’s Performance

5.1 Compare with ICL and SFT

SWE (749 Tasks) TaleSuite (6 Tasks)
Zero-shot 5.3% 18.5
ICL 76.4% 45.6
SFT 8.0% 17.8
EPD 51.4% 43.8

Use GICL to evaluate “How much of the gains of by ICL has been retained”

$$G_{ICL} = \frac{\text{Score} - \text{Zero-shot Score}}{\text{ICL Score} - \text{Zero-shot Score}} \times 100 \% $$

SWE (749 Tasks) TaleSuite (6 Tasks)
GICLEPD 64.8% 93.4%
GICLSFT 3.8% −2.6%

5.2 Compare with RL

EPD can save a lot of samples compared with RL.

SWE/TaleSuite Environment Samples per Task EPD Fewer Environment Samples
ICL + EPD(SWE) 51.4% pass@1 52.5 -
PPO(SWE) 17.7% pass@1 504.9 9.6×
GRPO (TaleSuite) 29.9 448 57.2×

6. Conclusion

6.1 Conclusion

Core Innovations

  1. A New Problem: Experience Distillation

Before this paper, there were three separate lines of work:

  • ICL: Highly sample-efficient, but the knowledge is temporary — once the context is removed, the capability vanishes
  • Context Distillation: Can consolidate contextual knowledge into model weights, but requires the student and teacher to run in the same environment — in agent settings, this means additional environment interactions
  • RL: Can permanently consolidate knowledge, but sample efficiency is abysmal

The paper defines the gap at the intersection of these three as a new problem: how to distill what an agent learns via ICL from its own experience into model weights, without incurring additional environment interactions.

  1. One Elegant Simplification: k = 1 Branched Rollout

This is the most ingenious step in the entire paper.

Trajectory-level KL → simplified to per-step policy KL (Equation 1) → still requires environment rollouts → replace with a world model? Error compounds → branched rollouts? Push to the limit k = 1 → world model gone, environment interaction gone (Equation 2).

It transforms “have the teacher re-run the entire trajectory” into “have the teacher make a single revised decision at each already-recorded history point.” The cost is an approximation (substituting htexp for ht′); the gain is zero additional environment interaction — and experiments show the approximation works perfectly well in practice.

  1. Three Engineering Complements
  • Experience Preprocessing: Solves the problem of raw trajectories being too long and noisy
  • Enhanced Teacher Reasoning: Makes the teacher write out a reasoning chain, distilling the “why” alongside the “what” — nearly doubles performance
  • Branch Packing: Packs independent samples into sequences, boosting training efficiency by over 10× with no performance loss (slight improvement, actually)
  1. Counterintuitive Empirical Findings

┌────────────────────────────────┬─────────────────────────────────────────────────────┐ │ Intuition │ Reality │ ├────────────────────────────────┼─────────────────────────────────────────────────────┤ │ A world model should help │ Removing it helps more — error compounding is fatal │ ├────────────────────────────────┼─────────────────────────────────────────────────────┤ │ SFT on raw trajectories should │ Almost zero gain (3.8%) — raw actions are too │ │ do something │ low-quality │ ├────────────────────────────────┼─────────────────────────────────────────────────────┤ │ Student-sampled Reverse KL │ Only 9.1% — without experience context, the student │ │ should work │ rarely samples good trajectories │ ├────────────────────────────────┼─────────────────────────────────────────────────────┤ │ Longer rollouts are better │ Shorter is better — k = 1 is optimal │ └────────────────────────────────┴─────────────────────────────────────────────────────┘

These results demonstrate that in the agent experience setting, many classical intuitions are wrong.

  1. A New Paradigm

Rather than “ICL vs RL vs Distillation — pick one,” the paper chains them into a pipeline:

Few-shot trial-and-error (ICL, temporary learning) → Periodic distillation (EPD, consolidation) → Loop (continual distillation)

This combines the sample efficiency of ICL with the persistence of weight updates. The continual distillation result — 7.1 → 47.0 over 5 rounds — shows the loop genuinely works.

Assessment

Strengths

Well-defined and important problem. As agent tasks grow longer and environment interactions become more expensive (real laboratories, human feedback, enterprise codebases), the need for “learn more, interact less” will only intensify. The paper identifies a valuable gap at the intersection of ICL, distillation, and RL.

Clean and elegant method. The derivation that pushes k = 1 to the limit and eliminates the world model is beautiful — the key move is subtraction, not added complexity. Good methods are often simple.

Thorough, comprehensive experiments. Two domains, comparisons against SFT and RL, ablation of every component, OOD generalization, continual learning, qualitative case studies, diagnostic experiments (ColorButton explains why RL fails) — nearly every question a reader might ask is answered preemptively.

Counterintuitive empirical conclusions are valuable. Especially “world models are harmful” and “SFT is nearly useless” — these findings offer clear guidance for follow-up work.

Limitations

Teacher generation cost remains the bottleneck. The paper acknowledges this candidly (Section 4.10). Although the distillation process itself requires no environment interaction, having the teacher generate revised decisions for all 45,301 interaction turns incurs substantial API costs. Scaling from 1 to 16 teacher-generated samples per turn raises GICL from 31.8% to 64.8% — suggesting more teacher data would yield further gains, but cost is already prohibitive.

64.8% retention, not 100%. On SWE tasks, 35% of the ICL gain is lost. The sources of this gap are not fully characterized — is it approximation error from substituting htexp for ht′? Insufficient teacher reasoning depth? Limited student capacity? No systematic analysis.

Depends on strong ICL capability. The method presupposes that the base model already has sufficient ICL ability to “look at experience and make better decisions.” If the base model’s ICL is weak, the entire pipeline fails. This imposes an implicit requirement on the foundation model.

Limited generalization. On OOD tasks, EPD improves from 4.62% to 8.84% — a real gain, but the absolute number remains low. What gets distilled appears to be more “task-specific repair strategies” than “general programming ability.”

No theoretical analysis. The paper is purely empirical. How large is the bias introduced by the k = 1 approximation? Under what conditions does the distribution shift between htexp and ht′ seriously harm distillation quality? Theoretical guarantees are absent.

Overall Verdict

▎ A well-defined problem, an elegant method, rigorous experiments, and multiple ▎ counterintuitive findings — an exemplary piece of work that poses a question, offers a ▎ viable solution, and leaves ample room for follow-up. It doesn’t solve everything, but it ▎ opens a new hole worth filling.

6.2 Chinese Edition

  1. 提出了一个新问题：Experience Distillation

以前有三个独立的技术线：

  • ICL：样本效率高，但知识是临时的（上下文一丢就没了）
  • Context Distillation：能把上下文知识固化到权重，但要求学生和教师跑同一个环境——这在 Agent 场景里意味着额外环境交互
  • RL：能永久固化，但样本效率极低

论文把三者之间的真空地带定义为一个新问题：如何在不增加环境交互的前提下，把 Agent 从经验中 ICL 学到的东西蒸馏到权重？

  1. 一招关键化简：k = 1 分支展开

这是整篇论文最精巧的一步。

轨迹级 KL → 化简为每步策略 KL（公式 1）→ 仍需环境 rollout → 世界模型替代？误差累积 → 分支展开？推到极限 k = 1 → 世界模型没了，环境交互也没了（公式 2）。

把”让教师重跑整条轨迹”变成了”让教师在已有历史的每个节点上只重新做一个决策”。代价是一种近似 （用 htexp 替代 ht′），换来的是零额外环境交互——而且实验证明这个近似在实际中完全够用。

  1. 三个工程配套
  • 经验预处理：解决原始轨迹太长太嘈杂的问题
  • 增强教师推理：让教师写推理链，把”为什么”也蒸馏进去，效果翻倍
  • 分支打包：把独立样本打包成序列，训练效率提升 10× 以上，性能不降反升
  1. 实证上的反直觉发现

┌────────────────────────────────────┬──────────────────────────────────────────┐ │ 直觉以为 │ 实际结果 │ ├────────────────────────────────────┼──────────────────────────────────────────┤ │ 有世界模型应该更好 │ 去掉世界模型反而更好，误差累积是致命伤 │ ├────────────────────────────────────┼──────────────────────────────────────────┤ │ SFT 模仿原始轨迹应该有点用 │ 几乎零收益（3.8%），因为原始动作质量太差 │ ├────────────────────────────────────┼──────────────────────────────────────────┤ │ 学生在线采样（Reverse KL）应该也行 │ 仅 9.1%，学生没经验根本采样不到好轨迹 │ ├────────────────────────────────────┼──────────────────────────────────────────┤ │ 完整 rollout 越长越好 │ 越短越好，k = 1 最优 │ └────────────────────────────────────┴──────────────────────────────────────────┘

这些结果说明：在 Agent 经验场景下，很多经典直觉是错的。

  1. 指明了一种新范式

不是”ICL vs RL vs Distillation 三选一”，而是把它们串成流水线：

少量试错（ICL 临时学会）→ 定期蒸馏（EPD 固化）→ 循环往复（持续蒸馏）

兼具 ICL 的样本高效和权重更新的持久性。持续蒸馏 5 轮从 7.1 → 47.0 的结果证明这个循环真的能转起来。

评价

优点

问题定义清晰且重要。 随着 Agent 任务越来越长、环境交互越来越贵（真实实验室、人类反馈、企业 级代码库），“少交互、多学习”的需求只会更强。论文在 ICL、蒸馏、RL 的交叉点上找到了一个有价值的空白。

方法简洁优雅。 k = 1 这一步推到极限、消除世界模型的推导非常漂亮——不是靠堆复杂度，而是靠做减法。好方法往往简单。

实验扎实全面。 两个领域、和 SFT/RL 对比、消融每个组件、OOD 泛化、持续学习、定性案例、诊断实验（ColorButton 解释为什么 RL 失败）——几乎每个读者会问的问题都预先回答了。

反直觉的实证结论有价值。 特别是”世界模型有害”和”SFT 几乎无效”这两个结果，对后续研究有明确的指导意义。

不足和局限

教师生成成本仍是瓶颈。 论文自己坦承了这一点（4.10 节）。虽然蒸馏过程不需要环境交互，但让教师为 45,301 个轮次逐个生成修正决策，API 调用成本很高。从 1 个样本扩展到 16 个样本，GICL 从 31.8% 涨到 64.8%——说明更多教师数据还能提升，但成本已经扛不住了。

保留了 64.8% 而非 100%。 SWE 上丢了 35% 的 ICL 增益。这部分损失的来源不完全清楚——是 htexp 替代 ht′ 的近似误差？还是教师推理链不够长？还是学生容量不够？没有系统分析。

依赖强 ICL 能力。 这个方法的前提是教师模型本身有足够的 ICL 能力，能”看了经验之后做出更好的决策”。如果基础模型的 ICL 能力弱，整条流水线就不工作。这意味着对基础模型有隐性要求。

泛化有限。 OOD 任务上从 4.62% 到 8.84%，确实有提升，但绝对值仍然很低。蒸馏出来的更多是”任务 特定的修复策略”而非”通用的编程能力”。

没有理论分析。 论文是纯实证的。k = 1 近似引入的偏差有多大？在什么条件下 htexp 和 ht′ 之间的分布差异会严重影响蒸馏质量？缺少理论保证。