Paper Summary – CODESKILL

1. Introduction

There used to have 2 ways of skill evolving:

• Sequential Skill Editing: Update/Edit Skill after each incoming trace.
• ReasoningBank-style Retrieval Memories: Distill experience from an agent’s past executions into transferable memory items, store them in an external memory bank, and at inference time retrieve the semantically most relevant items and inject them into the agent’s context for reuse.

The thought of Trace2Skill likes Human Experts: inspect broad traces, abstract recurring patterns, and write compact procedures that are reusable across cases.

2. Trace2Skill

There three stages of Trace2Skill:

• Collect Trajectories
• Propose Trajectory-Level Patches in Parallel
• Consolidate Patches into One Portable Skill

2.1 Skill & Problem Formalization

A skill directory S = (M, ℛ)nowadays mainly consists of two parts:

• The root SKILL.md file, denoted as M
• The auxiliary files including references, scripts, or assets denoted as ℛ.

The root document M stores broadly applicable procedural knowledge, while auxiliary files ℛ provide deterministic tools or lower-frequency details.

Let π_θ be a fixed LLM agent using skill S.

Let 𝒫(S; π_theta, 𝒟) denote the pass rate of that agent on task set 𝒟.

An evolving set 𝒟_evolve, and a disjoint test set 𝒟_test, skill evolution constructs a new skill from evolving-set trajectories without updating θ: S^* = ℰ(S₀, D_evolve; π_θ),   P(S^*; π_θ, D_test) > P(S₀; π_θ, D_test). 

[!NOTE]

The evolving set 𝒟_evolve is to evolve the skill; the 𝒟 itself dose not change.

The work evaluates two initializations for S₀:

• human-expert skill
• LLM-generated draft skill

2.2 Stage 1: Trajectory Generation

This stage uses a Re-Act-Style harness. For each evolving-set task, the fixed agent runs with S₀ and produces a trajectory τ_i containing the query, reasoning/tool-use history, final output, and a binary correctness outcome.

The resulting corpus 𝒯 is split into:

• Failures $T^- $
• Successes $T^+ $

2.2 Stage 2: Parallel Patch Proposal

A group of analyst sub-agents independently proposes skill patches from individual trajectories.

Failures are sent to an error analyst $A^- $ ,successes to a success analyst 𝒜⁺.

And analyst read 𝒮₀ and propose a patch for it, leading to a patch pool. 𝒫 = 𝒫⁻ ∪ 𝒫⁺ The analyst roles are intentionally asymmetric.

• 𝒜⁺ uses a single-pass workflow to identify reusable behavior patterns from successful trajectories.
  1. Read trajectory
  2. Abstract reusable pattern
  3. Read S₀, write a skill patch
• 𝒜⁻ uses a ReAct-style loop that can inspect traces and artifacts, compare outputs against ground truth, and validate candidate fixes before proposing a patch.
  1. Read trajectory + artifacts + ground-truth value
  2. Suppose a reason for the failure
  3. Write a minimal repair fix to verify the failure reason. If the reason is not correct , go back to step 2.
  4. If the reason confirmed, write a skill batch.

In short, for failure trajectory, this stage need analyst to find out the reason.

Here’s a example of the skill patch:

2.3 Stage 3: Patch Consolidation

In stage 2, the work produces many patches storing in a patch pool.

In stage 3, the work needs to consolidate the patches into one coherent update patch p^* and apply it to S₀.

Patches are merged hierarchically for L = ⌈log_Bmerge|P|⌉ levels; at each level ℓ, up to B_merge patches are synthesized into one patch: p^{(ℓ + 1)} = ℳ(π_θ, S₀, {p₁^(ℓ), …, p_Bmerge^(ℓ)}),   ℓ = 0, …, L − 1 where ℳ deduplicates, resolves conflicts, and preserves non-overlapping insights.

π_θ itself serves as trajectory generator, analyst, and merge operator, so no external evolution/teacher model is required.

The final p^∗ is translated into diff-style edits and applied with deterministic guardrails:

• reject edits to missing files
• withhold line-range conflicts
• validate the updated skill format

2.3.1 Patch Consolidation

Now we have:

• Patch Pool: P = P⁺ ∪ P⁻
• Preliminary Skill: S = (M, ℛ)

Now we need to consolidate the patches from patch pool into the S₀.

There are too many patches to be consolidated. If all patches are consolidated at the same time, there may have many conflicts. So the authors consolidate the patches in many times. L = ⌈log_Bmerge|P|⌉

• B_merge: merge batch size, how many patches can be merged in one round.
• L: merge level.
• P: size of patch pool; how many patches need to be merged.

e.g. 323 patches in patch pool, B_merge = 32.

• Level 1: 11 merges, 10 time ℳ (Merge Operator, the LLM itself) merges 32 patches, and 1 time ℳ merge 3 patches; totally generate 11 new patches.
• Level 2: Merge 11 patches into one final patch.