Paper Summary – SWE-RL
1. Introduction
This paper introduces SWE-RL, the first approach to scale RL-based LLM reasoning for real-world software engineering.
Leveraging a lightweight rule-based ==reward== (e.g., the similarity score between ground-truth and LLM-generated solutions), SWE-RL enables LLMs to autonomously recover a developer’s reasoning processes and solutions by learning from extensive open-source software evolution data—the record of entire software development cycles, including code snapshots, code changes, and events such as issues and pull requests.
Surprisingly, despite performing RL solely on software evolution data, Llama3-SWE-RL has even emerged with ==generalized reasoning skills==. For example, it shows improved results on five out-of-domain tasks, namely, ==function coding==, ==library use==, ==code reasoning==, ==mathematics==, and ==general language understanding==, whereas a supervised-finetuning baseline even leads to performance degradation on average.
2. Implementation
2.2 Whole Picture
This paper aims to use GitHub existing PRs data to tune mode.
So we should collect enough PRs data as training materials.
Then we should tune model with the dataset, and evaluate the effect.
PR data structure:
1 | main branch |
The author wanna train model with dataset including:
- Code before fixing: merge base (commit C)
- Code after fixing: head commit
- Intervening commits (commit D, E)
Only clone whole repo can get the full commits we need.
2.1 Priliminary PR Data Preparation
Data Source:
- GH Archive records all GitHub events (PR creation, Comment, Merge…) between 2015/01/01 and 2024/08/31;
- Git clone about 4, 600, 000 repos;
2.1.1 Data Aggregation
Aggreagate data from GH Archive and GitHub repo.
Then get about 24,000,000 PRs.
| Field | Source | Description |
|---|---|---|
| pr_id | Github PR number | unique mark |
| base_commit head_commit |
base.sha head.sha |
locate code state |
| ==issue_text== | the issue statement in PR | RL input |
| conversation | all comments, review, commit events of the PR |
records developers’ discussion of the PR |
| ==code_at_merge_base== | code after git checkout base_commit |
RL input |
| ==code_after_fix== | code after git checkout head_commit |
ground-truth |
| intermediate_commits | all commits in git log base_commit..head_commit | record |
| cumulative_patch | git diff base_commit..head_commit | alternative format for reward calculate |
The core fields are:
- issue_text: model input: “Please fix the issue”
- code_at_merge_base: model input: “This is current code”
- code_after_fix: ground-truth: “Correct code is like this”
2.1.2 Exclude SWE-bench Repos
SWE-bench includes 12 repos(pandas、django、flask、sympy、matplotlib…)
The relative PRs should be excluded.
2.1.3 Correct Bias Behavior
In earlier phase, the authot finds out that the model learns to fix every files mentioned in code_at_merge_base field.
e.g.
Train Data:
- Input: issue + file_A.py + file_B.py
- Output: fix file_A.py and file_B.py
- Model learns: All given files should be modified to fix bug.
Evaluate Phase(SWE-bench):
- Input: issue + file_A.py + file_B.py + file_C.py + file_D.py
- Output: Modify all 4 files … Generate extra bugs.
How to ==correct== this bias behavior?
“To mitigate this problem, we ==prompt== Llama-3.1-70B-Instruct to ==generate== a list of relevant but unmodified files given each ==PR description== and the changed files, and include the contents of these files in our final dataset.”
e.g.
original code_at_merge_base:
1
2
3code_at_merge_base = {
"pandas/core/indexes/base.py": "..."
}fixed code_at_merge_base:
1
2
3
4
5
6code_at_merge_base = {
"pandas/core/indexes/base.py": "...",
"pandas/core/indexes/datetimelike.py": "...",
"pandas/core/indexes/datetimes.py": "...",
"pandas/io/parquet.py": "...",
}
2.1.4 Filter PRs
There are too many low-quality and meaningless PRs in the 24,000,000 PRs set. So the author take some steps to filter out high-quality PRs.
Exclude Bot PRs
The author says: “We remove the bot-generated PRs whose title, description, or username contains keywords ‘==[bot]==’, ‘==dependabot==’, ‘==renovate==’, ‘==bump==’, or ‘==automerge==’.”
e.g:
1
2
3
4"Fix PyArrow timestamp index .loc bug" (author: johndoe) ✅ Keep
"Bump django from 3.2 to 4.0" (author: dependabot) ❌ Delete
"Update package versions [bot]" (title includes [bot]) ❌ Delete
"chore(deps): bump requests to 2.31.0" (description includes bump) ❌ DeleteExclude extreme PRs
The core thought is to exclude PRs not for bug repairment.
Two exteme PRs:
- empty PRs: diff with 0 lines ➡️ Model learns nothinng
- extremely large PRs: diff with > 100, 000 lines ➡️ Developer may upload data file by mistake. It’s a noisy PR.
CodeLlama hunk level filter
Hunk means a paragraph of codes, the minimal unit of git diff.
CodeLlama (Rozière et al., 2023) finds out that many hunks in PR is valueless in Model Tuning. So CodeLlama defines a set of rules to mark these valueless hunks.
1
2
3
4
5
6
7
8
9
10
11
12
13← First hunk
if isinstance(key, list) or (
isinstance(key, type(self))
- and key.freq is None
+ and (not hasattr(key, 'freq') or key.freq is None)
):
- keyarr = keyarr._with_freq(None)
+ if hasattr(keyarr, '_with_freq'):
+ keyarr = keyarr._with_freq(None)
← Second hunk
def take(self, indexer):
keyarr = super().take(indexer)Type Example Description Lock file update “msvc-extension”: “^1.3” →
“msvc-extension”: “^1.4”Mechanical version number change; no logic involved Dependency version change numpy>=1.24
numpy>=1.26Same as above Configuration file modification timeout: 30
timeout: 60Parameter tuning; no code reasoning involved Pure whitespace / formatting Indentation adjustments,
newline character normalizationNo semantic change Comment addition/removal Only added a single line # TODODoes not affect program behavior “We then ==removed== any PRs where these filters flagged all code changes.”
If a PR’s all hunks are the flagged hunk type, this PR will be deleted.
Filter with rules:
Filter out final 273 PRs with the following 3 rules:
- PR has least one linked issue
- PR’s issue should describe a bug-fixing request
- PR’s code changes should involve programming files
2.2 RL
The basic framework of reinforcement learning is that:
Agent takes actions in environment;
Environment gives agent rewards;
Agent maxmize the cumulative rewards by adjusting policy.
2.2.1 Sample a Batch of Training Dataset
1 | D_batch = {(issue_1, code_1, gt_1), (issue_2, code_2, gt_2), ..., (issue_32, code_32, gt_32)} |
Each piece of data (PRs) contains :
1 | isssue |
2.2.2 Sampling 16 Rollouts from Each of the 32 Problems in Every Batch
For 1 PR:
1 | prompt = SYSTEM_PROMPT + "This is issue: {issue_text}" + "This is source code: {code_at_merge_base}" |
Ask LLM to generate 16 rollouts:
Rollout is a plain string with format. The rollout is composed of 2 parts: think and solution. Solution uses ==search/replace== format (not unified diff) which is more stable for LLM geneartion.
1 | <think> |
1 | rollout_1:"<think>problem occurs in base.py...</think><solution>### base.py\n<<<<<<< SEARCH\n...</solution>" |
temperature=1 means that during each generation, the
model introduces randomness when selecting the next token.
2.2.3 Calculate Reward
Now we have 32 × 16 = 512 rollouts
2.2.3.1 Input
Input (code_context equals to code_at_merge_base):
1
2
3
4
5
6
7
8
9
10
11
12
13
14code_context = { //code_at_merge_base
"pandas/core/indexes/base.py": """
class Index:
def take(self, indexer):
...
if keyarr.dtype.kind in "mM":
if isinstance(key, list) or (
isinstance(key, type(self))
and key.freq is None # ← bug 在这里
):
keyarr = keyarr._with_freq(None)
return keyarr, indexer
""".strip()
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15oracle_new_content = { //code_after_fix(ground-truth)
"pandas/core/indexes/base.py": """
class Index:
def take(self, indexer):
...
if keyarr.dtype.kind in "mM":
if isinstance(key, list) or (
isinstance(key, type(self))
and (not hasattr(key, "freq") or key.freq is None)
):
if hasattr(keyarr, "_with_freq"):
keyarr = keyarr._with_freq(None)
return keyarr, indexer
""".strip()
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21output = // Rollout
"""
<think>
The bug is that PyArrow-backed Index doesn't have `freq` attribute.
Need to add hasattr check. Also `_with_freq` may not exist.
</think>
<solution>
```python
### pandas/core/indexes/base.py
<<<<<<< SEARCH
and key.freq is None
):
keyarr = keyarr._with_freq(None)
=======
and (not hasattr(key, "freq") or key.freq is None)
):
if hasattr(keyarr, "_with_freq"):
keyarr = keyarr._with_freq(None)
>>>>>>> REPLACE
</solution>
"""1
2
3
4
5
6
7
8
9
10
11
#### 2.2.3.2 Handle Rollout (Extract modified codes)
1. Format check and extract :
```python
for tag in ["<think>", "</think>", "<solution>", "</solution>"]:
if output.count(tag) != 1: # each label must appear 1 time
raise FormatError1
2
3
4thought = output.split("<think>")[1].split("</think>")[0].strip()
answer = output.split("<solution>")[1].split("</solution>")[0].strip()
if len(thought) == 0:
raise FormatError # <think> can't be emptyoutput of this step:
1
2thought = "The bug is that PyArrow-backed Index doesn't have `freq` attribute..."
answer = "```python\n### pandas/core/indexes/base.py\n<<<<<<< SEARCH\n..."If any check goes failed,
reward = -1.0and skip other steps.
Parse search/replace
1
2
3
4pred_search_replaces = parse_search_replace(answer)
SEARCH_REPLACE_REGEX = r"```.*?\n### (.*)\n<<<<<<< SEARCH\n([\s\S]*?)\n=======\n([\s\S]*?)\n>>>>>>>
REPLACE\n```"Perform regular expression matching on the answer. Extract matching group:
1
2
3
4"pandas/core/indexes/base.py"
" and key.freq is None\n ):\n keyarr = keyarr._with_freq(None)"
" and (not hasattr(key, \"freq\") or key.freq is None)\n ):\n if
hasattr(keyarr, \"_with_freq\"):\n keyarr = keyarr._with_freq(None)"Output:
1
2
3
4
5
6
7pred_search_replaces = {
"pandas/core/indexes/base.py": [
("and key.freq is None\n ):\n keyarr = keyarr._with_freq(None)",
"and (not hasattr(key, \"freq\") or key.freq is None)\n ):\n if hasattr(keyarr,
\"_with_freq\"):\n keyarr = keyarr._with_freq(None)")
]
}If not matched,
→ FormatError → reward = -1.0
2.2.3.3 Apply Change (Change code according to search and replace)
reward.py: 262:
1 | pred_new_content = apply_code_change(code_context, pred_search_replaces) |
reward.py: 95-126:
1 | new_content = "\n" + code_context[path] |
2.2.3.4 Calculate Reward by Comparing Text Similarity
After apply_code_change, we can regard the rollout
answer (pred_new_content) is a available answer, we can
evaluate its goodness or badness by comparing the text similarity.
We should compare the diff files’ similarity instead of full codes’. This is because most code of the two files are the same. These ==identical== parts will ==dilute== the similarity. The only truly ==distinctive== parts are those few lines of changes.
reward.py : 263
1 | reward, metadata = calculate_reward(code_context, oracle_new_content, pred_new_content) |
Generate unified diff file for ground-truth answer and rollout answer.
reward.py : 129-152:
1
2oracle_patch = get_normalized_patch(code_context, oracle_new_content)
pred_patch = get_normalized_patch(code_context, pred_new_content)1
2
3
4
5
6
7# oracle_patch: diff of ground-truth file and original file
get_normalized_patch(
code_context, # code_at_merge_base: "and key.freq is None..."
oracle_new_content # code_after_fix: "and (not hasattr(key, 'freq')..."
)
# generation:
# {"base.py": "-and key.freq is None\n+and (not hasattr...\n- keyarr = keyarr._with_freq(None)\n+ if hasattr...\n+ keyarr..."}1
2
3
4
5
6
7# pred_patch: diff of ground-truth file and rollout file
get_normalized_patch(
code_context, # code_at_merge_base: "and key.freq is None..."
pred_new_content # rollout: "and (not hasattr(key, 'freq')..."
)
# generation:
# {"base.py": "-and key.freq is None\n+and (not hasattr...\n- keyarr = keyarr._with_freq(None)\n+ if hasattr...\n+ keyarr..."}Compare similarity of each file
compute_change_similarities:
1
2
3
4
5for path in {"pandas/core/indexes/base.py"}:
pred_change = pred_patch[path]
oracle_change = oracle_patch[path]
similarity = difflib.SequenceMatcher(None, pred_change, oracle_change).ratio()Get average similarity of all similarities
1
reward = sum(map(lambda x: x["similarity"], similarities)) / len(similarities)
2.2.4 Group Normalization
In the last phase, we generate rewards for each rollout.(32 × 16 = 512 Rewards )
The absolute reward value is meaningless:
- PR_1: Simple bug, e.g. fix a function sign, reward = [0.92, 0.88, 0.95, 0.91, …], average reward is approximately 0.9.
- PR_2: Difficult bug, e.g. fix a corss-file bug, reward = [0.12, -1.0, -1.0, 0.08, …], average reward is approximately 0.1.
- A reward value 0.5 in PR_1 should be rewarded, in PR_2 should be punished.
The ==group normalization== turns absolute reward value into relative rank in group.
$$ \text{Advantage}_i = \frac{\text{reward}_i - \text{mean}(r_1+r_2+...+r_G)}{\text{std}(r_1+r_2+...+r_G)} $$
$\text{std} = \sqrt()$
…
Advantages shows if the modification is right.
e.g.
1 | rollout# reward decription |
1 | rollout reward advantage decription |
Cross-group comparation is meaning less.
2.2.5 GRPO
Group Relative Policy Optimization
Before GRPO updating, forzen one current weights πθold
2.2.5.1 Possibility Ratio
For same prompt q. same rollout oi
- Old Policy: πθold(oi|q)
- New Policy: πθ(oi|q)
Possibility Ratio: $\frac{\pi_{\theta (o_i|q)}}{\pi_{\theta_{old}}(o_i|q)}$
- ratio = 2.0 : The new model believes that this output should have a 2-fold probability → the model is ==learning== in this direction.
- ratio = 0.5 : The new model believes that this output has only a 50% probability → the model is ==suppressing== this direction
- ratio = 1.0 : The new and old models have the same view on this output → no change
2.2.5.2 Clip
Clip ensures that the contribution of any rollout to the gradient during each parameter update will not exceed the range of [0.8, 1.2] × advantage.
The model can be steadily and gradually improved, rather than being led by the nose by individual extreme rollouts.
clip(Possibility Rate, 1 − ϵ, 1 + ϵ), (ϵ = 0.2)
If the ration exceeds the limition, it will be truncated.
e.g.
- ratio = 7.0 ➡ trucated as 1.2
- ratio = 0.1 ➡ trucated as 0.8
- ratio = 1.1 ➡ no trucation
2.2.5.3 Loss
The original model is Llama 3.3 70B Instruct. We use
reward to train the ability of fixing the PRs we provided.
However, if we don’t limit it, the model will go overfitting: The PRs dealing ability improve while the other abilities descend.
$$ i = -( i A_i,; (i, 1-, 1+) A_i ) + D{}({})\ = _{i=1}^{N} _i\ _i =
A_i =
r_i = R(o_i),G = 16,N = 512 $$