Paper Summary – SWE-smith

Posted on In Views:
Paper Summary of "SWE-smith: Scaling Data for Software Engineering Agents".

1.Introduction

However, the most effective agents still rely on proprietary LMs, as building open source LMs for SE remains bottlenecked by the lack of large-scale, high-quality training data. This article creates, ==SWE-smith==, a novel pipeline for generating software engineering training data at scale.

SWE-bench:

  • Find real bug
  • Build up environment
  • Verification

SWE-smith:

  • Build up environment
  • Create bug
  • Verification

Contribution:

  • 4 kinds bug generation strategies
    • LM Generation
    • Procedural Modification
    • Combine Bugs
    • Invert PRs

2. Main Work(Generate bug.diff)

2.1 Choose & Install Repository

SWE-smith requires ==Python== projects with ==unit tests==.

Filter Repository:

  1. Target repositories for the 5, 000 most downloaded packages listed in the ==Python Package Index== (PyPI) as of November 18, 2024
  2. Sort the PyPI packages by GitHub stars
  3. Remove any PyPI package with less than 1, 000 stars, as well as all 12 SWE-bench test repositories from consideration.

Creaete corresponding image:

  1. Get the newest ==commit== and ==SWE-agent== to automatically attempt to install and run tests in 100 steps.
  2. Manually Check:
    • Are the installation steps reasonable (without damaging the system environment) ?
    • Are the test commands correct (have they actually run a complete test instead of a dummy run) ?
  3. Check if more than 80% of existing tests pass
  4. Create a ==Docker Image== for the repository

2.2 Creating Task Instance Candidates

4 methods to create task instances:

  1. LM Generations
  2. Procedural Modification
  3. Combine Bugs
  4. Invert PRs (Most effective)

2.3 Execution-based Validation of Candidates

1
2
3
4
5
6
7
8
9
10
11
git checkout origin-code
⬇️
run tests ➡️ full pass
⬇️
git apply bug_patch.diff
⬇️
run tests ➡️ at least one test case fails 
⬇️
FAIL_TO_PASS > 0 ✅
FAIL_TO_PASS = 0 ❌
Timeout ➡️ ⚠️Skip

2.4 Generating problem statements

End to End Demonstration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
完整的端到端示例

以 pandas 的 PyArrow 索引 bug 为例：

[1] 环境搭建
    128个仓库之一: pandas-dev/pandas@95280573
    → Docker镜像: swesmith.x86_64.pandas-dev__pandas.95280573

[2] Bug生成 (以PR Mirror为例)
    找到PR #53652 (修复PyArrow timestamp索引的bug)
    → 反转PR的diff → 得到引入bug的patch
    → 产出: bug__pr_mirror__7znr0kum.diff

[3] Docker验证
    pre-gold:  原始代码 → pytest → test_getitem_pyarrow_index PASS ✅
    post-gold: 应用bug → pytest → test_getitem_pyarrow_index FAIL ❌
    → FAIL_TO_PASS = [test_getitem_pyarrow_index]
    → ✅ bug有效

[4] 生成训练实例
    instance_id: "pandas-dev__pandas.95280573.pr_53652"
    patch:       引入bug的diff
    problem_statement: "PyArrow timestamp indexes not working with .loc indexing"
    FAIL_TO_PASS: [test_getitem_pyarrow_index]

[5] 收集专家Trajectory
    Claude 3.7 Sonnet + SWE-agent 在该实例上运行
    → Agent通过多轮交互成功修复bug
    → 完整interaction记录转为messages格式 (.jsonl)

[6] 训练
    Qwen 2.5 Coder 32B + 5016条成功trajectory
    → torch.tune full_finetune_distributed
    → SWE-agent-LM-32B (SWE-bench Verified 40.2%)

3. Data Process & Tuning Model

3.1 Verification bug.diff is Effective

  1. Run pre-gold test before applying patch, test log:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    test_get_loc_naive_dti_aware_str_deprecated PASSED
    test_indexing_with_datetime_tz PASSED
    test_indexing_fast_xs PASSED
    test_consistency_with_tz_aware_scalar PASSED
    test_indexing_with_datetimeindex_tz[setitem] PASSED
    test_indexing_with_datetimeindex_tz[loc] PASSED
    test_nanosecond_getitem_setitem_with_tz PASSED
    test_getitem_str_slice_millisecond_resolution[DataFrame] PASSED
    test_getitem_str_slice_millisecond_resolution[Series] PASSED
    test_getitem_pyarrow_index[DataFrame] PASSED    ← Notice ⚠️
    test_getitem_pyarrow_index[Series] PASSED       ← Notice ⚠️

  2. Apply bug.diff file, then run test again

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    test_get_loc_naive_dti_aware_str_deprecated PASSED
    test_indexing_with_datetime_tz PASSED
    test_indexing_fast_xs PASSED
    test_consistency_with_tz_aware_scalar PASSED
    test_indexing_with_datetimeindex_tz[setitem] PASSED
    test_indexing_with_datetimeindex_tz[loc] PASSED
    test_nanosecond_getitem_setitem_with_tz PASSED
    test_getitem_str_slice_millisecond_resolution[DataFrame] PASSED
    test_getitem_str_slice_millisecond_resolution[Series] PASSED
    test_getitem_pyarrow_index[DataFrame] FAILED    ← ❗
    test_getitem_pyarrow_index[Series] FAILED       ← ❗

  3. Create report.json

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    {
        "FAIL_TO_PASS": [
            "pandas/tests/indexing/test_datetime.py::TestDatetimeIndex::test_getitem_pyarrow_index[DataFrame]",
            "pandas/tests/indexing/test_datetime.py::TestDatetimeIndex::test_getitem_pyarrow_index[Series]"
        ],
        "PASS_TO_PASS": [
              "pandas/tests/indexing/test_datetime.py::TestDatetimeIndex::test_get_loc_naive_dti_aware_str_deprecated",
              "pandas/tests/indexing/test_datetime.py::TestDatetimeIndex::test_indexing_with_datetime_tz",
              "... total 9 ..."
          ],
        "FAIL_TO_FAIL": [],
        "PASS_TO_FAIL": []
    }
    Failed Meaning Details
    FAIL_TO_PASS Pre-gold PASS → Post-gold FAIL 2 testcases disrupted by bug ✅
    PASS_TO_PASS Pre-gold PASS → Post-gold PASS 9 testcases are not be affected ✅
    FAIL_TO_FAIL Pre-gold PASS → Post-gold FAIL -
    PASS_TO_FAIL Pre-gold PASS → Post-gold FAIL -

  4. Judge Code (valid.py:144):

    1
    2
    3
    4
    if len(report[FAIL_TO_PASS]) == 0:
        return {"status": "0_f2p"}
    else:
        return {"status": "1+_f2p"} # ✅ Valid Bug

3.2 Assemble SWE-bench Format Test Dataset

  1. Read report and verify result (harness/gather.py :316-339)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    # read report.json
    with open(path_results) as f:
        results = json.load(f)

    # 3 conditions needed
    n_f2p = len(results[PASS_TO_FAIL])
    n_p2p = len(results[PASS_TO_PASS])

    # 1. not timeout
    # 2. at least one testcase is disrupted
    # 3. at least one testcase is passed
    if KEY_TIMED_OUT in results or n_f2p == 0 or n_p2p == 0:
        return SKIP   # ← abort

  2. Assemble ==task_instance== (harness/gather.py :341-353) :

    1
    2
    3
    4
    5
    6
    7
    8
    task_instance = {
          "instance_id":   subfolder,              # e.g. "pandas-dev__pandas.95280573.pr_53652"
          "patch":         patch_content,           # bug diff contents
          "FAIL_TO_PASS":  results[PASS_TO_FAIL],   # from report.json, note the task_instance[f2p] is from result[p2f] beacuse of the rule of SWE-bench is f2p.
          "PASS_TO_PASS":  results[PASS_TO_PASS],   # from report.json
          "image_name":    rp.image_name,           # Docker Image name
          "repo":          rp.mirror_name,          # GitHub repo name
      }

  3. Create new Git branch (harness/gather.py :466-569) :

Create new branch for each valid instance on Github, then:

1
2
3
4
5
6
7
git clone repo
git checkout main
git apply bug.diff			# apply bug.diff
git commit -m "Bug Patch"	# commit bug code

remove the testcases that will fail
 git commit -m "Remove F2P Tests"  # commit

Delete the testcase that will fail:

1
2
3
4
5
6
7
f2p_test_files, _ = rp.get_test_files(task_instance)
if f2p_test_files:
    for test_file in f2p_test_files:
        os.remove(test_file_path)  # delete the test that will fail

	git add . && git commit -m "Remove F2P Tests"

Why delete this file ?

  • This way SWE-agent can find the tests need to be repaired by git diff.

  • git diff main -> deleted file -> bug position

  • ```sh Agent git checkout → testcase not exist Agent git log: commit 1: “Bug Patch” commit 2: “Remove Tests” Agent inference: Deleted files in commit 2 is used to verify bug Agent git checkout main – test_xxx.py -> Restore testcase … 

    1
    2
    3
    4
    5
    6
    7
    8

       - ==I don't think it make sense.==

    4. Assemble all instances (harness/gather.py :269-273) :

       ```python
       task_instances_path = LOG_DIR_TASKS / f"{run_id}.json"
       json.dump(task_instances, f, indent=4)

3.3 Generate problem_statement by LLM (issue_gen/generate.py)

Input for LLM:

  1. Demonstration :

    Randomly select 2 genuine GitHub issues from the SWE-bench Verified as style references.

  2. Patch (<patch> label) :

    Content of bug.diff

  3. test_output (<test_output> label) :

    Testcases that failed in post-gold. (Introducing by bug.diff)

  4. test_source_code (<test_source_code> label) :

    Analyze the failed testcases’ source code by Python AST. Gather the full test source code.

LLM constrains (ig_v2.yaml) :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
instance: |-
  <IMPORTANT>
  - DO NOT GIVE AWAY THE FIX!
  - DO NOT SAY THAT EXISTING TEST(s) FAILED.
  - DO NOT SUGGEST RUNNING TESTING COMMANDS (e.g., pytest).
  - Mimic the style of the demonstration issues.
  </IMPORTANT>

  <patch>
  {{patch}}
  </patch>

  <test_output>
  {{test_output}}
  </test_output>

  <test_source_code>
  {% for test in test_funcs[:5] %}
  {{test}}
  {% endfor %}
  </test_source_code>

Generate problem_statement (generate.py :376-387) :

1
2
3
4
5
6
7
8
9
10
11
if self.portkey_model:
      response = self.portkey_model.query(
          messages, n=self.n_instructions, stream=False
      )
  else:
      response = completion(
          model=self.model,
          messages=messages,        # ← prompt
          n=self.n_instructions,     # ← generate several indicates
          temperature=0,
      )

Extract statement (generate.py :398-401):

1
2
3
4
problem_statements = [
      choice.message.content
      for choice in response.choices
  ]

Now the instances:

1
2
3
4
5
6
7
8
9
{
    "instance_id": "...",
    "patch": "diff --git ...",
    "FAIL_TO_PASS": [...],
    "PASS_TO_PASS": [...],
    "image_name": "...",
    "repo": "...",
    "problem_statement": "### Bug Description\n\n  When using PyArrow-backed timestamp indexes with .loc indexing, an AttributeError is  raised because the 'freq' attribute doesn't exist on PyArrow-backed Index objects...\n\n ### Steps to Reproduce\n\n ```python\nimport pandas as pd\n..."
}

3.4 Collect traj by Expert Model ( Rejection Sampling Fine-Tuning) (_gen_trajs.sh)

SWE-smith aims to train a student model (Qwen 2.5 Coder 32B)

  • If use student to solve the issues, the success rate is too low (~11%), and we cannot collect enough “successful trajectories”.
  • Running with ==Claude 3.7 Sonnet== (success rate ~36%) can collect a large number of high-quality “correct answer demonstrations”.
1
2
3
4
5
sweagent run-batch \
	--config agent/swesmith_gen_claude.yaml \
	--instances.path <xxx.json>
	--output_dir trajectories
	--agent.model.temperature 0.0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
{
    "trajectory": [
      {
        "step": 1,
        "thought": "I'll help you implement the necessary changes...",
        "action": "find /testbed -type f -name '*.py' | grep -v test_",
        "observation": "OBSERVATION:\n/testbed/pandas/core/indexes/base.py\n..."
      },
      {
        "step": 2,
        "thought": "Let's look at the specific code in base.py...",
        "action": "cat -n /testbed/pandas/core/indexes/base.py | head -6077",
        "observation": "OBSERVATION:\n6077  and key.freq is None..."
      },
      // ... multi turns
      {
        "step": N,
        "thought": "The fix is working. Let's submit.",
        "action": "submit",
        "observation": "Task completed."
      }
    ],
    "info": {
      "model": "claude-3-7-sonnet-20250219",
      "temperature": 0.0,
      "cost": 2.34
    }
  }

1
2
3
4
5
6
7
8
9
10
trajectories/<run_id>/
├── pandas-dev__pandas.95280573.pr_53652/
│   ├── pandas-dev__pandas.95280573.pr_53652.traj    ← Traj
│   └── pandas-dev__pandas.95280573.pr_53652.patch   ← Funal Patch
├── getmoto__moto.694ce1f4.pr_7331/
│   ├── getmoto__moto.694ce1f4.pr_7331.traj
│   └── getmoto__moto.694ce1f4.pr_7331.patch
├── ...
└── ...

3.5 Filter Success Patches (swesmith/harness/eval.py)

1
2
3
4
python -m swesmith.harness.eval \
	--dataset_path <instances_with_statement> \
	--predictions_path <path.diff> \
	--run_id <run_id>
1
2
3
4
5
6
7
8
9
10
11
12
def run_evaluation(pred, instance, run_id, ...):
     # 1. Apply patch.diff and run test in docker.
     run_patch_in_container(
         instance, ..., patch=pred[KEY_PREDICTION]
     )

     # 2. Read log，check if all f2p testcases go PASSED.
     report = get_eval_report(pred, instance, test_log_path)

     # 3. return resolved or not.
     resolved = report.get("resolved", False)
     return {"status": "completed", "resolved": resolved}
1
2
3
4
5
6
7
8
9
10
11
logs/run_evaluation/<run_id>/
├── <instance_id1>/
│   ├── test_output.txt     ← full test log
│   └── report.json         ← {"resolved": true, "tests_status": {...}}
├── <instance_id2>/
│   ├── test_output.txt     ← full test log
│   └── report.json         ← {"resolved": false, "tests_status": {...}}
....
│
│
└── report.json             ← Summary Report

Summary Report:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
    "resolved": 1813,
    "unresolved": 3237,
    "total": 5050,
    "ids_resolved": [
        "pandas-dev__pandas.95280573.pr_53652",
        "getmoto__moto.694ce1f4.pr_7331",
        "..."
    ],
    "ids_unresolved": [
        "modin-project__modin.8c7799fd.combine__...",
        "..."
    ]
}

3.6 Trans .traj into OpenAI messages

  1. Filter success trajectories (collect_trajs.py : 57-58) :

    1
    2
    3
    4
    report = json.loads(report_path.read_text())
    is_resolved = report.get("resolved", False) 
    ...
    traj["resolved"] = is_resolved

  2. Format Conversion (utils.py : 81-111) :

    Package thought and action as content:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    def transform_traj_xml(traj):
        for message in get_messages(traj):
            if message["role"] == "assistant":
                content = message["thought"]
                content += "\n\n" + tool_call_to_action(message["tool_calls"])
    		elif message["role"] == "system":
                content = SYSTEM_PROMPT
    		else:  # tool / user
                content = message["content"][0]["text"]
    		new_traj.append({"role": role, "content": content})
    	return {"messages": new_traj}

  3. e.g. 

    Unconverted .traj:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    {
        "trajectory": [
          {
            "step": 1,
            "thought": "I'll help you implement the necessary changes to fix the issue. Let me first     explore the repository structure.",
            "action": "find /testbed -type f -name '*.py' | grep -v __pycache__ | grep -v test_ | head
      -20",
            "observation": "/testbed/pandas/core/indexes/base.py\n/testbed/pandas/core/indexes/datetime
      s.py\n/testbed/pandas/core/indexes/datetimelike.py\n..."
          },
          {
            "step": 2,
            "thought": "The PR description mentions the error occurs in base.py around line 6039. Let
      me look at that specific code.",
            "action": "cat -n /testbed/pandas/core/indexes/base.py | sed -n '6070,6090p'",
            "observation": "6070    if keyarr.dtype.kind in \"mM\":\n6071        if isinstance(key,
      list) or (\n6072            isinstance(key, type(self))\n6073            and key.freq is
      None\n6074        ):\n6075            keyarr = keyarr._with_freq(None)"
          },
          {
            "step": 3,
            "thought": "I see the issue. The code accesses `key.freq` but PyArrow-backed Index objects
      don't have a `freq` attribute. Let me create a reproduction script first.",
            "action": "cat > /testbed/reproduce.py << 'EOF'\nimport numpy as np\nimport pandas as
      pd\n\nt = pd.date_range('2023-01-01', '2023-01-04', freq='1h')\nx = np.random.randn(len(t))\ndf =
      pd.DataFrame(x, index=t, columns=['values'])\ndf.to_parquet('frame.parquet')\ndf2 =
      pd.read_parquet('frame.parquet',
      dtype_backend='pyarrow')\nprint(df2.loc[df2.index[:-5]].head())\nEOF",
            "observation": "File created successfully."
          },
          {
            "step": 4,
            "thought": "Now let me run the reproduction script to confirm the error.",
            "action": "cd /testbed && python reproduce.py",
            "observation": "AttributeError: 'Index' object has no attribute 'freq'"
          },
          {
            "step": 5,
            "thought": "Confirmed. Now I'll fix the issue. I need to add a hasattr check before
      accessing `key.freq`. Let me edit the file.",
            "action": "sed -i 's/and key.freq is None/and (not hasattr(key, \"freq\") or key.freq is
      None)/' /testbed/pandas/core/indexes/base.py",
            "observation": "Command executed successfully."
          },
          {
            "step": 6,
            "thought": "I also need to handle the `_with_freq` call which doesn't exist on
      PyArrow-backed Index objects.",
            "action": "sed -i '/keyarr = keyarr._with_freq(None)/i\\                if hasattr(keyarr,
      \"_with_freq\"):' /testbed/pandas/core/indexes/base.py",
            "observation": "Command executed successfully."
          },
          {
            "step": 7,
            "thought": "Let me rerun the reproduction script to verify the fix.",
            "action": "cd /testbed && python reproduce.py",
            "observation": "                       values\n2023-01-01 00:00:00 -0.708777\n2023-01-01
      01:00:00  0.869770\n2023-01-01 02:00:00  1.113152\n2023-01-01 03:00:00 -0.041397\n2023-01-01
      04:00:00 -0.402612"
          },
          {
            "step": 8,
            "thought": "The fix works. Let me submit the changes.",
            "action": "submit",
            "observation": "Task completed."
          }
        ],
        "info": {
          "model": "claude-3-7-sonnet-20250219",
          "temperature": 0.0
        }
      }

    transform_traj_xml():

    1
    2
    3
    4
    5
    6
    7
    8
    step 1 → assistant: thought + action → assemble together
    step 2 → assistant: thought + action → assemble together
    step 3 → assistant: thought + action → assemble together
    step 4 → user:       observation     → use as content
    step 5 → assistant: thought + action → assemble together
    step 6 → assistant: thought + action → assemble together
    step 7 → user:       observation     → use as content
    step 8 → assistant: thought + submit → assemble together

    Unconverted .jsonl:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    {
      "messages": [
        {
          "role": "system",
          "content": "You are a helpful assistant that can interact with a computer to solve
    tasks.\n\nYou have access to the following functions:\n\n---- BEGIN FUNCTION #1: bash
    ----\nDescription: Execute a bash command in the terminal.\n..."
        },
        {
          "role": "user",
          "content": "I've uploaded a python code repository in the directory /testbed. Consider the
    following PR description:\n\n### Bug Description\n\nWhen using PyArrow-backed timestamp indexes
    with .loc indexing..."
        },
        {
          "role": "assistant",
          "content": "I'll help you implement the necessary changes to fix the issue. Let me first
    explore the repository structure.\n\n<function=bash>\n<parameter=command>find /testbed -type f
    -name '*.py' | grep -v __pycache__ | grep -v test_ | head -20</parameter>\n</function>"
        },
        {
          "role": "user",
          "content": "OBSERVATION:\n/testbed/pandas/core/indexes/base.py\n/testbed/pandas/core/indexe
    s/datetimes.py\n/testbed/pandas/core/indexes/datetimelike.py\n..."
        },
        {
          "role": "assistant",
          "content": "The PR description mentions the error occurs in base.py around line 6039. Let
    me look at that specific code.\n\n<function=bash>\n<parameter=command>cat -n
    /testbed/pandas/core/indexes/base.py | sed -n '6070,6090p'</parameter>\n</function>"
        },
        {
          "role": "user",
          "content": "OBSERVATION:\n6070    if keyarr.dtype.kind in \"mM\":\n6071        if
    isinstance(key, list) or (\n6072            isinstance(key, type(self))\n6073            and
    key.freq is None\n6074        ):\n6075            keyarr = keyarr._with_freq(None)"
        },
        {
          "role": "assistant",
          "content": "I see the issue. The code accesses `key.freq` but PyArrow-backed Index objects
    don't have a `freq` attribute. Let me create a reproduction script
    first.\n\n<function=bash>\n<parameter=command>cat > /testbed/reproduce.py << 'EOF'\nimport numpy
    as np\nimport pandas as pd\n\nt = pd.date_range('2023-01-01', '2023-01-04', freq='1h')\nx =
    np.random.randn(len(t))\ndf = pd.DataFrame(x, index=t,
    columns=['values'])\ndf.to_parquet('frame.parquet')\ndf2 = pd.read_parquet('frame.parquet',
    dtype_backend='pyarrow')\nprint(df2.loc[df2.index[:-5]].head())\nEOF</parameter>\n</function>"
        },
        {
          "role": "user",
          "content": "OBSERVATION:\nFile created successfully."
        },
        {
          "role": "assistant",
          "content": "Now let me run the reproduction script to confirm the
    error.\n\n<function=bash>\n<parameter=command>cd /testbed && python
    reproduce.py</parameter>\n</function>"
        },
        {
          "role": "user",
          "content": "OBSERVATION:\nAttributeError: 'Index' object has no attribute 'freq'"
        },
        {
          "role": "assistant",
          "content": "Confirmed. Now I'll fix the issue. I need to add a hasattr check before
    accessing `key.freq`.\n\n<function=bash>\n<parameter=command>sed -i 's/and key.freq is None/and
    (not hasattr(key, \"freq\") or key.freq is None)/'
    /testbed/pandas/core/indexes/base.py</parameter>\n</function>"
        },
        {
          "role": "user",
          "content": "OBSERVATION:\nCommand executed successfully."
        },
        {
          "role": "assistant",
          "content": "I also need to handle the `_with_freq` call which doesn't exist on
    PyArrow-backed Index objects.\n\n<function=bash>\n<parameter=command>sed -i '/keyarr =
    keyarr._with_freq(None)/i\\                if hasattr(keyarr, \"_with_freq\"):'
    /testbed/pandas/core/indexes/base.py</parameter>\n</function>"
        },
        {
          "role": "user",
          "content": "OBSERVATION:\nCommand executed successfully."
        },
        {
          "role": "assistant",
          "content": "Let me rerun the reproduction script to verify the
    fix.\n\n<function=bash>\n<parameter=command>cd /testbed && python
    reproduce.py</parameter>\n</function>"
        },
        {
          "role": "user",
          "content": "OBSERVATION:\n                       values\n2023-01-01 00:00:00
    -0.708777\n2023-01-01 01:00:00  0.869770\n2023-01-01 02:00:00  1.113152\n2023-01-01 03:00:00
    -0.041397\n2023-01-01 04:00:00 -0.402612"
        },
        {
          "role": "assistant",
          "content": "The fix works. Let me submit the changes.\n\n<function=submit>\n</function>"
        }
      ],
      "resolved": true,
      "instance_id": "pandas-dev__pandas.95280573.pr_53652"
    }

3.7 Full Tuning

train/run/ft_torchtune.py

configs/train/full_ft_qwen_32b.yml

1
2
modal run swesmith/train/run/ft_torchtune.py \
	--config configs/train/full_ft_qwen_32b.yml
1
2
3
4
5
6
7
8
9
10
11
12
@app.function(
      gpu=f"H100:{N_GPUS}",        # 2 H100 GPU by default
      timeout=N_HOURS * HOURS,     # default 10 hours
  )
  def run_train(config_name, config, n_gpus):
      # write in temp conf file
      with open(config_path, "w") as f:
          yaml.dump(config, f)

      # run full tuning
      command = "tune run full_finetune_distributed --config {config_path}"
      subprocess.run(command.split(), ...)

Train config (full_ft_qwen_32b.yml) :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
model:
    _component_: torchtune.models.qwen2_5.qwen2_5_32b_instruct   # Qwen 2.5 Coder 32B

dataset:
    _component_: torchtune.datasets.chat_dataset
    source: json
    data_files: /data/archive/swesmith.trajs.250413.jsonl
    conversation_column: messages		# read field: messages
    conversation_style: openai			# OpenAI format
    train_on_input: False            # Only calculate Loss of the assistant token

  # ===== Train Config =====
  epochs: 3                          # 3 轮
  batch_size: 1
  lr: 1e-4                           # Learning Rate
  optimizer: AdamW                   # Optimizor
  lr_scheduler: cosine_warmup        # Cosine Annealing + warmup
  max_seq_len: 32768                 # 32K token context

  # ===== 精度和显存优化 =====
  dtype: bf16
  enable_activation_checkpointing: True
  compile: True                           # torch.compile

message # role content Calculate Loss?
system
user		 “You are a helpful assistant…”
“I’ve uploaded a python code repository…”
1 assistant “I’ll help you…<function=bash>…” ✅
2 user “OBSERVATION:/testbed/pandas/…”
3 assistant “The PR description mentions…<function=bash>” ✅
4 user “OBSERVATION: if keyarr…”
5 assistant “I see the issue…Let me create a script… ✅
6 user “OBSERVATION:created successfully.”
7 assistant “Now let me run the script… ✅
8 user “OBSERVATION:: ’Inde…
9 assistant “Confirmed. Now I’ll fix… ✅
10 user “OBSERVATION:executed
11 assistant “Let me rerun…< ✅
12 user “OBSERVATION:values…”
13 assistant “The fix works. Let me submit.<function=submit>” ✅

e.g.

1
2
3
4
5
6
7
assistant: "I see the issue. Let me create a script.
			<function=bash>
			<parameter=command>cat > reproduce.py << 'EOF'
			import pandas as pd
			...
			EOF</parameter>
			</function>"
1
2
3
4
Model Input: "...OBSERVATION:\nFile created.\n I see the issue. Let me create"
Next token prediction: next token is ?
Right Answer: "a"
CE Loss: -log P("a" | Input)

…