SWE-bench

Posted on Edited on In Views:
Analysis of SWE-bench

[TOC]

1. Code Analysis

1.1 Architecture

The architecture of this benchmark can be divided into 4 layer:

  1. Entry Layer
  2. Inference Layer & Harness Layer
  3. Core Engine Layer
  4. Data Collection Layer
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
┌───────────────────────────────────────────────────────────────┐
│  Entry Layer: Entry Points                                    │
│  Command Line / Python API / Modal Cloud                      │
│  Layer that interplays with User Directly					    │
├───────────────────────────────────────────────────────────────┤
│  Inference Layer             	  │  Harness Layer              │
│  Generate Patch              	  │  Evaluate the patch	    	│
│																│
│  - run_api.py (Call LLM)     	  │  - run_evaluation.py        │
│  - run_llama.py (Local Llama)   │  - prepare_images.py        │
│  - run_live.py (GitHub Issue)   │  - reporting.py             │
├───────────────────────────────────────────────────────────────┤
│  Docker Layer                               					│
│  Docker Building / Test Execuation  / Evaluation / Log Parsing│
│																│
│  - docker_build.py    - grading.py                            │
│  - docker_utils.py    - test_spec/                            │
│  - log_parsers/       - constants/                            │
├───────────────────────────────────────────────────────────────┤
│  Data Collection Layer                                	    │
│																│
│  Collect Issue from Github → Build Test Dataset				│
│  - collect/   - versioning/                                   │
└───────────────────────────────────────────────────────────────┘

1.2 Core Data Structure

1.2.1 SWEbenchInstance (SWE-bench Instance)

1
2
3
4
5
6
7
8
9
10
11
12
class SWEbenchInstance(TypedDict):
    repo: str              # repo name, e.g. "sympy/sympy"
    instance_id: str       # Unique ID, e.g. "sympy__sympy-20590"
    base_commit: str       # git commit SHA where the issue exists
    patch: str             # patch from GitHub PR, standard answer
    test_patch: str        # patch newly added for this issue
    problem_statement: str # statement of the issue
    hints_text: str        # hint text
    created_at: str        # time when issue created
    version: str           # version
    FAIL_TO_PASS: str      # JSON: test list that should maintain a "fail→pass" status after repairing 
    PASS_TO_PASS: str      # JSON: test list that should maintain a "pass→pass" status after repairing

The core judgement of the evaluation process are FAIL_TO_PASS and PASS_TO_PASS.

FAIL_TO_PASS: Tests that fail when the bug exists and pass when the bug is repaired.

PASS_TO_PASS: Tests that should pass when the bug both exists and is repaired.

1.2.2 TestSpec (Test Specification)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
@dataclass
class TestSpec:
    instance_id: str        # Instance ID
    repo: str               # repo name
    version: str            # version
    repo_script_list: list  # Bash Commands List: to install repo code
    eval_script_list: list  # Bash Commands List: to evaluate result
    env_script_list: list   # Bash Commands List: to configurate env
    arch: str               # CPU arch: arm & x64
    FAIL_TO_PASS: list      # ditto
    PASS_TO_PASS: list      # ditto
    language: str           # language: py, java, js, go, rust...
    docker_specs: dict      # Docker specifications
    namespace: str          # DockerHub namespace

1.3.3 Three-Layer Structure of Docker Image

1
2
3
4
5
6
7
8
9
10
sweb.eval.x86_64.sympy__sympy-20590:latest    ← Instance Image
  │  each task instance contains one, including specific repo code
  │
  └─ sweb.env.py.x86_64.a1b2c3...:latest      ← Environment Image
       │  each repo version has one, including pip/apt dependencies
       │
       └─ sweb.base.py.x86_64:latest          ← Base Image
            │  each language contains one, including Python/Java/...
            │
            └─ ubuntu:22.04                     ← OS

Image Naming Rule :

sweb.{Layer}.{Language}.{Arch}.{Hash}:{Lable}

e.g. Test 3 different Django Issues

1
2
3
django__django-15022  (Django 4.1, Python 3.11)
django__django-15277  (Django 4.1, Python 3.11)
django__django-11333  (Django 3.2, Python 3.9)

If the docker is not layered, it is needed to create 3 different docker images:

1
2
3
django-15022 image ──── Ubuntu + Python + Django 4.1 + Code → 2GB
django-15277 image ──── Ubuntu + Python + Django 4.1 + Code → 2GB
django-11333 image ──── Ubuntu + Python + Django 3.2 + Code → 2GB

Totally 6 GB needed.

But if we layer the image:

  1. Base Layer: we need 2 images for Python 3.9 and Python 3.11, total 2 * 1 = 2GB.
  2. Env Layer: we need 2 images for Django 3.2 and Django 4.1, total 2 * 2 = 4GB
  3. Instance Layer: we need 3 images for each issue, total 50MB × 3 ≈ 0.15GB
  4. total 6.15 GB

It seems that the cost of 2 plans seem similar. But if we have 2294 instance, what will happen ?

  • No layer: 2294 * 2GB = 4588 GB
  • 8 images of Base Layer + 200 images of Env Layer + 2294 images of instance Layer = 8 * 1GB + 200 * 2GB + 0.15GB * 2294 ≈ 523 GB
  • 88.60% Memory are saved !

1.3 Inference Pipeline

1.3.1 Default Inference (Not Available Now)

In fact, the main function of SWE-bench is to evaluate the score of patch generated by LLM. It doesn’t care about how you generate the patch. There are three original scripts to call LLM to generate patch. However there are a lot of limits of these scripts because they were just created for the paper demonstration.

There are three ways of run interference command:

  1. Call cloud LLM API (OpenAI ChatGPT / Ahthropic Claude)

    1
    2
    3
    4
    python -m swebench.inference.run_api \
        --dataset_name_or_path princeton-nlp/SWE-bench_oracle \
        --model_name_or_path gpt-4 \
        --output_dir ./outputs

  2. Local LLM

    1
    2
    3
    4
    python -m swebench.inference.run_llama \
        --model_name_or_path princeton-nlp/SWE-Llama-13b \
        --dataset_path princeton-nlp/SWE-bench_oracle \
        --output_dir ./outputs

  3. Reason about issues that appear in real-time on GitHub

    1
    2
    3
    python -m swebench.inference.run_live \
        --model_name gpt-3.5-turbo-1106 \
        --issue_url https://github.com/some/repo/issues/123

1.3.1.1 How Did The Original Inference Scripts Work?

1.3.2 How to Inference Personally

1.3.2.2 Prepare for the Env

1
conda activate /

1.3.2.1 Start a local LLM by vLLM (This section should be moved to Part 2 )

A local LLM has been deployed and is driven by vLLM:

1
2
3
4
5
6
7
8
9
10
docker run -d --name vllm_server --gpus all \
    -v /data/disk2/wye/models:/models \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model /models/Qwen2.5-Coder-7B-Instruct \
    --dtype auto \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.9 \
    --port 8000

1.3.2.2 Full run_local_vllm.py file

A inference script has been added:

1
swebench/inference/run_local_vllm.py

Parameters:

Parameter Type Default Description
–dataset_name str princeton-nlp/SWE-bench_Lite Dataset:
princeton-nlp/SWE-bench_Lite (300)
princeton-nlp/SWE-bench_verified (500)
princeton-nlp/SWE-bench (2249)
–split str test 数据集分割：test
–output_dir str /data/disk2/wye/SWE-bench/patches_by_llm JSONL output directory, auto-create if not exist
–model_short_name str None (Auto-Detect) If not setted, auto-get from vLLM/v1/models
–max_instances int None(Run all instances) Limit to only run the first N instances; 3 = test
–vllm_url str http://localhost:8000/v1 vLLM server URL
–model_name str /models/Qwen2.5-Coder-7B-Instruct the model’s name for vLLM
–max_context_files int 5 context file nums for phase2

1.3.2.3 Inference Commnds

Start vLLM:

1
2
3
4
5
6
7
8
9
10
docker run -d --name vllm_wye_swebench --gpus all \
    -v /data/disk2/wye/models:/models \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model /models/Qwen2.5-Coder-7B-Instruct \
    --dtype auto \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.9 \
    --port 8000

Check if the vLLM is started:

1
docker logs -f vllm_wye_swebench
1
cd /data/disk2/wye/SWE-bench

Run test:

1
python -m swebench.inference.run_local_vllm --max_instances 3

Run SWE-bench_Lite(default):

1
python -m swebench.inference.run_local_vllm

Run SWE-bench_Verified:

1
2
python -m swebench.inference.run_local_vllm \
      --dataset_name princeton-nlp/SWE-bench_Verified

Run Full:

1
2
python -m swebench.inference.run_local_vllm \
      --dataset_name princeton-nlp/SWE-bench

[!NOTE]

The inference script will catch the dataset automatically.

1
2
3
4
5
(base) ubuntu@ubuntu:/data/disk2/wye/SWE-bench$ ll /home/ubuntu/.cache/huggingface/datasets/

...
drwxrwxr-x  3 ubuntu ubuntu  4096 Jul  1 12:06 princeton-nlp___swe-bench_lite/
...

1.4 Evaluation Pipeline

If you run the main test command:

1
2
3
4
5
python -m swebench.harness.run_evaluation \
    --dataset_name princeton-nlp/SWE-bench_Lite \
    --predictions_path predictions.jsonl \
    --max_workers 4 \
    --run_id my_run

The actual functions call chain:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
main()                                    # run_evaluation.py:main()
  ├─ get_predictions_from_file()          # 读取模型预测的 patch
  ├─ load_swebench_dataset()              # 从 HuggingFace 下载/加载数据集
  ├─ get_dataset_from_preds()             # 匹配合并: 哪些实例有预测结果
  │
  ├─ build_env_images()                   # 【阶段1】构建 Docker 环境镜像
  │   ├─ build_base_images()              #   先构建基础镜像 (Python/Java运行时)
  │   ├─ get_env_configs_to_build()       #   找出需要构建的环境镜像
  │   └─ run_threadpool(build_image)      #   多线程并行构建
  │
  ├─ run_instances()                      # 【阶段2】并行跑所有实例
  │   └─ for each instance (多线程):
  │       └─ run_instance()               #   单个实例的评测
  │           ├─ build_container()        #     创建 Docker 容器
  │           ├─ container.start()        #     启动容器
  │           ├─ copy_to_container()      #     把 patch.diff 拷进容器
  │           ├─ git apply patch.diff     #     尝试打补丁
  │           ├─ copy_to_container()      #     把 eval.sh 拷进容器
  │           ├─ exec_run_with_timeout()  #     执行 /bin/bash /eval.sh
  │           ├─ get_eval_report()        #     解析测试输出, 评分
  │           └─ cleanup_container()      #     清理容器
  │
  ├─ clean_images()                       # 【阶段3】清理 Docker 镜像
  └─ make_run_report()                    # 【阶段4】生成最终评测报告

1.4.1 Main

test split of SWE-bench Multimodal does not support the local evaluation because of the vast cost of memory.

1
2
3
4
5
6
7
8
if dataset_name == "SWE-bench/SWE-bench_Multimodal" and split == "test":
        print(
            "⚠️ Local evaluation for the test split of SWE-bench Multimodal "
            "is not supported. Please check out sb-cli "
            "(https://github.com/swe-bench/sb-cli/) for instructions on how "
            "to submit predictions."
        )
        return

Check ID length:

1
assert len(run_id) > 0, "Run ID must be provided"

Check / Create report_path:

1
2
3
4
if report_dir is not None:
        report_dir = Path(report_dir)
        if not report_dir.exists():
            report_dir.mkdir(parents=True)

SWE-bench’s docker image has 2 pulling ways (DockerHub or Local):

namespace Image Source Action
“swebench” (default) DockerHub docker pull swebench/sweb.eval.xxx
None (pass -n none) Local docker build -t sweb.eval.xxx .

force_rebuild: Only works when you use local build. It will delete the old image and re-run docker build command.

Load predictions (records including patches generated by LLM for solving the issues).

And trans the list into hash map:

1
2
predictions = get_predictions_from_file(predictions_path, dataset_name, split)
predictions = {pred[KEY_INSTANCE_ID]: pred for pred in predictions}

Import dataset tested:

1
2
3
dataset = get_dataset_from_preds(
        dataset_name, split, instance_ids, predictions, run_id, rewrite_reports
    )

Full dataset including skipped test instances:

1
full_dataset = load_swebench_dataset(dataset_name, split, instance_ids)

Evaluating in the cloud:

1
2
3
4
5
6
7
if modal:
	if not dataset:
		print("No instances to run.")
	else:
        validate_modal_credentials()
        run_instances_modal(predictions, dataset, full_dataset, run_id, timeout)
	return

Evaluating locally:

  1. Set the maximum number of file descriptors (fds) that can be opened by a process. Docker APi opens several sockets/fds for each container. The default fds num 1024 might be insufficient.

    1
    2
    3
    4
    if platform.system() == "Linux":
            # resource.setrlimit: equal to setrlimit(2) sys call
            resource.setrlimit(resource.RLIMIT_NOFILE, (open_file_limit, open_file_limit))
        client = docker.from_env()

  2. recording the existing images before evaluating:

    1
    existing_images = list_images(client)

  3. Dataset comes from get_dataset_from_preds, it might be empty. The evaluation will go on only when it’s not empty. If it’s local docker image and it’s not a re-grading process, create shared dockers(base + env) for all instances:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    build_env_images(
                    client,
                    dataset,
                    force_rebuild,
                    max_workers,
                    namespace,
                    instance_image_tag,
                    env_image_tag,
                )

  4. Run all the instances(Here, each instance will generate an independent docker based on the env_images):

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    run_instances(
                predictions,
                dataset,
                cache_level,
                clean,
                force_rebuild,
                max_workers,
                run_id,
                timeout,
                namespace=namespace,
                instance_image_tag=instance_image_tag,
                env_image_tag=env_image_tag,
                rewrite_reports=rewrite_reports,
            )

  5. Clean images and generate report:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    clean_images(client, existing_images, cache_level, clean)
        return make_run_report(
            predictions,
            full_dataset,
            run_id,
            client,
            namespace,
            instance_image_tag,
            env_image_tag,
        )

1.4.2 Run_Instance

1.4.2.1 Preparation

入参:

1
2
3
4
5
6
7
8
9
10
def run_instance(
    test_spec: TestSpec,          # test specification
    pred: dict,                   # prediction {instance_id, model_name_or_path, model_patch}
    rm_image: bool,               # rm image after run_instance?
    force_rebuild: bool,          # rebuild image forcely?
    client: docker.DockerClient,  # Docker client
    run_id: str,                  # unique run_id 
    timeout: int | None = None,   # timeout
    rewrite_reports: bool = False,# True = don't run test again, re-generate report.json from existing test_output.txt
) -> dict:

Initialization:

1
2
3
4
instance_id = test_spec.instance_id
model_name_or_path = pred.get(KEY_MODEL, "None").replace("/", "__")
log_dir = RUN_EVALUATION_LOG_DIR / run_id / model_name_or_path / instance_id
report_path = log_dir / LOG_REPORT

rewrite_reports method (existing test_output.txt) :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
if rewrite_reports:
        test_output_path = log_dir / LOG_TEST_OUTPUT
        if not test_output_path.exists():
            raise ValueError(f"Test output file {test_output_path} does not exist")
        # re-generate a new report from existing test_output file
        report = get_eval_report(
            test_spec=test_spec,
            prediction=pred,
            test_log_path=test_output_path,
            include_tests_status=True,
        )

        with open(report_path, "w") as f:
            # change report into a JSON	 string.
            f.write(json.dumps(report, indent=4))
        return {
            "completed": True,
            "resolved": report[instance_id]["resolved"],
        }

If evaluation is interrupted mid-run, any instances whose report.json has already been written are automatically skipped on restart. This means you can safely kill the process (Ctrl+C, crash, etc.) and rerun it later — only unfinished instances will be evaluated, without duplicating completed work.

1
2
3
4
5
6
if report_path.exists():
        report = json.loads(report_path.read_text())
        return {
            "completed": True,
            "resolved": report[instance_id]["resolved"],
        }

Create a shortcut in the logs directory, pointing to the image’s build directory. The purpose is to facilitate debugging - when you view the logs of a specific instance, you can follow this link to directly locate the corresponding Docker build files (Dockerfile, build logs, etc.).

1
2
3
4
5
6
7
8
9
10
11
12
13
if not test_spec.is_remote_image:
        build_dir = INSTANCE_IMAGE_BUILD_DIR / test_spec.instance_image_key.replace(
            ":", "__"
        )
        image_build_link = log_dir / "image_build_dir"
        if not image_build_link.exists():
            try:
                # symlink_to(): equal to ln -s <build_dir> <image_build_link>
                image_build_link.symlink_to(
                    build_dir.absolute(), target_is_directory=True
                )
            except:
                pass

log:

1
2
3
log_dir.mkdir(parents=True, exist_ok=True)
    log_file = log_dir / LOG_INSTANCE
    logger = setup_logger(instance_id, log_file)

1.4.2.2 Apply Patch & Run eval.sh

Env & Variable:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
container = None
eval_completed = False
report = {}

try:
    # If instance's image does not exist, trigger build_instance_image ()... run command `docker create` not `docker run`
    container = build_container(
        test_spec, client, run_id, logger, rm_image, force_rebuild
    )
    
    # container.start() equals to docker start <container_id>
    container.start()
    logger.info(f"Container for {instance_id} started: {container.id}")
    
    # Apply patch phase:
    
    # Create a local patch.diff file
    patch_file = Path(log_dir / "patch.diff")
    patch_file.write_text(pred[KEY_PREDICTION] or "")
    
    logger.info(
            f"Intermediate patch for {instance_id} written to {patch_file}, "
            f"now applying to container..."
        )
    
    # Copy the diff file to docker
    copy_to_container(container, patch_file, PurePosixPath(DOCKER_PATCHDOCKER_PATCH))
    
    # Try to apply the diff file
    # Try tree times. The three commands are in descending order of strictness.
    # Break if one of three commands succeed.
    """
    e.g.
    workdir = "testbed"
    user = "root"
    git_apply_cmd = "git apply --verbose"
    DOCKER_PATCH = "/tmp/patch.diff"
    
    Full Command:
    docker exec -w /testbed -u root <container> git apply --verbose /tmp/patch.diff
    """
    applied_patch = False
    for git_apply_cmd in GIT_APPLY_CMDS:
        val = container.exec_run(
            f"{git_apply_cmd} {DOCKER_PATCH}",
            workdir=DOCKER_WORKDIR,
            user=DOCKER_USER,  
        )
        # val.exit_code == 0 means apply successful.
        if val.exit_code == 0:
            logger.info(f"{APPLY_PATCH_PASS}:\n{val.output.decode(UTF8)}")
            applied_patch = True
            break
        else:
            logger.info(f"Failed to apply patch to container: {git_apply_cmd}")
            
	# If all three git apply commands fail: 
    if not applied_patch:
        logger.info(f"{APPLY_PATCH_FAIL}:\n{val.output.decode(UTF8)}")
        raise EvaluationError(
            instance_id,
            f"{APPLY_PATCH_FAIL}:\n{val.output.decode(UTF8)}",
            logger,
        )

	# Record the status of repo before executing eval.sh
    # git -c core.fileMode=false diff: Ignore the permission changes (Only track the diff of contents)
    git_diff_output_before = (
        container.exec_run(
            "git -c core.fileMode=false diff", workdir=DOCKER_WORKDIR
        )
        .output.decode(UTF8)
        .strip()
    )
    logger.info(f"Git diff before:\n{git_diff_output_before}")

Prepare the eval.sh file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# eval.sh is generated by properites of TestSpec.eval_script, including:
# - apply test_patch
  """
	test_patch is a diff file provided by the issue provider, which includes the test item that will trigger the isssue.
  """
# - run test commands
# - restore test file

eval_file = Path(log_dir / "eval.sh")
eval_file.write_text(test_spec.eval_script)
logger.info(
    f"Eval script for {instance_id} written to {eval_file}; "
    f"copying to container..."
)
copy_to_container(container, eval_file, PurePosixPath("/eval.sh"))

Run eval.sh:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# record the test_output, timed_out, total_runtime
test_output, timed_out, total_runtime = exec_run_with_timeout(
    container, "/bin/bash /eval.sh", timeout
)

test_output_path = log_dir / LOG_TEST_OUTPUT
logger.info(f"Test runtime: {total_runtime:_.2f} seconds")

# write test_output into a local file
with open(test_output_path, "w") as f:
    f.write(test_output)
    logger.info(f"Test output for {instance_id} written to {test_output_path}")
    if timed_out:
        f.write(f"\n\nTimeout error: {timeout} seconds exceeded.")
        raise EvaluationError(
            instance_id,
            f"Test timed out after {timeout} seconds.",
            logger,
        )

Record the repo status( Check if the eval.sh changed the repo accidently):

1
2
3
4
5
6
7
8
git_diff_output_after = (
    container.exec_run(
        "git -c core.fileMode=false diff", workdir=DOCKER_WORKDIR
    )
    .output.decode(UTF8)
    .strip()
)
logger.info(f"Git diff after:\n{git_diff_output_after}")
1
2
if git_diff_output_after != git_diff_output_before:
    logger.info("Git diff changed after running eval script")

1.4.2.3 Grading

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
logger.info(f"Grading answer for {instance_id}...")
report = get_eval_report(
            test_spec=test_spec,
            prediction=pred,
            test_log_path=test_output_path,
            include_tests_status=True,  # True = including state of each instance's test case
)

logger.info(
	f"report: {report}\n"
	f"Result for {instance_id}: resolved: {report[instance_id]['resolved']}"
)

with open(report_path, "w") as f:
	f.write(json.dumps(report, indent=4))
eval_completed = True

1.4.2.4 Exception Dealing & Clear Env

Exception:

1
2
3
4
5
6
7
8
9
10
11
except (EvaluationError, BuildImageError) as e:
    error_msg = traceback.format_exc()
    logger.info(error_msg)
    print(e)
except Exception as e:
    error_msg = (
        f"Error in evaluating model for {instance_id}: {e}\n"
        f"{traceback.format_exc()}\n"
        f"Check ({logger.log_file}) for more information."
    )
    logger.error(error_msg)

Clear Env:

1
2
3
4
5
6
7
8
9
10
11
12
13
finally:
    cleanup_container(client, container, logger)
    # optional
    if rm_image:
        remove_image(client, test_spec.instance_image_key, logger)
        
	# close logger system
    close_logger(logger)
    
    return {
        "completed": eval_completed,
        "resolved": report.get(instance_id, {}).get("resolved", False),
    }

本文链接：https://wangyier.top/2026/07/15/SWE-bench/

版权声明：本博客所有文章除特别声明外，均采用 CC BY-NC-SA 4.0 许可协议。转载请注明来自 The Great Library！