> ## Documentation Index
> Fetch the complete documentation index at: https://hydroxai.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# SWE-bench

> The gold standard benchmark for evaluating AI coding assistants on real-world software engineering tasks from GitHub.

## Overview

**SWE-bench** is the most widely cited benchmark for evaluating AI systems on real-world software engineering tasks. Created by researchers at Princeton University, it tests whether an AI model can resolve actual GitHub issues from popular open-source Python repositories.

Unlike synthetic coding benchmarks, SWE-bench uses **real bug reports and feature requests** from projects like Django, Flask, scikit-learn, sympy, and more. The model must understand the codebase, localize the relevant files, and produce a patch that passes the project's test suite.

## Key Details

| Property         | Value                                                   |
| ---------------- | ------------------------------------------------------- |
| **Created by**   | Princeton NLP (Carlos E. Jimenez et al.)                |
| **Released**     | October 2023                                            |
| **Task type**    | Code generation / Bug fixing                            |
| **Dataset size** | 2,294 task instances (full), 500 (Verified), 300 (Lite) |
| **Languages**    | Python                                                  |
| **Evaluation**   | Unit test pass rate                                     |
| **Leaderboard**  | [swebench.com](https://www.swebench.com)                |

## How It Works

1. **Input**: The model receives a GitHub issue description and access to the repository codebase
2. **Task**: Generate a code patch (diff) that resolves the issue
3. **Evaluation**: The patch is applied to the repository and the project's existing test suite is run
4. **Success**: A task is "resolved" only if **all** relevant tests pass after applying the patch

```
GitHub Issue Description
        │
        ▼
┌──────────────────┐
│  AI Model        │  → Analyzes codebase + issue
│  (SWE-agent,     │  → Localizes relevant files
│   Devin, etc.)   │  → Generates patch
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│  Test Suite      │  → Applies patch to repo
│  Evaluation      │  → Runs unit tests
│                  │  → Pass/Fail verdict
└──────────────────┘
```

## Variants

### SWE-bench Full

The complete dataset of 2,294 task instances across 12 Python repositories. Due to noisy or ambiguous tasks, the full set is less commonly used for leaderboard ranking.

### SWE-bench Verified

A human-validated subset of **500 tasks** where annotators confirmed the problem statements are clear and the test cases are correct. This is the **primary leaderboard** used by most teams.

### SWE-bench Lite

A smaller subset of **300 tasks** designed for faster evaluation cycles. Useful for development and iteration.

## Why It Matters

SWE-bench is the closest benchmark to **real-world developer work**:

* Tests end-to-end software engineering, not just code completion
* Requires understanding large codebases (thousands of files)
* Demands reasoning about test requirements and edge cases
* Patches must be production-quality (they run against real test suites)

Performance on SWE-bench is directly correlated with how useful a model is as an AI coding assistant in production workflows.

## Notable Results

| Model / System                | SWE-bench Verified | Date |
| ----------------------------- | ------------------ | ---- |
| OpenAI o3 + Codex             | \~65%              | 2026 |
| Claude 3.5 Sonnet + SWE-agent | \~55%              | 2025 |
| GPT-4o + Agentless            | \~45%              | 2025 |
| DeepSeek-V3                   | \~42%              | 2025 |
| Claude 3 Opus (baseline)      | \~22%              | 2024 |

<Note>
  Scores evolve rapidly. SWE-bench results depend heavily on the scaffolding (agent framework) used with the model, not just the model itself.
</Note>

## Limitations

* **Python only** — Does not test other programming languages
* **Agent-dependent** — Performance varies significantly based on the scaffolding / agent framework used
* **Repository scope** — Limited to 12 specific open-source projects
* **Test-based evaluation** — A correct fix that doesn't match the expected test structure may be marked as failed

## References

* [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://arxiv.org/abs/2310.06770) — Original paper by Jimenez et al. (2023)
* [SWE-bench Leaderboard](https://www.swebench.com) — Live rankings