Map / Core AI / Benchmarks & Leaderboards / SWE-bench

SWE-bench

Evaluates LLM performance on real-world software issues collected from GitHub

CuratedSoftware Engineering Benchmarks

I use this adds SWE-bench to your stack and finds people with similar stacks. Every mark feeds the counts on Pulse. Sign in to mark it.

No ratings yet

Who uses it

0 use this

0use it now

0tried it, dropped it

0want to try

Nobody has marked it yet. Press I use this, Tried it or Want to try to be the first.

Ratings

Only from people who use it or tried it

No ratings yet

The first rating comes from someone who uses SWE-bench or tried it.

Use SWE-bench?

What it does

SWE-bench is a benchmark that tests whether language models and coding agents can resolve real GitHub issues: given a repository and an issue, the model writes a patch that is checked against the project's tests. It was created by researchers at Princeton, runs in a Docker-based evaluation harness, and has several variants, including Verified, Lite, Multimodal and Multilingual, each with a leaderboard.

Alternatives on the map

All 26 alternatives

Benchmarks & Leaderboards, starting with the other Software Engineering Benchmarks.

Compare with

People who use it also use

Not enough marks yet to tell. When people who use SWE-bench mark the rest of their stack, it shows up here.

Open website