evalship

What it does

evalship finds the flaws in your LLM evals, says on every pull request what really broke and why, and tells you which prompt and model changes win beyond chance.

Setup is one line for your coding agent: "Set up https://evalship.com/setup.md". The agent reviews every scorer, judge and prompt in your evals and finds their flaws, such as a judge that scores 1.0 when its call fails, or a privacy score too small to ever fail a reply. Each flaw comes with fix instructions. The agent then wires your evals into GitHub Actions or your machine and opens a setup pull request.

On every pull request, evalship comments with the evals the change really broke and the likely cause, sets flaky evals aside by checking them against main's history, and flags AI calls that no eval exercises. The evalship check can block a merge that breaks an eval.

Optimize has your coding agent try changes to your prompts and models on your machine: a cheaper or newer model, a lower reasoning effort, clearer instructions, less context. It starts from why your failing evals fail, runs each variant against the same evals, and evalship says which win beyond chance and by how much, in pass rate, cost and time. You pick what ships, as a pull request.

Your evals keep running with your own keys, in your own GitHub Actions or on your own machine. evalship reads the results from pytest, Vitest, Jest, promptfoo, Langfuse, Braintrust or LangSmith, so there's no SDK to add, and it works next to the eval platform you already use. Monitor lists every eval run with what to do next, and a weekly report each Monday sums up the regressions caught and what your evals cost.

Public repositories are free.

People who use it also use

Not enough marks yet to tell. When people who use evalship mark the rest of their stack, it shows up here.