Docs / Contributing
How ivralo is measured
The test suite never touches a model, so it cannot say whether a change to a skill made the agents better or worse. The evals can: they run real requests against the real model and check what came out. This page is for people working on ivralo itself.
What an eval is
An eval case is a YAML file under evals/cases/: a request typed as a person would type it, the files the workspace starts with, and what must be true afterwards. The expectations are checked against the workspace and the stored run, never against the model's wording:
- which presets the run opened (
opened,never_opened); - which files exist, which must not, which must be unchanged;
- a check on a produced file, such as a workbook's column sum or a PDF's page count;
- the project's own commands passing (
command_passes), which is how a code change is proven; - budgets for tool calls, errors and time, and the language of the reply.
Each run gets a fresh session and a scratch copy of the workspace, with long-term memory off. A case passes by majority over its runs, three by default. Every case has a level, easy, medium or hard, and a hard case names the trap that makes the obvious answer wrong.
What it needs
The eval suite lives only in a checkout of the repository. evals/ is not in the wheel, and ivralo eval reads the cases from the home, which in a checkout is the checkout itself. Install from a checkout sets one up.
Beyond the checkout it needs what a normal run needs: the database and a model. Any model reference works. Then build the fixtures once; they are generated from tracked builders and never edited by hand:
uv run python evals/fixtures/build.py
ivralo eval refuses to start, in one line, when a fixture a selected case needs is missing or was built by older builders.
Running it
uv run ivralo eval --list # the cases and their levels; runs nothing
uv run ivralo eval # every case that is not frozen
uv run ivralo eval coder-adds-a-shout-flag --runs 1
uv run ivralo eval --level hard --jobs 4
| Option | What it does |
|---|---|
--runs N | Runs per case (default 3) |
--level | Only the cases of this level; repeat for more |
--split | Only the cases of this split; see below |
--jobs N | Case runs at the same time (default IVRALO_EVAL_JOBS, else 1) |
--baseline <tag> | Compare with a release's committed baseline, evals/baselines/<tag>.json |
--fail-under P | Exit 1 when fewer than P percent of the cases pass |
--format markdown | The table on stdout, ready for a pull request |
Each finished run prints a line as it ends. The results land in evals/results/, which git ignores. Against a baseline, the pass rate is the one number compared; the other columns are measurements.
Splits
Every case belongs to one split:
| Split | What it is for |
|---|---|
tuning | The cases a skill may be tuned on |
held-out | The cases a tuned skill is checked against once. A case that names no split is here |
frozen | The incident benchmark's frozen holdout |
The frozen cases live apart, under evals/frozen/, with their keys and reviews. Only ivralo eval --split frozen, run on its own, reads them. A plain run leaves them out, and --explain, --tune and ivralo eval merge refuse them by name. Nothing on the tuning side ever sees them, so the number they give stays honest.
When a pull request needs it
If you changed skills/, agents/building/team.py or agents/roster/roster.py, the model has to judge it: run the evals and paste the table into the pull request. It takes minutes to hours and wakes the model, so run it on purpose, not as part of the test tiers.
A wrong answer becomes a case
When a run goes wrong, record it with a note on what it should have done. In a session:
/bad it should have read the whole sheet, not the first 100 rows
At the command line:
ivralo feedback --last --note "it should have read the whole sheet"
Both write a draft case under evals/cases/drafts/, which git ignores, so a draft never leaves the machine until you move it. Anything that would say the answer was right is left as a FILL ME for you to write; a recorded case never asserts that a wrong answer is right.
uv run ivralo eval drafts # list the drafts: filled or to fill
uv run ivralo eval --drafts # run the filled ones
git mv evals/cases/drafts/<name>.yaml evals/cases/
ivralo eval record --last does the same for any stored run, with or without a note.
How a skill changes
Skills are the expertise, and they change through the eval, by a pull request a person opens.
First, read why a case failed. --explain joins each failed run with the stored run and prints the calls, the skills read and the rule that failed, without asking any model:
uv run ivralo eval --explain coder-adds-a-shout-flag
Then --tune has a model propose one rewrite of one section of one shipped skill:
uv run ivralo eval --tune coding
- It finds the tuning cases whose runs read the skill, says what the tune will cost, and asks before it starts.
- It hands the tuner the failed runs, never a case's expectations and never a held-out case, and asks for one section rewritten.
- Code refuses a rewrite that leaves its section, grows too long, breaks the skill's lint, or copies a detail of a case into the skill.
- A rewrite is kept only when a strict majority of the failing cases improve and no case regresses.
- The best one is held to the held-out split once. One that wins on tuning and loses on held-out is reported, not proposed.
skills/ is never written. The result is a patch and a pull-request body under evals/results/tuning/<stamp>-<skill>/, and you open the pull request:
git switch -c fix/tune-coding
git apply evals/results/tuning/<stamp>-coding/skill.patch
gh pr create --body-file evals/results/tuning/<stamp>-coding/pull-request.md
The prompts, team.py and the roster are outside this loop: a change there moves every case at once, and its diff cannot say which failure it answers.