Docs / Incidents

Pilot on your incidents

Before you trust the Investigator with a live incident, measure it on incidents your team has already resolved. ivralo pilot run investigates each one from its evidence alone and scores the report against the cause your engineers found, all on your machine.

What a pilot does

Bring about ten resolved incidents, each as its evidence bundle, the revision the service ran, and the cause your engineers found. Then run:

ivralo pilot run ~/ivralo-pilot

/home/ops/ivralo-pilot/.pilot/20261023-093000/pilot.md
7 of 10 incidents matched the known cause (2 missed, 1 no report, 0 not run)

For each incident, ivralo makes a fresh clone of your repository at the incident's revision, copies the bundle into it, and runs the Investigator exactly as ivralo incident does. Only after the investigation has ended does it read the known cause and score the filed report against it.

The score is computed by a fixed check of the outcome, the class and the files the report names. No model judges it and no wording is compared. That makes it an accuracy number you can check yourself, on your own incidents.

The pack

A pack is a directory with one directory per incident. Each directory's name is the incident's name in the report:

ivralo-pilot/
  2026-03-checkout-500s/
    bundle/              the evidence
    revision             one line: the commit the service ran
    known-cause.yaml     what your engineers found
  2026-05-ledger-pool/
    bundle/
    revision
    known-cause.yaml

Directories whose names start with a dot, and files at the top of the pack (a README of your own), are skipped.

Entry	What it is
`bundle/`	An evidence bundle, read by the same rules as one under `incident/`. Put in what the on-call engineer had while the incident was open: logs, the deploy log, the alert, notes from before the cause was known.
`revision`	Exactly one non-empty line: the commit sha, tag or branch the service ran when it happened. It starts with a letter or digit, holds letters, digits and `. _ / + -`, is at most 200 characters, and never contains `..`.
`known-cause.yaml`	What your engineers concluded, in the keys below.

Leave the postmortem, the fix and anything written afterwards out of bundle/. Those are the answer, and the Investigator reads every file in the bundle. ivralo cannot tell a postmortem from a log.

known-cause.yaml

cause: >
  The 14:02 deploy dropped the upper-casing of coupon codes in parse(); every
  lower-case code raised KeyError in pricing.
outcome: DIAGNOSED
class: code
cites: [src/svc/parsing.py]
never_cites: [src/svc/legacy.py]

Key	Required	Meaning
`cause`	yes	Your engineers' words. Shown in the pilot report beside the Investigator's cause; never scored.
`outcome`	yes	`DIAGNOSED`, `NOT_CODE_ISSUE`, `INSUFFICIENT_EVIDENCE` or `ESCALATED`.
`class`	no	`code`, `config`, `data`, `infrastructure`, `external` or `ambiguous`. When given, the report must agree.
`cites`	no	Files a right report must name.
`cites_any`	no	A cause two files share: the report must name at least one.
`never_cites`	no	Files that look guilty and are not: the report must not name them.
`says_insufficient`	no	`{settles: [pattern, ...]}`: the report must say the evidence was insufficient and name, in a line it wrote, what would settle it.
`no_action`	no	`{reason: [pattern, ...]}`: for an incident that needed nothing done, such as a false alarm or a maintenance window; the report must give the reason.

A DIAGNOSED known cause should name what was diagnosed: a class, or files in cites or cites_any. One that names none of them still runs, but any diagnosis matches it, so the report marks that incident as scored on outcome only and counts how many were.

A mistake in the file is reported in the file's own name, such as known-cause.yaml: outcome must be one of ..., and that incident is listed as not run while the others still run.

Keeping the answer away from the Investigator

The Investigator works in a clone that holds the repository at the revision and a copy of bundle/, nothing else:

The clone is made with git clone --no-local, so it shares no objects with your repository. Its origin is removed and nothing in its .git names your repository's path.
History up to the revision is kept, because the Investigator compares the deploys the bundle names. Branches and tags past the revision are deleted and their commits pruned, since a later commit, the fix, is the answer.
Each incident runs in its own session (incident-pilot-<name>-...) with long-term memory off.

Two things are yours to get right: keep the pack outside the repository (ivralo refuses a pack inside it, since a revision that tracked the pack would carry the known causes into the clone), and keep the answer out of the bundle.

Your repository is only read. No branch, worktree or file is added to it, and the clones are removed at the end.

Running it

ivralo pilot run ~/ivralo-pilot                         # against the repository you are in
ivralo pilot run ~/ivralo-pilot --repository ~/src/ledger
ivralo pilot run ~/ivralo-pilot --out ~/pilot-results   # the report goes there
ivralo pilot run ~/ivralo-pilot --format json           # the report as one JSON object
ivralo pilot run ~/ivralo-pilot --fail-under 70         # exit 1 under 70% matched

Flag	Meaning
`--repository PATH`	The repository the revisions are in. Default: the workspace.
`--out PATH`	Where the report goes. Default: `<pack>/.pilot/<stamp>/`. It must be outside the repository (`.ivralo/` included); inside the pack it must be a dot directory at the pack's top.
`--format text` or `json`	Text prints the report's path and the summary line; JSON prints the whole report as one object.
`--hosted`	Lets a model that leaves the machine read the evidence.
`--fail-under P`	A bar from 0 to 100: exit 1 when fewer than P percent of the incidents matched.

The incidents run one after another. Each run's trace is drawn on stderr; stdout carries only the result.

Hosted models

The same privacy rule as ivralo incident applies. A model that leaves the machine is refused the evidence unless you pass --hosted. A hosted pilot also needs masking: IVRALO_INCIDENT_MASK=true and at least one identifier of your own in identifiers.json. Otherwise the pilot does not start. See Privacy and masking.

Scores and exit codes

Each incident comes to one of four:

Status	Means
matched	The report agreed with the known cause on everything it names. Shown as matched (outcome only) when the known cause was a `DIAGNOSED` naming no class and no file; the summary line then ends `; N scored on outcome only`.
missed	It did not. The reason is given, such as `outcome DIAGNOSED, expected NOT_CODE_ISSUE` or `the report does not cite src/svc/parsing.py`.
no report	The Investigator filed none, even after the one second request, and the reason is given.
not run	The incident's directory was incomplete, its revision is not in the repository, or the run failed before it could start.

Exit	Meaning
`0`	The report was written, whatever the incidents came to.
`1`	The report was written and the match rate is under `--fail-under`, said on stderr.
`2`	The pilot could not start: the repository, the pack, the format, an `--out` inside the repository, a bar outside 0 to 100, or a hosted model without `--hosted` and masking.

Under --fail-under every incident counts: a not-run incident counts against the bar, and one scored on outcome only counts for it. Read those before you trust a pass.

The pilot report

pilot.md and pilot.json hold every incident's status, its known cause and the Investigator's cause side by side. Each incident's own report.md, report.json and digest are copied beside them into <name>/.

Because the report holds every known cause, both files are readable by you alone (mode 0600 on Linux and macOS, an owner-only ACL on Windows). Nothing in the report leaves the machine unless you send it.