Investigate an incident

Before you start

ivralo incident runs inside the repository of the service the incident happened in. The repository must be a git repository: half of what an investigation establishes is about the code that was deployed.

Copy the evidence into a folder inside it, incident/ by default, then run the command from the repository's root. What a bundle may hold, and how it is read, is on The evidence bundle.

The first time ivralo runs in a folder it asks whether you trust it, as every command does. Without that yes the repository's own declarations, such as its language server or its # Evidence roots, are not used.

The command

ivralo incident [BUNDLE] [--format json] [--hosted] [--session ID]

A plain run prints two lines:

.ivralo/incidents/incident-20261012-101500/report.md
DIAGNOSED - class code

The second line is the report line: the outcome and, unless the outcome is ESCALATED, the class. Reading the report explains both.

What happens during a run

1. Checks. ivralo checks the workspace, the bundle and the model before anything starts. A refused bundle or a hosted model without permission stops it here, with one sentence on stderr.
2. A team of one. The team is the Investigator alone, under a leader that only routes the request to it. The leader holds nothing else.
3. The digest. The Investigator's first call reads the bundle. Code computes the digest and writes digest.json.
4. The judgement. The Investigator reads the digest and the code it points at, then files the report through one tool that checks every reference.
5. The report. report.md and report.json are written beside the digest, under .ivralo/incidents/<id>/.

The Investigator holds reads and that one filing. It cannot edit files, run Python or shell commands, query a database or send anything. It has a budget of 40 tool calls per run.

Questions at a terminal and in a pipe

At a terminal the Investigator may ask you something it cannot establish from the evidence, such as which deploy you mean. The question appears on stderr and you type the answer after your answer >. stdout stays reserved for the result.

In a pipe, or under a scheduler, nobody is there to answer. The team is built unattended and a question is refused; the Investigator files with what it has.

If nothing is filed

An Investigator that stops without filing, and without a refusal, is asked once more, never twice. The second request quotes its own last words and says why it is being asked again.

If the model's reply was cut at its output limit, the second request runs with the allowance doubled. The limit is IVRALO_MAX_TOKENS, or IVRALO_NUM_PREDICT on Ollama, and ivralo doctor shows it. A run whose model endpoint failed, one you stopped and one that ended on a refusal are not asked again; a model stream that broke mid-reply is.

Exit codes

Sessions

Each investigation is its own session, so incidents run one after another share nothing. The id is built from the bundle's name and the local time, with four random hex digits at the end:

incident-20261015-101500-3fa9              the default incident/ bundle
incident-night-of-the-12th-20261015-101500-3fa9   another bundle

The report folder is named after the bundle's path and the same stamp, without the hex digits: incident-20261015-101500 for incident/. Both stamps use the machine's local time, the clock ivralo runs shows. Timestamps inside the report stay in UTC.

--session shared joins the shared session instead. Within one session, a second filing for the same bundle with the same content is refused with the first report's line and path. A changed bundle or a new session starts a new investigation.

Following up

In text mode the session id is printed on stderr, together with three commands you can run as printed:

ivralo trace -s incident-20261015-101500-3fa9
ivralo why -s incident-20261015-101500-3fa9 last "KeyError: 'coupon'"
ivralo runs -s incident-20261015-101500-3fa9

• ivralo trace -s <id> draws the run: every tool call and its result.
• ivralo why -s <id> last "<claim>" follows a sentence of the report back to the bundle file and line it rests on. It exits 0 when the claim is fully traced, 1 when it is not, 2 when it cannot read the run. More on ivralo why.
• ivralo runs -s <id> lists the session's runs.

These commands follow the shared session unless you pass -s, so always name the incident's session. ivralo sessions lists it too, and ivralo --session <id> continues the conversation in the session.

From inside a session

In an ivralo session, /incident [bundle] runs the same investigation as a turn: the same checks before it starts, one routing request to the Investigator, and the report's path and report line as the turn's last line. It is refused while another run is in flight.

--hosted belongs to the command line only. The session's team was built with IVRALO_INCIDENT_HOSTED_MODEL as it stood when the session started, so to allow a hosted model there, set the variable and start the session again.

From the local API

POST /incident on the local API runs the same investigation and streams its progress. Its body takes bundle, hosted, session and attended; with attended true (the default) a question from the Investigator arrives as an approval the client answers. Alerts from Alertmanager and PagerDuty start investigations the same way; see Alert intake.