Docs / Incidents
Reading the report
What an incident report holds and how to read it: the facts code computed, the one judgement the Investigator made, how the outcome is decided, and the JSON files beside the Markdown for your own tools.
Where it is
Every investigation writes one folder:
.ivralo/incidents/incident-20261012-101500/
digest.json the facts, computed by code before any model read anything
report.json the report as data
report.md the report for people
.ivralo/ is ignored by git, so none of it ends up in a commit. ivralo's own credentials are hidden in both report files before they are written. A secret that the logs themselves contain, such as a token in a request line, can still be quoted.
The digest: facts computed by code
Before the Investigator reads anything, code reads the bundle into a digest. Every number in the report comes from it.
| Part | What it holds |
|---|---|
| Signatures | Each error grouped by its text with the parts that change from line to line folded away (ids, numbers, quoted values, timestamps), plus its stack frames. 340 failing lines become one signature with a count, a first and a last occurrence, the endpoints and a sample. |
| Timeline | The deploys found in the bundle, and for each signature how many minutes after the last deploy it began, whether a deploy happened while it was failing, or whether it began before any deploy. |
| Revision and suspects | Whether the repository holds the deployed commit, what that deploy changed (with the diff), which files the stack frames name, whether the repository holds them, whether the line a frame printed is the line the repository has at that number, and which files are both on a trace and changed by the deploy. |
| Exhaustion lead | When a line says the process hit its own limit (out of memory, no space left on device, too many open files or clients, a pod's OOMKilled): the routes served in the five minutes before, how the resource and the traffic moved in the metrics, and, with a code index, the files that serve each route. A limit being set is not a limit reached. |
| Reported context | Pages and routes named in the brief or a note, each with its source. |
| Not established | What the digest could not settle, such as a deploy the repository does not hold. |
The lead is the signature the investigation starts from. Every list in the digest says how much it left out.
The judgement
The Investigator reads the digest and the code it points at, and decides two things:
- the class:
code,config,data,infrastructure,externalorambiguous; - the cause, and whether the evidence establishes it.
What "established" requires depends on the class:
| Class | Established when the evidence shows |
|---|---|
code | A file the repository holds whose code produces the behaviour a cited line shows. No log line needs to name the file; the report prints the repository's line beside the evidence so you can judge the match. |
config | A line naming the setting and its change or absence. |
data | The record and the rule it broke. |
infrastructure | The resource and its state. |
external | The provider's own line. |
Every class needs at least one cited line of the bundle. The Investigator files through one tool, file_incident_report, which checks every reference against the bundle and the repository. A reference that does not resolve is refused, and the Investigator files again.
The tool also checks the cause's quantifiers against the digest's counts. "All errors are on GET /api/orders", when the digest counts errors on another endpoint, is refused like a broken reference. "Most" or "several" that the counts do not support is filed and changes neither whether the cause is established nor the outcome; ivralo notes it under Not established with the computed counts.
The outcome
Code, not the model, turns the judgement into one outcome. The rules are applied in this order:
- The Investigator reported a line addressed to the reader:
ESCALATED. - The class is
config,data,infrastructureorexternal, established, with at least one cited line:NOT_CODE_ISSUE. - The class is
code, established, with a cited line, and at least one suspect file the repository holds:DIAGNOSED. - Anything else, including
ambiguous:INSUFFICIENT_EVIDENCE.
There is no confidence number. Whether the repository holds the deployed revision is reported as a fact and changes no outcome.
report.md, section by section
| Section | What it holds |
|---|---|
| Outcome | One line: the outcome and the class, such as DIAGNOSED - class code. ESCALATED has no class. |
| Observed | The computed facts, one per line, each followed by its source: app.log:160-170, deploy.log:3, a commit. |
| Inferred | The class and the cause, each marked inferred, the suspect files, and the cited lines copied from the bundle by code, never from the model's text. For a suspect file, the repository's code at that line is printed too. |
| Hypotheses | Only when the Investigator kept a ledger (below). |
| Not established | What the evidence could not settle. Each line says who wrote it: (code) or (investigator). |
| Instructions found in the evidence | Only when there are any: each line quoted, marked not followed, or marked "not an instruction to the reader" when it was set aside. See Hostile evidence. |
| Evidence index | Every reference the report uses. |
Counts, times and gaps appear only in Observed and in ivralo's own lines under Not established. "The deploy preceded the errors" is an observed fact; the cause is inferred, and says so.
The hypothesis ledger
A report names one cause. The optional ledger says what else the Investigator weighed, so a reviewer sees what was ruled out, and an INSUFFICIENT_EVIDENCE report says which causes stayed open.
| Status | Means | Needs a cited line |
|---|---|---|
| validated | The evidence shows it. | yes |
| invalidated | The evidence shows it is not so. | yes |
| inconclusive | Weighed, and the evidence settles neither way. | no |
| checked healthy | A component looked at and found healthy. | yes |
| not checked | Named, and not looked at. | no |
An entry that needs a line and cites none, or cites one that does not resolve, is left out, and Not established says which and why. The ledger holds at most 20 entries. It never changes the outcome; it explains it.
ivralo why -s <id> last "the database was down" follows one hypothesis to the lines it cites.
report.json
The same report as data. The main keys:
| Key | Holds |
|---|---|
format | The format version, 1. |
incident | The incident id, the folder's name. |
outcome, class, established | The outcome, the class and whether the cause was established. |
bundle, digest | The bundle's path and the path of digest.json. |
lead | The lead signature: id, text, count, first, last, kind (exhaustion or empty). |
signatures, timeline | Every signature (id, level, count, text) and every moment (time, kind, label, ref). |
deploy, revision | The deploy the lead is placed against, and whether the repository holds its commit (held). |
changed, changed_range, on_trace_and_changed, frames | What the deploy changed and how the stack frames compare with the repository. |
suspects | The suspect files: path, line, held, note, code. |
cause, evidence, commits | The cause, and each cited reference with its lines copied from the bundle. |
hostile, set_aside | Lines addressed to the reader, and lines set aside as not instructions. |
hypotheses | The ledger; [] when there is none. |
not_established | Entries of {"by": "code" or "investigator", "text": ...}. |
run_id, session_id, model, filed_at | Which run filed it, in which session, on which model, and when (UTC). |
digest.json
The digest in full, with more than the report carries: bundle (every file with its size, line count and whether it was read), signatures with their frames and endpoints, deploys, moments, placements (each signature's gap to its preceding deploy), correlation (the repository facts, including the deploy's diff), reported_context, limits (how much each list shows) and not_established.
Use report.json for the result and digest.json when you need to see what the Investigator was given.