Skip to content
ivralo
Sections

Docs / Incidents

Reading the report

What an incident report holds and how to read it: the facts code computed, the one judgement the Investigator made, how the outcome is decided, and the JSON files beside the Markdown for your own tools.

Where it is

Every investigation writes one folder:

.ivralo/incidents/incident-20261012-101500/
  digest.json     the facts, computed by code before any model read anything
  report.json     the report as data
  report.md       the report for people

.ivralo/ is ignored by git, so none of it ends up in a commit. ivralo's own credentials are hidden in both report files before they are written. A secret that the logs themselves contain, such as a token in a request line, can still be quoted.

The digest: facts computed by code

Before the Investigator reads anything, code reads the bundle into a digest. Every number in the report comes from it.

PartWhat it holds
SignaturesEach error grouped by its text with the parts that change from line to line folded away (ids, numbers, quoted values, timestamps), plus its stack frames. 340 failing lines become one signature with a count, a first and a last occurrence, the endpoints and a sample.
TimelineThe deploys found in the bundle, and for each signature how many minutes after the last deploy it began, whether a deploy happened while it was failing, or whether it began before any deploy.
Revision and suspectsWhether the repository holds the deployed commit, what that deploy changed (with the diff), which files the stack frames name, whether the repository holds them, whether the line a frame printed is the line the repository has at that number, and which files are both on a trace and changed by the deploy.
Exhaustion leadWhen a line says the process hit its own limit (out of memory, no space left on device, too many open files or clients, a pod's OOMKilled): the routes served in the five minutes before, how the resource and the traffic moved in the metrics, and, with a code index, the files that serve each route. A limit being set is not a limit reached.
Reported contextPages and routes named in the brief or a note, each with its source.
Not establishedWhat the digest could not settle, such as a deploy the repository does not hold.

The lead is the signature the investigation starts from. Every list in the digest says how much it left out.

The judgement

The Investigator reads the digest and the code it points at, and decides two things:

  • the class: code, config, data, infrastructure, external or ambiguous;
  • the cause, and whether the evidence establishes it.

What "established" requires depends on the class:

ClassEstablished when the evidence shows
codeA file the repository holds whose code produces the behaviour a cited line shows. No log line needs to name the file; the report prints the repository's line beside the evidence so you can judge the match.
configA line naming the setting and its change or absence.
dataThe record and the rule it broke.
infrastructureThe resource and its state.
externalThe provider's own line.

Every class needs at least one cited line of the bundle. The Investigator files through one tool, file_incident_report, which checks every reference against the bundle and the repository. A reference that does not resolve is refused, and the Investigator files again.

The tool also checks the cause's quantifiers against the digest's counts. "All errors are on GET /api/orders", when the digest counts errors on another endpoint, is refused like a broken reference. "Most" or "several" that the counts do not support is filed and changes neither whether the cause is established nor the outcome; ivralo notes it under Not established with the computed counts.

The outcome

Code, not the model, turns the judgement into one outcome. The rules are applied in this order:

  1. The Investigator reported a line addressed to the reader: ESCALATED.
  2. The class is config, data, infrastructure or external, established, with at least one cited line: NOT_CODE_ISSUE.
  3. The class is code, established, with a cited line, and at least one suspect file the repository holds: DIAGNOSED.
  4. Anything else, including ambiguous: INSUFFICIENT_EVIDENCE.

There is no confidence number. Whether the repository holds the deployed revision is reported as a fact and changes no outcome.

report.md, section by section

SectionWhat it holds
OutcomeOne line: the outcome and the class, such as DIAGNOSED - class code. ESCALATED has no class.
ObservedThe computed facts, one per line, each followed by its source: app.log:160-170, deploy.log:3, a commit.
InferredThe class and the cause, each marked inferred, the suspect files, and the cited lines copied from the bundle by code, never from the model's text. For a suspect file, the repository's code at that line is printed too.
HypothesesOnly when the Investigator kept a ledger (below).
Not establishedWhat the evidence could not settle. Each line says who wrote it: (code) or (investigator).
Instructions found in the evidenceOnly when there are any: each line quoted, marked not followed, or marked "not an instruction to the reader" when it was set aside. See Hostile evidence.
Evidence indexEvery reference the report uses.

Counts, times and gaps appear only in Observed and in ivralo's own lines under Not established. "The deploy preceded the errors" is an observed fact; the cause is inferred, and says so.

The hypothesis ledger

A report names one cause. The optional ledger says what else the Investigator weighed, so a reviewer sees what was ruled out, and an INSUFFICIENT_EVIDENCE report says which causes stayed open.

StatusMeansNeeds a cited line
validatedThe evidence shows it.yes
invalidatedThe evidence shows it is not so.yes
inconclusiveWeighed, and the evidence settles neither way.no
checked healthyA component looked at and found healthy.yes
not checkedNamed, and not looked at.no

An entry that needs a line and cites none, or cites one that does not resolve, is left out, and Not established says which and why. The ledger holds at most 20 entries. It never changes the outcome; it explains it.

ivralo why -s <id> last "the database was down" follows one hypothesis to the lines it cites.

report.json

The same report as data. The main keys:

KeyHolds
formatThe format version, 1.
incidentThe incident id, the folder's name.
outcome, class, establishedThe outcome, the class and whether the cause was established.
bundle, digestThe bundle's path and the path of digest.json.
leadThe lead signature: id, text, count, first, last, kind (exhaustion or empty).
signatures, timelineEvery signature (id, level, count, text) and every moment (time, kind, label, ref).
deploy, revisionThe deploy the lead is placed against, and whether the repository holds its commit (held).
changed, changed_range, on_trace_and_changed, framesWhat the deploy changed and how the stack frames compare with the repository.
suspectsThe suspect files: path, line, held, note, code.
cause, evidence, commitsThe cause, and each cited reference with its lines copied from the bundle.
hostile, set_asideLines addressed to the reader, and lines set aside as not instructions.
hypothesesThe ledger; [] when there is none.
not_establishedEntries of {"by": "code" or "investigator", "text": ...}.
run_id, session_id, model, filed_atWhich run filed it, in which session, on which model, and when (UTC).

digest.json

The digest in full, with more than the report carries: bundle (every file with its size, line count and whether it was read), signatures with their frames and endpoints, deploys, moments, placements (each signature's gap to its preceding deploy), correlation (the repository facts, including the deploy's diff), reported_context, limits (how much each list shows) and not_established.

Use report.json for the result and digest.json when you need to see what the Investigator was given.