Docs / Incidents

The evidence bundle

An evidence bundle is the folder of logs and notes an investigation reads. This page covers what goes in it, how ivralo reads it, what it refuses, and how to let the Investigator search logs that git ignores.

What a bundle is

A bundle is a directory inside the workspace, the repository of the service that had the incident. It is incident/ unless you name another:

ivralo incident evidence/night-of-the-12th/

Every file under it is evidence, and nothing outside it is. ivralo never writes into a bundle.

What to put in it

Put in what the on-call engineer had while the incident was open:

incident/
  alert.md          the alert as it fired
  app.log           the service's log around the time
  deploy.log        what was deployed, and when
  metrics.csv       a few series, if you have them
  notes.md          what people said before the cause was known

Application logs. Plain text, JSON Lines, syslog (RFC 3164 or RFC 5424 times) and access logs are all read. Python, Java, Go and Node stack traces are recognised by the shape of their frames.
A deploy log. A line naming revision= or commit= is a deploy, and so is a line in plain words with a time, a verb such as deployed, rolled out or released, and a commit or a version: 10:01 checkout-service deployed commit 6ad91e2. Without deploys there is no timeline.
Metrics. A CSV, or a dashboard copied as a Markdown table. They are read when the logs show a resource running out.
A brief. A top-level INCIDENT or README file (.txt, .md or .rst, or no extension) saying what is being asked. Its sentences are treated as the bundle's brief, not as instructions; see Hostile evidence.

Leave out a postmortem or the fix, if they exist. The Investigator reads every file in the bundle.

How it is read

Rule	Detail
Text only	Each file is read as UTF-8 text, its lines numbered from 1.
Not text	A file with a NUL byte in its first 8192 bytes, or bytes that are not UTF-8, is listed and not read. A `.gz` log is one of these: unpack it first.
Links	A symbolic link that points outside the bundle is listed and not read.
Dot paths	Dot-files and dot-directories inside the bundle are skipped altogether.
Lines per file	At most 200,000. The digest says where reading stopped.
Bundle size	At most 50 MB in all. A file past that is listed and not read.

A reference into the bundle is file:line or file:first-last, relative to the bundle: app.log:160-170. The workspace-relative spelling, incident/app.log:160, is accepted too. Every claim in the report cites lines this way.

What is refused

Before anything starts, ivralo incident refuses, in one sentence on stderr:

a bundle outside the workspace, after links are resolved;
the workspace itself, or a folder above it;
a path through a dot-directory, such as .ivralo/, .git/ or incident/.old/;
a path that does not exist or is not a directory;
a bundle with no file that could be read;
a workspace that is not a git repository.

The command exits 2 on any of these.

Logs that git ignores

A bundle is usually ignored by git, and so are the service's own logs/. The incident tool reads the bundle whatever .gitignore says. The Investigator's search tools, glob_files and grep_files, follow git's list of files, so on their own they cannot search an ignored folder.

Name such folders under # Evidence in the workspace's IVRALO.md, one path or glob per line, relative to the workspace:

# Evidence

- incident/
- logs/
- archive/*/checkout.log

A root must stay inside the workspace. An absolute path such as /var/log/checkout/, a path that climbs out with .., or a link that resolves outside is reported as a problem by ivralo doctor, not used.

What a root does and does not do:

It adds the files under it to what glob_files and grep_files see, up to 50,000 files in all.
It never opens a closed path. .env, .ivralo/ and .git/ stay closed inside a root.
It is honoured only in a folder you trusted. A cloned repository's IVRALO.md does not decide what the model searches.
Nothing is a root by convention, not even incident/. Declare it if you want the Investigator to search the files the digest was made from.

ivralo doctor has an evidence section that lists the roots, how many files each adds, and any problem.

The Investigator runs no shell commands. It reads the bundle through its own tool and the repository through the file and code tools.

Bundles written by alerts

When alert intake takes in a firing alert, it writes a bundle of one file into the workspace:

alerts/alertmanager-5a1f0c3e9b2d7e44-20261023T094315Z-7c/alert.md

The folder name is the source, the alert's fingerprint (a PagerDuty incident's id), the time it was taken in (UTC) and two random hex digits, so a new delivery never overwrites an earlier bundle. The file holds the alert's name and status, its source and fingerprint, when it started and ended, then sections for its labels, its annotations, its links and the delivery. Every value is the sender's text, one field to a line, cut at 2000 characters. Links are written as text and never fetched.

A stub is an ordinary bundle. Copy logs or a deploy log into its folder and investigate it again with them:

ivralo incident alerts/alertmanager-5a1f0c3e9b2d7e44-20261023T094315Z-7c/

Stubs live in the working tree, not under .ivralo/, because a bundle under a dot-directory is never evidence. Add alerts/ to .gitignore to keep them out of git status.