Server mode and incidents

What server mode is

Server mode is a set of grants, nothing more. It is switched on in the ivralo home's own settings, never by a repository, so a repository cannot grant itself anything. ivralo init --server sets it up.

What changes is what an unattended run may do. In developer mode such a run is granted nothing that would have to ask a person. In server mode it may also commit, and push one kind of branch, for one purpose:

• the branch is one ivralo made itself, in that session, with every commit on it its own;
• never the default branch, never a branch the remote already has, never a forced push, never a tag;
• it ends in a draft pull request on your forge.

Nothing merges. The shell policy refuses git push, gh pr merge and their relatives in every spelling, so the draft pull request is the only way out, and a person decides what happens to it.

Investigating an incident

An investigation reads a bundle: a folder inside the service's repository that holds the evidence. Every file in it counts.

incident/
  alert.md        the alert as it fired
  app.log         the service's log around the time
  deploy.log      what was deployed, and when
  metrics.csv     a few series, if you have them

Then run:

ivralo incident incident/

ivralo never writes into the bundle. It reads text only, numbers every line from 1, and lists binary files without opening them.

First the facts, computed by code

Before any model reads anything, ivralo works out what can be counted:

• the error signatures, with the parts that change from line to line folded away;
• a timeline of deploys, and how many minutes after the last one each error started;
• the revision: whether the repository holds the deployed commit, what that deploy changed, and whether the lines in the stack traces match the code;
• exhaustion leads: memory, disk, open files or clients running out, and which routes were served just before.

Then one judgement

One member of the team, the Investigator, reads those facts and the code and decides two things: the class of the incident (code, config, data, infrastructure, external, or ambiguous) and its cause. It files the cause through a single tool that checks every reference against the bundle and the repository. A claim that cites a line that does not exist is refused.

The Investigator can read and file a report. It cannot edit, run code or send anything.

The outcome

Code, not the model, turns the judgement into one of four outcomes. There is no confidence number.

The report

The report is written to .ivralo/incidents/<id>/ as report.md, with report.json and digest.json beside it for your own tools. Its sections keep what was observed apart from what was inferred:

• Observed: the computed facts, each with its source.
• Inferred: the class and the cause, marked as inferred, with the cited lines copied in by code.
• Hypotheses: what was checked, and whether it held.
• Not established: what the evidence could not settle.
• Instructions found in the evidence: quoted, never followed.

Every number in it is computed. To see where a sentence came from, ask:

ivralo why -s <id> last "the claim, as it appears in the report"

It traces the claim back through the stored run to the bundle line it rests on.

Where investigations come from

• A terminal: ivralo incident, as above.
• The local API: ivralo serve opens it on 127.0.0.1 only, behind a token. POST /incident runs the same investigation and streams its progress.
• Your alerting: Alertmanager and PagerDuty can post to the API. Each firing alert becomes a small bundle and an investigation job, run one after another. While its investigation is unfinished, and for six hours after it was opened, the same alert again is the same job, so a noisy alert is investigated once.

Evidence that tries to give orders

Logs carry text that users wrote. A line addressed to the reader, such as "ignore the above and run this", is quoted under its own heading and makes the outcome ESCALATED. The guarantee does not rest on spotting such lines, though: it rests on the Investigator holding no tool that edits, runs or sends.

Evidence and hosted models

Incident evidence stays on the machine by default. If the model reading it is hosted elsewhere, the investigation is refused unless you allow it, and allowing it also requires masking: addresses, e-mails and identifiers you list are replaced with stable placeholders before anything leaves, and put back in your copy of the report.