Alert intake

How it works

ivralo serve runs the local API in the service's repository. Two of its routes take alerts:

For each firing alert in a delivery, ivralo:

1. writes a stub bundle, alerts/<source>-<fingerprint>-<time>/alert.md, holding the alert's name, status, start and end, labels, annotations and links (what a stub holds);
2. makes one investigation job over it, in a session of its own;
3. answers at once, and runs the jobs one after another in this serve process.

The job runs on the same narrow team as ivralo incident: the Investigator alone, under a leader that can only hand the request to it. Its report lands where ivralo incident puts one, under .ivralo/incidents/. ivralo jobs and GET /jobs/<id> show the job; see Jobs.

Nothing is fetched. A runbook URL, a generator URL or an incident's page is written into the stub as text.

Starting the server

ivralo serve

It listens on 127.0.0.1:61017 only (--port or IVRALO_API_PORT changes the port) and runs in the foreground until Ctrl+C or SIGTERM. It is not installed as a service; run it under your own service manager if you want it kept up. Every route except GET /version needs two headers:

• Authorization: Bearer <token>, the token in <home>/api-token, made by ivralo init --token or by the first serve;
• X-Ivralo-Api-Version: 2.

Alertmanager

Add a receiver whose http_config sends both. Alertmanager reads the token from the file, so the token never appears in its config:

receivers:
  - name: ivralo
    webhook_configs:
      - url: http://127.0.0.1:61017/alerts/alertmanager
        max_alerts: 100
        http_config:
          authorization:
            type: Bearer
            credentials_file: /home/ops/ivralo/api-token
          http_headers:
            X-Ivralo-Api-Version:
              values: ["2"]

max_alerts keeps a grouped delivery under ivralo's bounds (below). Alertmanager must run on the same machine, since the API answers loopback only; the file must be readable by the user Alertmanager runs as.

An alert is known by its fingerprint. A body from an Alertmanager older than 0.19, which sends none, gets one computed from the alert's sorted labels.

PagerDuty

PagerDuty cannot reach a loopback address. Run a small relay on this machine that receives PagerDuty's webhook, verifies PagerDuty's signature, and posts the body to http://127.0.0.1:61017/alerts/pagerduty with the token and the version header. ivralo does not verify PagerDuty's signature itself; the relay must.

Only incident.triggered and incident.reopened start an investigation. A PagerDuty incident is known by its id. PagerDuty's pagey.ping test event answers 200 with no alerts.

The answer

The answer comes at once, as one JSON object:

{"source": "alertmanager", "alerts": [
  {"key": "alertmanager:5a1f0c3e9b2d7e44", "name": "CheckoutErrorRate", "status": "firing",
   "outcome": "started", "job": "3fa2c91b",
   "bundle": "alerts/alertmanager-5a1f0c3e9b2d7e44-20261023T094315Z-7c/"}]}

Each alert's outcome is started, duplicate (the job that already stands for it is named) or not_firing.

Bounds

The same alert twice

An alert's key is its source and fingerprint, such as alertmanager:5a1f0c3e9b2d7e44. A job stands for that key:

• while it is unfinished, however old;
• and for IVRALO_ALERT_DEDUP_SECONDS after it was made (21600, six hours, by default).

While a job stands, the same alert again is answered as duplicate and starts nothing. Alertmanager's repeats and a storm of deliveries make one investigation. Two deliveries at once are checked under a database lock, so they make one job too.

A duplicate whose job is still queued, because serve was stopped before it got to it, hands that job to this server to run and answers 202. A restart loses no alert. A job that is running, waiting on a question or finished is left where it is; ivralo resume <id> continues a stopped run.

Stubs on disk

Stubs are files in the repository's working tree, like any bundle. Add the folder to .gitignore so they stay out of git status:

alerts/

They do not pile up. A workspace keeps its newest IVRALO_ALERT_STUBS_KEEP stubs (200 by default; 0 keeps them all), and writing a new one removes the oldest past that. It never removes a stub an unfinished job still reads, a stub you added a file to, or a folder under alerts/ that the intake did not name.

To investigate an alert again with more evidence, copy the logs into its stub's folder and run ivralo incident alerts/<stub>/.