Observability

Before blaming the model

When a run went wrong, read what the model actually saw before changing a prompt or a model:

ivralo trace last

Most surprises are there in plain sight: a tool that failed, a policy that refused a command, a file that was not what the model expected, a request to the model that timed out.

Listing runs: ivralo runs

ivralo runs

The session's runs, newest first: what was asked, who did it, what it cost. -n sets how many (20 by default). Every run command takes --session (-s) to read another conversation's runs; ivralo sessions lists them.

Opening a run: ivralo trace

ivralo trace takes a run: last (the default) or the first eight characters or more of its id. It prints every tool call with its arguments and output, and the member that made it.

The HTML file has a timeline with one bar per tool call, every call with its complete output, and the answer. It inlines its style and loads nothing from the network, so it opens on a machine without ivralo, for example as an attachment to an issue:

ivralo trace last --html --out trace.html

In the session, /trace does the same for the last run.

A failed call says why

Every tool reports a failure in one shape, and the trace shows its code beside the call. The whole run's failures are counted on one line at the end:

  shell     rmdir /s /q .                         0.0s  failed REFUSED_BY_POLICY
  read_url  https://example.com/pricing           1.2s  failed NETWORK_BLOCKED

failures: 1 NETWORK_BLOCKED, 1 REFUSED_BY_POLICY

The list has a few more codes (RATE_LIMITED, INVALID_ARGUMENT, CONFLICT, RESOURCE_EXHAUSTED, PARSE_ERROR, PATCH_CONFLICT, VERIFICATION_FAILED), each named for what happened.

A failed request to the model

A request to the model that failed leaves a model row in the trace: the HTTP status, the provider's code, the request id, the start of the response body and how big the request was. A throttled endpoint (429), an unavailable one (503) and a dropped connection are asked again up to three times; a timeout once; a refused connection never. A 400 is a request the model cannot take, often a context that is too big, and is never repeated.

Exporting a conversation: ivralo export

ivralo export last --format md

Writes a whole conversation as one file: every turn, its tool calls and the replies. --format is html (one self-contained page, the default) or md. The file goes to .ivralo/exports/ in the workspace unless --out names a path. /export does the same in the session.

Where a claim came from

The trace shows everything a run did. To follow one number or sentence of the answer back to the file, the line or the tool output it rests on, use ivralo why. See Where a claim came from. To run a stored run again against what its tools answered last time, see Record and replay.

The machine: ivralo doctor

Beyond the services, ivralo doctor reads the machine:

• disk: free space on the workspace's drive (the row fails under 2 GiB, because history and the index are written there), how much history keeps, and the database's size;
• ollama: the server's version and, for each loaded model, how much of it the GPU holds. A model partly on the CPU fails its row: that is the usual reason every token is slow;
• tracing: whether spans leave the machine, and where.

See Doctor for every section.

OpenTelemetry

ivralo can send every run's spans (the team, each member, each model call, each tool call) to an OTLP/HTTP collector: Jaeger, Grafana Tempo, Arize Phoenix or the OpenTelemetry Collector.

It is off until you name an endpoint, because a span carries the prompt, every tool argument and every tool output. The packages are an optional extra, otel. From a checkout:

uv sync --extra otel

For a tool install, add it to the extras you install with, for example ivralo[pg,otel]. Then set the endpoint in .env:

IVRALO_OTEL_ENDPOINT=http://localhost:4318
IVRALO_OTEL_SERVICE=ivralo

An endpoint set without the extra installed is a warning at startup and a finding in ivralo doctor, never a crash: the run goes on untraced.