Jobs

What a job is

A job is one row in the jobs table of ivralo's database, beside the sessions and runs. The row says what the job is, where it has got to, which process is running it and what it is waiting for.

There is no daemon. A job runs in the process that starts it, and the database is the only thing that lasts. If that process dies, the row is still there, and another process can pick the job up.

Alert investigations are jobs too. When Alertmanager or PagerDuty posts to the local API, each firing alert becomes a small evidence bundle under alerts/ in the workspace and a queued request job that investigates it. They run one after another in the ivralo serve process that took them in, and ivralo jobs lists them like any other job. See Alert intake.

Starting a job

Add --job to plan, ask --plan or ask. The job's row is written first and its id printed, then the work runs in the same terminal:

ivralo plan --job "add a /health endpoint with a test"

job 3f9a1c2e started

From that line on, a closed terminal, a Ctrl+C or a reboot leaves a job that ivralo resume 3f9a1c2e continues. plan --job runs the drafted plan at once; ask --plan --job shows the plan and asks first (--yes in a pipe).

A job runs unattended. Where a plain ivralo ask would refuse a call that a person must approve, a job parks and waits instead.

The states

queued -> running -> waiting_for_approval -> running -> completed
                  -> failed
                  -> cancelled
queued -> blocked (another job holds the workspace) -> queued

Every move is checked against this machine and written to the job's log with the time and the reason. ivralo job <id> prints the log.

Parking: waiting for a person

A job pauses where a person must say yes: an edit (unless permissions.json's allow_tools names it), a shell command the policy asks about, or a send. Reads, Python, allowed commands and denied commands behave as they always do. A question to you (ask_user) is declined, since nobody is there, and never parks.

When a call needs you, the job records it as its question (what it would do, its effects and its risk), becomes waiting_for_approval, gives the workspace back and exits with code 0:

job 3f9a1c2e parked
it waits for a person: send_email - ivralo wants to send: send_email  to=boss@example.com
answer it: ivralo resume 3f9a1c2e --approve (this call) | --approve-job (and every edit of this job) | --reject NOTE

Answer it with one of three flags:

The process that answers continues the job. The paused run is read back from the database and goes on from the held call, so the work before it is not done again. ivralo resume <id> with no answer prints the question and exits 1, changing nothing.

Nothing but your answer grants a parked call: no flag, setting or retry. A job carries no pre-approval, so a send always parks. A parked job holds no lease, so another queued job in the workspace may run meanwhile.

The lease

One job at a time runs in a workspace. The running job holds the workspace's lease: its row names the process (pid and host) and a heartbeat the process refreshes every IVRALO_JOB_HEARTBEAT_SECONDS (10 by default).

A lease whose heartbeat is older than IVRALO_JOB_LEASE_SECONDS (60 by default), or whose process is gone from this machine, is stale, and the next process takes it over. A crash never leaves a workspace locked.

A request to the local API holds the lease the same way, through an api job. A job started meanwhile is blocked and queued again when the request ends. Only the API runs an api job: ivralo resume refuses one, and ivralo jobs run never picks one.

Resuming after a crash

A job whose process died stays running with a lease nobody holds. ivralo resume <id> takes it over:

step 3 restarts; the attempt before it left: db.py changed (their versions from before are in history)

Steps already delivered stay done. The step in flight runs again from its start. What the interrupted attempt changed stays in the workspace: nothing is rolled back on its own, and ivralo restore takes it from there. A request job runs its request again and says so.

The commands

ivralo jobs run is what the operating system's scheduler calls. With nothing to run it says so and exits 0. A schedule made with --jobs registers it, see Scheduled requests. The commands write the result to stdout; an unknown id, a finished job or a database that is down goes to stderr with exit 1, so they work in scripts.

Notifications

With Telegram or Slack configured (IVRALO_TELEGRAM_BOT_TOKEN and IVRALO_TELEGRAM_CHAT_ID, or IVRALO_SLACK_WEBHOOK_URL), a job sends one line when it parks, completes or fails:

3f9a1c2e parked: send_email - ivralo wants to send: send_email  to=boss@example.com; ivralo resume 3f9a1c2e --approve
3f9a1c2e completed
3f9a1c2e failed: step 2 did not deliver: b.md (absent)

The line is written by ivralo, never by the model. Without a channel nothing is sent, and ivralo jobs is where it shows.

In the session

The session reads this workspace's jobs every five seconds. Its top bar names one: a parked job first, else the running one and its step. /jobs lists them; /jobs <id> on a parked job opens the approval dialog with the three answers a job takes; /plan --job drafts a plan and runs it as a job once you approve it. The session never runs a job itself: an answer starts ivralo resume in a process of its own, so closing the session changes nothing for the job.