Durable Monitors

Some work does not finish inside the turn that started it — a CI run, a MUR fleet run, a Codex or Claude Code subprocess. A durable monitor keeps checking until the source gives a real answer, across daemon restarts, and tells you once when something is worth your attention.

mur monitor add --file wait-for-ci.yaml   # validates, probes once, registers
mur monitor list                          # what still needs attention
mur monitor show <id> --history           # evidence + append-only history
mur monitor cancel <id>                   # stop watching (never cancels the work)
mur monitor retry <id>                    # bring an exhausted monitor back

State lives in SQLite at ~/.mur/monitor/monitors.db. The daemon polls due monitors every 15 seconds under fenced leases, so two daemons cannot both claim one monitor, and a restart catches up the checks it missed rather than losing them.

unknown is never failed

The distinction the whole feature is built on: not being able to read a source is a monitor problem, not a work problem.

The API rate-limited us. The run is not there yet. The token was rejected. The process is gone with no exit record. Each of those is reported as unknown — never as failed. A monitor that says your build failed means your build failed.

Deadlines (stalled 20m · soft 3h · hard 8h) count from when the work really started, not from when you registered the monitor.

It tells you once

Routine polling says nothing at all. MUR speaks up when a monitor first stalls, when progress resumes, when it crosses its soft deadline, when it settles, when the monitor itself goes unhealthy, and when it gives up — once each, not once per poll.

Every message names the monitor, its source, what is known, when the next check is, and the single next step. They go to the daemon log always, and to an OS notification if you opt in:

# ~/.mur/config.yaml
notifications:
  desktop: true   # default false

Desktop is off by default on purpose: a background daemon that starts popping banners the moment you upgrade is a hostile default. The log channel cannot be switched off, so a notable event always leaves a trace somewhere.

mur monitor show reports each notification's delivery state, so a notification that failed is visible rather than silently lost.

Acting on what it finds

A monitor can act when the work settles. Which actions it may take is written in its spec:

actions:
  on_failure:
    - type: collect_logs
    - type: rerun

Reading-only steps run unattended — recording evidence, sending the notification. Anything that would change something outside MUR stops and asks. The request is pinned to that exact action, and approving a different one never releases it:

mur monitor show 01a0a75e            # prints the exact command, including the id
mur channel approve monitor-<id> <hitl-id>

Approvals defer, they do not time out. A monitor waiting on you waits — it does not eventually decide to act alone, and waiting is not counted as a failed attempt. MUR gives up after three remedies that did not fix anything and marks the monitor exhausted, rather than retrying forever.

The risk tier of an action comes from a fixed table keyed on the action type — never from the action's own parameters, and never from a model. An action type MUR does not recognise is treated as the most restrictive tier and has no implementation, so it is both gated and unrunnable.

Restarting a failed CI run

rerun is the one action that writes to the outside world, and it needs its own credential:

source:
  type: github_actions
  reference: owner/repo/123
  credential_ref: keychain:mur/github-default        # read-only: observing the run
  write_credential_ref: keychain:mur/github-write    # required for rerun

mur monitor add refuses a spec that asks for rerun without the write grant — at creation, not after you have already approved something that was never going to run.

They are separate because approving an action is not the same as consenting to a standing capability. A token handed over so MUR could watch a run can usually restart it too; the grant makes that a decision you make rather than one your token makes for you.

A rerun restarts only the jobs that failed, not the whole run. The new run it starts is not itself monitored — register a second monitor if you want that one watched. A rerun MUR could not dispatch is a failed remediation attempt, not a verdict on the original work.

When the rules run out

A monitor's spec lists what to do about a failure. Sometimes that list is empty, and sometimes everything in it fails. MUR can ask a model what to do next — off by default:

# ~/.mur/config.yaml
monitor_resolver:
  enabled: true          # opt-in; off, nothing in this path runs
  model: claude_haiku    # optional — otherwise the backend `mur chat` resolves

Off is the default deliberately. Switching it on lets a background daemon send a monitor's context to a model on its own schedule, with nobody watching, and that is a decision to make once and on purpose rather than one an upgrade makes for you. While it is off, no request leaves the machine.

It is consulted in exactly two situations, both of them "the structured rules could not settle this":

• every remedy the spec listed for that failure ran and failed
• the spec listed no remedy for that outcome at all

One consultation per observation cycle, not per check. What it is shown — the source, the outcome, the error, what was already tried, a bounded tail of any logs collected — goes through the same redaction every other thing MUR writes passes through, and secrets are replaced before anything is truncated, so a cut can never leave half a credential behind.

What a model is allowed to say

Very little, on purpose:

So a model proposing rerun produces exactly the pinned approval request a spec-declared rerun would, and you release it the same way. A proposal is not a shortcut around the gate — it is an entry into it.

A consultation that fails, or whose answer was refused, is recorded and changes nothing. The monitor settles without advice: not getting advice is not the same as the work having failed, which is the same principle that makes an unreadable source unknown rather than failed.

Registering without typing

mur fleet run registers a monitor for its own run. Registration never fails the run: if it cannot complete, the run still proceeds and says plainly that the work is running but not monitored, naming the run id, and retries from a durable outbox on later ticks.

Kicking off downstream work and applying a known remedy are recognised action types that do not execute yet. Approving one is recorded and MUR says it cannot carry it out — rather than pretending it did.