A reliability policy is the one thing you author to make a pipeline observable: a set of rules, each watching some subjects and describing a condition that means trouble. From that single declaration the platform derives everything else — a rolled-up health state per subject, durable incidents when a condition persists, and notification routing when one opens. You declare intent once; the derivations are deterministic and event-sourced, so they replay identically and never drift. The policy lives at project.dsl/reliability as a single document:

A rule

  • subjects — what the rule watches; at least one. Each is a ReliabilitySubjectRef with kind = node (node_id, optional output_port), graph (graph_id), or semantic_relationship (semantic_relationship_id).
  • kind — the condition kind: run, assertion, freshness, or semantic_verification. Exactly the matching condition block must be present.
  • severity — the health severity when the condition is met: degraded (still usable, worth attention) or unhealthy (broken).
  • enabled — omit or true to arm; explicit false disables the rule without deleting it.
  • id / definition_hash — server-owned; leave unset when authoring. The id is content-addressed from the rule’s definition and stays stable across re-saves, so toggling enabled or renaming never churns evaluation history.
  • name, description, owner — display and accountability only; excluded from rule identity.

Condition kinds

run — fires when a subject accumulates N consecutive failed terminal runs.
By default only scheduled runs count as evidence — a manual re-run you’re actively debugging shouldn’t trip production health. Widen the evidence set with origins (scheduled, manual, api, webhook, email, retry, backfill); an empty list means scheduled-only. assertion — fires on failed data assertions attached to the subject.
min_severity is warn or error (default error); assertion_ids narrows to specific assertions, empty means any. freshness — fires when data misses a deadline, without needing a run to fail.
mode is schedule — deadlines follow the subject’s own cron triggers, so a 6am job that doesn’t produce by 6:15am (with a 15-minute grace) is late — or max_age, where you set max_age_seconds for an explicit staleness ceiling. grace_seconds pads every deadline; timezone (IANA, default UTC) is used for the deadline arithmetic so DST is handled correctly. Freshness rules watch nodes, not whole graphs. semantic_verification — fires when a subject’s semantic checks fail; the condition block is empty ({}).

Data assertions

An assertion-kind rule reacts to failed data assertions, which you author separately on the node whose output you want checked — on its config.assertions. Assertions run after the node executes and produce pass/fail evidence the reliability policy can watch:
Assertion types and their config shapes: Each assertion has a severity of error or warn and a server-minted, content-addressed id (leave id unset when authoring). An assertion-kind reliability rule reacts to any failing assertion on its subjects by default, or to specific ones via assertion.assertion_ids, gated by assertion.min_severity.

Health is derived

You never write health — the platform derives it. Each terminal run (and each missed freshness deadline) is evaluated against the pinned policy, producing an observation only on a transition (healthy → degraded, degraded → unhealthy, recovered). Between transitions, repeated confirmations compact rather than duplicate. The per-subject state rolls up by worst-severity precedence and is readable at project.dsl/health (read-only). Because evaluation is pinned to the exact (policy version, rule id, definition hash) that was live when the run committed, a later policy edit never rewrites past health.

Incidents are correlated

When a rule’s condition persists, matching observations correlate into a durable incident — a single, stable record you acknowledge, resolve, and that recurs sensibly, rather than a stream of raw alerts. Configure it per rule:
  • group_bysubject (default: one incident per failing subject) or rule (one incident spanning all the rule’s subjects, resolving only when none still fail).
  • resolutionauto_on_recovery (default: the incident resolves itself when the subject recovers) or manual (a human must resolve it; it never auto-reopens — a later failure opens a fresh incident).
  • suppression_seconds — quiet notifications for this window after the incident opens.
  • recurrence_seconds — a failure within this window of a recovered resolution reopens the prior incident instead of opening a new one (default ~7 days).
Incidents freeze their affected-downstream set at open time and are readable at project.dsl/incidents. You never author an incident (the scope rejects a direct write) — you move its lifecycle with an ops edit on project.dsl/incidents/<incident_id>:
Acknowledge / resolve / reopen are operate-level actions (the same privilege as cancelling or retrying a run). A manual resolve reason is one of accepted_risk, false_positive, subject_retired, or superseded; recovered is system-only — the correlator mints it when the subject actually recovers.

Routing notifications

Declare named destinations on the document and reference them from a rule’s routes:
A destination is webhook (an HTTPS url plus an optional auth_secret_ref — a reference to a stored secret, never an inline credential) or email (a to list). A route points a rule’s incident notifications at a destination, optionally gated by min_severity. Destinations carry only secret references, so no credential ever lands in the policy document.

Branch lifecycle routes

Branch and merge lifecycle moments route through the same destinations, but from a document-level branch_routes list rather than from a rule — a merge, a stale plan or a rolled-back publication is produced by the branch tier itself, not by any rule’s subject evaluation.
Opt-in is explicit and has no default: a branch_routes entry delivers only the signals it names, and an empty signals list delivers nothing. This is deliberately the opposite of a rule route, where an empty list still means incident_opened — a channel you configured for incidents should not start carrying merge traffic because a new signal family shipped. The available signals are merge_completed, merge_plan_stale, merge_conflicted, branch_effect_failed and publication_rolled_back. Branch notifications carry identifiers, not narrative. The body names the branch and whichever correlation id applies — a merge request and plan digest, an effect invocation’s idempotency key and its disposition, a publication and revision — so a recipient looks the detail up under their own authority. Rejection messages, remote system responses and effect inputs are never included.

Authoring loop

Edit the project.dsl/reliability document like any other scope — read, edit, validate, commit — with optimistic concurrency on the document version. Rules are validated on write (a freshness rule rejects a graph subject, a run rule requires consecutive_failures >= 1, routes must reference declared destinations), so an ill-formed policy is refused at authoring time rather than silently ignored at evaluation.
  • Triggers — the cron schedules a freshness rule follows, and the automation whose runs run-condition rules watch.
  • Database writebacks — deliveries whose receipts and failures feed the run and assertion conditions.