Alerts

Alerts tell the right person when something needs attention, and stay quiet when it does not. An alert rule watches a condition, and a channel delivers the message. This guide shows how to set up both and how to keep on-call load low.

Alert rules

A rule has a condition, a scope, a severity and one or more channels. Rules live in synchronoid.yaml, so they are reviewed and deployed together with your pipelines. You can also edit them in the control room under Alerts.

alerts:
  source-paused:
    when: source.paused
    severity: critical
    channels: [oncall-pagerduty, data-alerts]

  run-failed:
    when: run.failed
    scope: { pipeline: "companies_*" }
    severity: high
    channels: [data-alerts]

  stale-data:
    when: asset.freshness_exceeded
    scope: { asset: companies_current }
    threshold: "26h"
    severity: medium
    channels: [email-data-team]

Channels

A channel is a named destination. Define it once, and use the name in any rule.

channels:
  data-alerts:
    type: slack
    webhook_url: ${secret:SLACK_WEBHOOK}
  oncall-pagerduty:
    type: pagerduty
    routing_key: ${secret:PAGERDUTY_KEY}
  email-data-team:
    type: email
    to: [data-team@example.com]
  ops-hook:
    type: webhook
    url: https://example.com/hooks/synchronoid-alerts
    signing_secret: ${secret:OPS_HOOK_SECRET}

• Slack. Messages show the failed asset, the error code and a link to the run. Replies in the thread are added to the alert history.
• PagerDuty. Critical alerts open an incident. When the problem clears, Synchronoid resolves the incident by itself.
• Email. Good for low-severity alerts and daily digests (digest: daily).
• Webhook. Sends a JSON body signed in the same way as change feed webhooks.

Test a channel before you rely on it:

$ syn alerts test oncall-pagerduty
Sent a test alert to oncall-pagerduty (status 202)

Grouping and quiet periods

If a provider is down, fifty assets can fail at the same time. Synchronoid groups alerts that share a source into one message and updates it as more failures arrive. You can tune this with group_by and cooldown.

  run-failed:
    when: run.failed
    group_by: [source]
    cooldown: "30m"          # do not repeat the same alert within 30 minutes
    mute_during: "maintenance"

On-call tips

• Page only for what needs a person now. A paused source, a stalled feed and stale data on a critical asset qualify. A single retry does not.
• Keep severities few. Send critical to PagerDuty, high to Slack, and the rest to email digests.
• Use the replay tools. Most failures are fixed by correcting the cause and running syn runs replay. See Retries and replays.
• Add a runbook link. Set runbook: https://... on a rule, and it appears in every message.
• Review weekly. The Alerts page shows which rules fired most. Delete or raise the threshold of any rule that people ignore.