Alerts and channels

When a backup fails, goes missing or finishes with warnings, BackupSentinel sends an alert to your alert channels. This page covers the four kinds of channel, what is sent and when, and the rules that keep alerts useful: quiet hours, escalation, routing per client and the cap for backups that flap.

Channels

Add channels in Settings → Alert channels with Add channel. Only owners and admins can add or change channels.

There are no SMS or voice alerts.

Each channel row has Send test, Pause or Resume, Configure and Delete. A paused channel gets nothing until you resume it. Setup notes per product: Email, Slack, Microsoft Teams, PagerDuty.

What is sent

There are three kinds of alert:

• Failed: a report says the backup failed. Critical.
• Report missing: no report arrived in the backup's window. Critical. See Job statuses and missing reports.
• Warning: a report says the backup finished with warnings.

An email alert's subject starts with the kind of problem and names the backup and the client, so you can triage from the inbox list.

One alert per backup. While a backup has an open alert, further failures do not send new alerts. The alert stays open in Needs attention until the backup recovers or someone resolves it.

A warning that becomes a failure is escalated in place. The same alert is raised to critical and sent again, and any acknowledgement or snooze on it is cleared, so someone looks again.

PagerDuty receives critical alerts only. Each alert is one PagerDuty incident.

Recovery notices

When a report says the backup succeeded, its open alert is resolved and a recovery notice ("Recovered: DC01 nightly is healthy again") goes to the channels that received the alert. When someone resolves an alert by hand, those channels get "Resolved by" with their name instead. PagerDuty gets a resolve for the incident in both cases.

A recovery notice only goes to channels where the alert was delivered successfully.

Delivery and retries

Each alert in Needs attention has a delivery line, such as "Sent to 2 channels · 1 failed · 0 skipped", or "Not sent" when it reached no channel.

• Each send waits at most 10 seconds for the channel to answer.
• A failed delivery is retried every 10 minutes, up to 5 attempts in total, while the alert is still open.
• Any failed delivery in the last 24 hours lists the channel in Needs attention as "Alert channel failing", so a broken webhook does not go unnoticed.

Testing a channel

Send test sends a test message to the channel straight away. It works for Email, Slack and Microsoft Teams, and is for owners and admins. A workspace can send up to 5 tests in 10 minutes.

There is no test for PagerDuty. To check a PagerDuty channel, confirm the routing key in PagerDuty itself.

Which alerts a channel gets

Under Which alerts in a channel's settings:

• Every client: the channel gets alerts for all clients. This is the default.
• Only some clients: tick the clients it covers. Useful when a client has its own Teams channel or a technician looks after a few clients.
• Escalations only: the channel never gets the first alert. It only receives escalations (below), for example a manager's mailbox or an on-call pager.

A client's page shows which channels cover it under Alerts. If none does, its problems only show in Needs attention.

On-call contacts

On a client's page, Also email this client's on-call contacts when a backup fails or goes missing sends critical alerts to the client's contacts marked On-call contact, up to five of them. They get the alert email; they do not need an account. Contacts are set on the client's Contacts tab. See Clients and devices.

Alert rules

The Alert rules card under Settings → Alert channels holds three workspace-wide rules. Times are in the workspace timezone.

Quiet hours

During Quiet hours (for example 22:00 to 07:00), warnings are not sent to channels. They are recorded as skipped and are not sent later; they stay in Needs attention and in the morning digest. Failed and missing backups are always sent, and so are escalations.

Escalate unacknowledged critical alerts

With Escalate unacknowledged critical alerts on, a failed or missing backup that nobody acknowledges, snoozes or resolves in time is sent once more, marked "Not acknowledged", to a channel you choose. The choices are after 30 minutes, 1 hour, 2 hours, 4 hours or 8 hours. The check runs every 15 minutes, and alerts older than 3 days are not escalated.

Weekly summary email

Weekly summary email sends the owner last week's success rate, incidents, the backups that failed most and the clients below their SLA target, on Monday morning from 07:00. It is off until you turn it on.

Daily digest

The workspace owner gets a daily digest at 07:00 workspace time with the health of every backup. It is on by default. Turn it off in Settings → General with Daily digest email, or with the unsubscribe link in the email. No digest is sent while the workspace has no backups.

The digest goes to the owner only. Other team members rely on alert channels and the dashboard.

Flapping backups

A backup that fails and recovers over and over could send hundreds of emails. Email alerts are capped at 100 per backup in any 24 hours. Past that, email alerts for that backup stop until the count drops, and the dashboard shows "Email alerts paused for N flapping backups". Needs attention still lists the backup.

Paused clients

A paused client's backups never open alerts, so nothing is sent for them. Reports are still stored and still update each backup's status. When the client resumes, alerts work as before.

Limits

• No SMS or voice alerts. Use PagerDuty, or a Teams or Slack channel on a phone.
• PagerDuty receives critical alerts only, and Send test does not work for PagerDuty.
• Warnings during quiet hours are never sent later.
• Backup size drops are listed in Needs attention only. They are never sent to channels.
• The daily digest and the weekly summary go to the workspace owner only.
• Failed deliveries are retried for up to 5 attempts in total, and only while the alert is open.
• An escalation is sent once, to one channel.