FailedFailed 2 times in a row
Backup job "DC-02 · vzdump all VMs" reported failed: [Failed] vzdump all VMs — repository unreachable
Told once per backup, in the place your team already looks.
An alert belongs to a backup, not to an email. It opens when the backup fails, goes missing or warns, changes in place when things get worse, and closes itself when the next good report arrives.
Where alerts can go.
Owners and admins set up channels in Settings, and can send a test to an Email, Slack or Teams channel before relying on it.
- Sent to the address you enter. Email alerts
- Slack
- An incoming webhook on hooks.slack.com. Slack alerts
- Microsoft Teams
- A Workflows webhook. Teams alerts
- PagerDuty
- Events API v2, for critical alerts. One incident per alert, resolved in PagerDuty when the backup recovers or someone resolves it. PagerDuty alerts
A failing backup is one alert, not one message a night.
Each backup has at most one open alert. A backup that fails again keeps the same alert, so the channel is not filled with copies of the same problem.
A backup that flaps between results is capped at 100 alert emails in 24 hours. Needs attention then says “Email alerts paused for 1 flapping backup today”.
- A warning opens the alert and sends it to your channels.
- More warnings from the same backup change nothing: the alert is already open.
- The backup then fails or goes missing: the same alert is raised to critical, its acknowledgement or snooze is cleared, and it is sent again so someone looks.
- The next OK report resolves it, and “Recovered: <job> is healthy again” goes to the channels that delivered it.
- Resolved by hand instead, those channels get “Resolved by <name>”.
When a message does not get through, or nobody picks it up.
Retries
A delivery that fails is tried again every 10 minutes, up to 5 attempts in all, for as long as the alert is open. Each attempt gives up after 10 seconds.
Every alert shows how it went, for example “Sent to 2 channels · 1 failed”, and a channel that failed in the last 24 hours shows in Needs attention as “Alert channel failing”.
Escalation
Turn on Escalate unacknowledged critical alerts and pick a delay: 30 minutes, 1, 2, 4 or 8 hours. A failed or missing backup that nobody has acknowledged, snoozed or resolved by then is sent once more, as “Not acknowledged”, to the channel you chose.
The check runs every 15 minutes. Alerts older than 3 days are not escalated.
Who gets which alerts, and when.
- Which alerts
- A channel takes every client’s alerts, or only some clients’.
- Escalations only
- The channel never gets the first alert, only the escalation: a manager’s inbox or an on-call pager.
- Quiet hours
- In the workspace timezone. Warnings are not sent during them; they stay in Needs attention and are not sent later. Failed and missing backups, and escalations, always go out.
- On-call contacts
- Turn it on for a client and its on-call contacts, up to 5, are emailed when a backup fails or goes missing.
- Paused clients
- No alert is created while a client is paused. Its reports are still stored.
The rest arrives as a summary.
Not everything needs a message the moment it happens. Two emails to the workspace owner give the overview: one every morning, one every week.
- Daily digest
- At 07:00 workspace time, on by default, skipped while there are no backups. The email has an unsubscribe link.
- Weekly summary
- On Mondays from 07:00: last week’s success rate, incidents, the backups that failed most and the clients below their SLA target. Off until you turn it on.
What it does not do.
- There are no SMS alerts.
- PagerDuty receives critical alerts only, and Send test does not work for PagerDuty.
- A warning held back by quiet hours is never sent later. It stays in Needs attention and the digest.
- Size drops are listed in Needs attention only. They are never sent to a channel.
- An escalation is sent once. There is no second escalation step.
Get told once, in the right place.
Add a channel, point one backup's report at its client address, and the next failure arrives where your team works.