Back to All Articles
Technical SREOct 9, 20269 min read

On-Call Survival Guide: Best Practices for Setting Up PagerDuty, Slack & Discord Escalations

Avoid alert fatigue and missed outages by configuring an effective incident escalation policy. Learn best practices for email, Slack, Discord, and PagerDuty alerting in modern SRE teams.

P
Prabin Jha
Founder & Lead Architect
LinkedIn

Key Takeaways (BLUF)

Bottom Line Up Front (BLUF): Without a structured escalation policy, on-call engineers suffer from alert fatigue by being woken up for minor warnings, or conversely, miss critical database crashes buried in noisy Slack channels. The best practice is a tiered routing approach: route low-priority warnings to default Email or passive Discord/Slack channels for next-day review, while reserving high-urgency SMS and PagerDuty escalations strictly for confirmed, multi-region-validated outages that impact customers directly.

The Three Tiers of Incident Escalation

A robust on-call escalation strategy divides monitoring alerts into three distinct urgency tiers. Sending every alert to every channel guarantees that your team will eventually ignore the alerts entirely—a phenomenon known as alert fatigue.

To maintain a healthy engineering culture and protect your team's sleep, classify your endpoints and monitors into the following tiers:

Urgency TierSLA ImpactAppropriate Alert ChannelsExample Incident
Tier 3 (Low / Informational)No immediate customer impactEmail (Default)SSL Certificate expires in 14 days; nightly backup cronjob delayed by 5 minutes.
Tier 2 (Warning / Degraded)Degraded performance, partial impactSlack / Discord (Dedicated Channel)API latency spikes over 2000ms; secondary non-critical worker node fails.
Tier 1 (Critical / Outage)Direct revenue or SLA lossPagerDuty / SMS / VoicePrimary database unreachable; payment gateway returning HTTP 500s.

Email Alerts: The Ultimate Catch-All (Default)

WakeMetrics includes instant email alerts by default on all plans. While email is rarely the best channel for waking someone up at 3:00 AM, it serves as the ultimate asynchronous catch-all and audit trail.

Best Practice: Route all Tier 3 informational alerts (such as SSL expiration warnings or non-critical staging environment downtime) exclusively to email. This ensures the alerts are recorded and visible the next morning, without interrupting deep work or off-hours rest.

Slack and Discord: Creating a Dedicated Incident Channel

ChatOps is the center of modern incident response. Routing synthetic monitoring alerts directly to Slack or Discord provides instant visibility to the entire team, allowing engineers to claim an incident (e.g., "I am looking into this 👁️") and collaborate in a threaded discussion.

However, routing alerts into `#general` or `#engineering` is a fatal mistake. The noise will drown out team communication.

  • Create Dedicated Channels: Create specific channels like `#alerts-production` and `#alerts-staging`.
  • Webhook Integration: Use standard HTTP Webhooks to connect WakeMetrics to Slack or Discord. This delivers rich, structured JSON payloads directly into the channel, highlighting the exact endpoint and HTTP status code.
  • Mute Rules: Encourage the broader team to keep `#alerts-staging` muted, while the on-call engineer enables push notifications for `#alerts-production`.

PagerDuty and Webhooks: The Nuclear Option

When a Tier 1 incident strikes—meaning customers cannot log in, check out, or use your core product—an asynchronous Slack message is insufficient. You need an immediate, overriding escalation to PagerDuty, Opsgenie, or a custom SMS webhook.

Best Practice: Never hook a single-node monitor to PagerDuty. Transient network blips will cause severe on-call burnout. Only trigger PagerDuty if your synthetic monitor uses 3-Stage Multi-Region Consensus to absolutely verify that the outage is a hard global failure.

{
  "incident_key": "prod-db-cluster-down",
  "routing_key": "YOUR_PAGERDUTY_INTEGRATION_KEY",
  "event_action": "trigger",
  "payload": {
    "summary": "CRITICAL: Payment Gateway API is returning 502 Bad Gateway",
    "source": "WakeMetrics Global Edge (Consensus Reached)",
    "severity": "critical",
    "custom_details": {
      "endpoint": "https://api.yourcompany.com/checkout",
      "failed_regions": ["Frankfurt", "New York", "Tokyo"],
      "latency_ms": "Timeout (10000ms)"
    }
  }
}

Automating Customer Communication

Escalation isn't just for internal engineers. Your customer support and success teams need to know when the system is down to intercept incoming tickets.

By routing alerts to a public or private Custom Domain Status Page, you can automatically flip your system status to "Degraded" or "Down," instantly informing your users and reducing the load on your support desk.

Frequently Asked Questions

Should I route all uptime alerts to PagerDuty?

No. Routing staging environments, low-priority cronjobs, or single-node transient failures to PagerDuty leads to severe alert fatigue. Only route mission-critical production endpoints that have been cross-validated by multi-region consensus to PagerDuty.

How do I connect WakeMetrics to Slack or Discord?

WakeMetrics supports native webhook integrations. Simply generate an Incoming Webhook URL in your Slack or Discord server settings, and paste it into the WakeMetrics monitor configuration. Alerts will be delivered instantly as structured rich-text cards.

Are email alerts included on the free plan?

Yes, instant email incident alerts are included by default on all WakeMetrics plans, ensuring you always have a baseline audit trail of your infrastructure's uptime.

Conclusion

Effective incident management relies on silencing the noise and amplifying the signal. By tiering your alerts across Email, Slack/Discord, and PagerDuty, your team can maintain high SLA standards without sacrificing their sanity. To configure smart, multi-region validated alerts, start testing with WakeMetrics for free.

Ready to Upgrade from UptimeRobot?

Migrate in under 60 seconds. Get 15-second checks across 450+ global locations with zero false alerts.