Monitoring and Alerting: Golden Signals and SLOs

The best starting point for monitoring any service is the four golden signals from Google's Site Reliability Engineering book: latency, traffic, errors and saturation. Measure those for each user-facing service, turn the important ones into service level indicators (SLIs), agree service level objectives (SLOs) that say how good is good enough, and alert only on conditions that need a human to act. Everything else belongs on a dashboard, not in someone's phone at 3 a.m.

Many businesses end up with the opposite: hundreds of charts, dozens of noisy alerts and no clear answer to the simple question "are our customers having a good experience right now?" This guide gives you a lean framework to fix that, with a worked example and an alert rule you can adapt.

Monitoring, observability and alerting: the difference

• Monitoring is collecting and watching known signals, such as error rate and CPU.
• Observability is being able to ask new questions of a running system using its metrics, logs and traces, especially when something unexpected happens.
• Alerting is deciding which conditions should interrupt a person.

The three pillars of telemetry are metrics (numbers over time), logs (events with detail) and traces (the path of a request across services). Metrics tell you that something is wrong, traces and logs help you work out why.

The four golden signals

Signal	Question it answers	Examples
Latency	How long do requests take?	Response time percentiles, separating successful from failed requests
Traffic	How much demand is there?	Requests per second, active sessions, messages consumed
Errors	How often do requests fail?	HTTP 5xx rate, failed payments, timeouts, wrong results
Saturation	How full is the system?	Memory and disk use, queue depth, connection-pool usage

Two practical notes. First, use percentiles for latency, such as the 95th or 99th, not averages. An average hides the slow experience of the unlucky few. Second, saturation should be measured against the real limit of the constraint, for example a database's connection limit, and not only CPU.

For infrastructure components, a related method called USE (utilisation, saturation, errors) works well for resources like disks and network interfaces. For request-driven services, RED (rate, errors, duration) is a simplified form of the golden signals. Pick one framework and apply it consistently.

SLIs, SLOs and SLAs explained

These three terms are often confused:

• SLI (service level indicator): a measurement of service behaviour, usually a ratio of good events to all events. Example: the proportion of requests that complete successfully in under a set time.
• SLO (service level objective): your internal target for that indicator over a window. Example: 99.9 per cent of requests succeed over a rolling 30 days.
• SLA (service level agreement): a contractual commitment to a customer, usually with consequences if missed. It should be looser than your internal SLO.

Choose SLIs that reflect user experience. For a web application, availability (successful responses) and latency (fast enough responses) are the usual pair. For a data pipeline, freshness and correctness matter more. Do not pick a target because it sounds impressive; pick one your users genuinely need, because each extra "nine" demands more engineering effort and cost.

Error budgets in plain arithmetic

An SLO implies an error budget: the amount of failure you can tolerate. If your target is 99.9 per cent availability over 30 days, the budget is 0.1 per cent of that period. Thirty days is 43,200 minutes, so the budget is about 43 minutes of full unavailability, or the equivalent in failed requests. If you have used most of the budget, you slow down risky releases and prioritise reliability work. If you have plenty left, you can ship faster. This turns the argument between "ship features" and "keep it stable" into a shared, measurable rule.

What to alert on

Alert on symptoms that users feel, not on every cause. High CPU is a cause; a rising error rate or slow checkout is a symptom. A good alert meets all of these tests:

1. It is urgent: it needs attention now, not tomorrow.
2. It is actionable: someone can do something about it.
3. It is real: it reflects user impact or imminent impact.
4. It is understandable: it links to a runbook that says what to check first.

Everything that fails those tests becomes a ticket, a dashboard panel or a daily report. Here is a Prometheus alerting rule for a simple error-rate symptom, using the Prometheus and Grafana stack that CloudWiss lists for monitoring and alerting:

groups:
  - name: web-availability
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{job="web",status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="web"}[5m]))
          > 0.02
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "More than 2% of web requests are failing"
          runbook_url: "https://wiki.example.com/runbooks/web-high-error-rate"

The for: 10m clause requires the condition to persist, which avoids paging on a short blip. The threshold of 2 per cent is only an illustration; derive yours from your SLO and traffic. More advanced setups use multi-window burn-rate alerts, which page when the error budget is being consumed too fast. The Google SRE Workbook describes the approach in detail.

Building a minimal monitoring set

1. List your critical user journeys. Sign-in, search, checkout, file upload, whatever the business depends on.
2. Define one or two SLIs per journey. Success rate and latency are enough to start.
3. Add synthetic checks. A scheduled probe that exercises the journey from outside catches problems your internal metrics miss, such as an expired certificate or a DNS fault.
4. Instrument the golden signals in each service and the load balancer.
5. Collect logs centrally with a request identifier, so a single request can be followed across services.
6. Add tracing when you have more than a couple of services talking to each other.
7. Write the paging policy. Who is on call, how they are reached and how issues escalate.
8. Write a short runbook for each paging alert.

Dashboards should mirror this: a top-level view per service showing the four signals, with deeper views for infrastructure and dependencies. Tools like Prometheus, Grafana, Datadog and New Relic can all serve this, and the right choice depends on your team and budget.

Monitoring and releases

Monitoring is what makes safe deployment possible. A canary release is only useful if you can compare the new version's error rate and latency with the old one. A CI/CD pipeline that deploys automatically should be able to check a health signal and halt or roll back. If you run Kubernetes, add readiness probes and resource metrics from day one. Keep an eye on telemetry cost too: logs and metrics can balloon, as noted in our guide to cloud cost levers.

Common mistakes

• Alerting on causes. CPU at 80 per cent might be perfectly fine. Alert on user-visible symptoms and use cause metrics for diagnosis.
• Averages instead of percentiles. They hide tail latency.
• No owner for alerts. An alert nobody owns is ignored. Route each to a team.
• Alert fatigue. If people mute channels, you have too many alerts. Review noisy ones monthly and delete or downgrade them.
• Monitoring only from inside. If your monitoring runs in the same environment that fails, it fails with it. Add an external check.
• Missing runbooks. A page with no guidance wastes the first ten minutes of every incident.
• Setting an SLO nobody agreed. An SLO needs buy-in from product and business owners, not just engineers.
• Never testing alerts. Trigger them deliberately to confirm they reach the right person.
• Ignoring business metrics. Orders per minute dropping to zero is often the fastest signal of all.

Summary

Measure latency, traffic, errors and saturation; define a few SLIs and SLOs tied to what customers experience; page people only for urgent, actionable symptoms; and keep everything else on dashboards. If you want monitoring that is designed rather than accumulated, CloudWiss's cloud and DevOps services include monitoring and alerting setup with tools such as Prometheus and Grafana, with documentation so your team can run it afterwards.