Kubernetes Monitoring: What to Watch and How to Turn It Into an Incident

Your load balancer can return 200 OK while half the Deployment is crash-looping. That is the Kubernetes monitoring problem in one sentence. HTTP checks at the edge answer a useful question. They do not answer whether the cluster underneath is healthy enough to stay that way.

If you already run services on Kubernetes, you know the control plane, kubelet, and scheduler well enough. The harder part is deciding which cluster signals deserve a page, which deserve a ticket, and how those alerts become an incident with enough context to act.

The cluster signals that matter

Pod restart rate. A single restart can be a deploy blip. A rising restart rate usually means crash loops, failed liveness probes, or repeated OOMKills. Alert on rate and window, not on every container exit.

OOMKill events. When a container exceeds its memory limit, the kernel kills it. From far away it can look like a tidy restart. Without OOMKill metrics or events, you will chase "flaky pods" for weeks.

Node pressure. CPU, memory, and disk pressure on a node punish every pod scheduled there. Application dashboards show random latency. The cause is often a noisy neighbor or a node that should have been cordoned.

Pending pods. Pods stuck in Pending point at scheduling failure, resource exhaustion, taints, or missing capacity. If Pending lasts more than a few minutes during normal operations, someone should look.

API server latency. When the Kubernetes API server slows down, deploys stall, controllers lag, and kubectl feels broken. This is a cluster-level incident even if user traffic still flows for a while.

Deployment rollout status. A rollout stuck at 50 percent may not fail your synthetic check if old pods still serve traffic. ProgressDeadlineExceeded and unavailable replica counts are the signals HTTP will not give you.

PVC binding and storage. Pods waiting on unbound persistent volumes fail quietly from the user's point of view. The app never becomes Ready. Storage and attach errors belong in monitoring, not only in a ticket opened after standup.

Network policy and DNS failures. Inter-service calls start failing while the ingress still looks fine. CoreDNS latency or NXDOMAIN spikes, and NetworkPolicy mischanges, often present as application 500s. Correlate them before you blame the service code.

Why HTTP-only monitoring is insufficient

Edge checks miss whole classes of failure that Kubernetes makes easy.

A subset of pods can crash and restart behind a Service that still has enough Ready endpoints to answer your probe. A canary or rolling update can freeze halfway without a clean 5xx signature. Resource contention on a shared node can create intermittent p99 spikes that a 60-second HTTP poll never catches. DNS inside the cluster can break service-to-service calls while external synthetic monitoring still hits a working ingress path.

HTTP monitoring remains necessary. Treat it as one layer, not the source of truth for cluster health.

Connecting signals to an incident workflow

Not every warning should open an incident. Over-paging teaches people to ignore the board.

Open an incident automatically when confidence and impact are high. Examples that usually qualify include OOMKill rate above a threshold on a production Deployment, pods Pending for more than five minutes in production namespaces, and API server latency sustained above 500ms. Those are not vibes. They are failure modes with clear blast radius.

Alert without creating an incident for early warnings. Elevated restart rate that has not yet crossed your incident threshold, node disk usage above 80 percent, and brief scheduling delays during a known scale-up can notify a channel or create a ticket. Promote them to incidents when they persist or worsen.

When an incident does open, include the namespace, Deployment or StatefulSet, node if relevant, and a link to the runbook. The first responder should not spend the first ten minutes reconstructing context from five dashboards.

Prometheus and Alertmanager as the bridge

Most clusters already emit the raw signals through kube-state-metrics, cAdvisor or equivalent, and control-plane metrics. Prometheus collects them. Alertmanager decides who hears about them.

A practical pattern is an Alertmanager receiver that posts to your incident management webhook. Keep labels disciplined. Use severity, namespace, deployment, and a short runbook_url. The incident tool should map those fields into the incident title, service, and priority so routing and escalation work the same way they do for uptime monitors.

A minimal receiver shape looks like this.

receivers:
  - name: incident-webhook
    webhook_configs:
      - url: "https://example.com/ingest/alertmanager"
        send_resolved: true

Pair it with routes that send severity: critical to the webhook and lower severities to chat. On the incident side, the workflow should be predictable. Alert fires, incident opens with Deployment and namespace in the title, the page goes to the service owner, and the runbook lists the first kubectl checks. For example, kubectl describe pod, recent events, and memory limits for OOMKill cases.

Resolved alerts should close or update the incident when your tooling supports it, so the board does not fill with corpses from transient spikes.

Common Kubernetes monitoring mistakes

Teams repeat the same errors.

They monitor only at the HTTP layer and learn about pod failures from customers. They skip OOMKill alerting and treat memory problems as random restarts. They page on every restart instead of restart rate, then silence the alert and miss the real crash loop. They blur cluster-level incidents with application incidents, so the API server being sick gets the same runbook as a single bad Deployment.

Another frequent miss is ignoring namespace boundaries. A noisy alert from a staging namespace trains people to ignore production pages that share the same alert name. Keep label sets and routes strict. Production critical alerts should never share a notification path with experimental workloads.

Fixing those habits does more for reliability than adding another pretty cluster dashboard. Your goal is a short list of cluster signals that predict user impact, wired into the same incident path your team already trusts.

Where this is heading for Vigiles

The destination is simple. Cluster signals should enter the same incident lifecycle you already trust for synthetic monitoring. Detect, alert, route, respond, communicate, and learn, with Kubernetes context attached from the first minute.

Kubernetes monitoring is on the Vigiles roadmap for Q4 2026, built to integrate with that same lifecycle rather than living as a separate console. Until then, the Prometheus-to-webhook path above is the honest way to stop treating the load balancer as your only cluster health check.

Watch the signals that predict failure inside the cluster. Page on the ones that matter. Leave the rest in a warning lane so on-call still answers when it counts.