Shadow On-Call: How to Onboard New Engineers Without Burning Them Out

You put a new hire on primary on-call in week three. They got a page overnight for a payment timeout that looked like a database problem and was actually a third-party webhook retry storm. They restarted the wrong service. The incident went from fifteen minutes to ninety. By morning they were apologizing in Slack and quietly deciding that on-call was something to avoid forever.

That pattern is common. The damage is not only the longer outage. It is the aversion that follows a first rotation that felt like a test they failed in public.

Shadow on-call exists to prevent that outcome.

What shadow on-call is

Shadow on-call means the new engineer joins the same rotation as the primary without owning response. They get the same pages on their phone. They watch how the primary acknowledges, reads the alert, checks recent deploys, opens the runbook, and decides whether to escalate. They sit in the incident channel as an observer. They attend the postmortem as a learner, not as the author of the narrative.

They are not expected to fix anything. They are expected to learn how your team actually operates under pressure, which is rarely what the wiki implies.

Why dropping someone straight onto primary fails

On-call is a skill with muscle memory. Reading an alert while half awake, knowing which dashboards matter, remembering that this service fails open and that one fails closed. Those habits form under real pages, not in a classroom.

If the first real page is also the first time you are responsible for production, panic fills the gaps. People over-restart. They skip the runbook because it feels slow. They hesitate to escalate because they think escalation means they failed. Poor decisions under pressure become the story of their first week on the rotation, and that story sticks.

A shadow rotation lets the same learning happen with a safety net. The primary still owns the outcome. The shadow still feels the pager.

How to structure the shadow rotation

One to two weeks of shadowing is enough for most 10 to 50 person teams, assuming the person already has a working knowledge of the stack. Do not treat the calendar as the graduation criterion. Treat it as a window in which readiness can be demonstrated.

Before the shadow week starts, do three practical things. Confirm their alert channels work on their phone, not only in a laptop browser. Walk them through where runbooks live and how incident severity is decided. Introduce them to the two or three people they are most likely to escalate to.

During quiet periods, the shadow should read recent incident timelines, skim the runbooks for the services that page most often, and ask the primary to narrate one past incident end to end. Quiet time is when the learning compounds. Active incidents are when it gets stress-tested.

What the shadow should do during an active incident

Receive the page. Open the same views the primary opens. Take notes with timestamps. Watch the decision path, not only the commands.

Ask questions after the incident is stable, not while the primary is mid-diagnosis. Mid-incident questions that can wait should wait. The exception is a safety catch, something like "you are about to restart the wrong pod." That kind of interrupt is useful. Curiosity about why a metric looks odd can wait fifteen minutes.

After resolution, write a short personal summary of what happened and what you would have done differently. Share it with the primary in the debrief. That exercise turns observation into retained judgment.

What the primary should do differently with a shadow

Narrate briefly. "I am checking deploys first because this alert almost always tracks a release." One sentence of context is enough. You do not need a lecture.

Avoid unexplained jargon. If you say "fail the shard over," say what that means once. The shadow is building a mental model. Undefined shortcuts leave holes.

Debrief after every page that required real work, even if the incident was short. Five minutes covering what you looked at, what you skipped, and what you would do next time is more valuable than a long retrospective days later when the details have gone soft.

If nothing pages during the shadow window, walk through a recent incident as a tabletop. Silence is not a reason to skip the skill transfer.

Readiness criteria for moving to primary

Graduate on demonstrated readiness, not on "we have done two weeks." Useful criteria look like this.

The shadow can explain the severity levels your team uses and give an example of each. They can find the runbook for the three services that page most often without help. They have successfully received a test page on every channel they will use on primary. In at least one real or tabletop incident, they correctly identified the first place to look before the primary said it aloud. They know who to escalate to for database, payments, and infrastructure issues, and they are willing to escalate without treating it as personal failure.

If those are not true yet, extend the shadow. An extra week is cheaper than a burned-out engineer and a messy first primary shift.

The reverse shadow

Seniority elsewhere does not make someone ready for your pager. An experienced engineer joining from another team, or returning from parental leave after six months away, should shadow before taking primary. Stacks drift. Runbooks go stale. Escalation paths change. Pride is a weak reason to skip the ramp.

Frame it as a team norm, not a judgment on their ability. "Everyone shadows before primary here" removes the awkwardness. The reverse shadow can be shorter, often a few days, but it should still include at least one real or tabletop incident with narration and debrief.

What happens when you skip it

Without a shadow, first rotations produce predictable failures. Someone restarts a healthy dependency because the alert named it and they did not know the dependency graph. Someone spends forty minutes debugging application code while a config flag from the morning deploy was the whole story. Someone freezes on a Sev-1 because they have never seen the severity process used for real and do not want to "overreact."

Those incidents are fixable. The harder cost is the engineer who decides on-call is hostile and starts negotiating their way out of the rotation. Once that pattern sets in across a few hires, your on-call culture erodes from the edges.

Shadow on-call is a small investment. One to two weeks of shared pages, clear readiness criteria, and a habit of narrating decisions. It is how you grow a rotation that people can join without bracing for damage.