Major incident management

Major incident, ordinary ticket, security incident

One event can be both a major incident and a security incident, for example ransomware that takes down a critical service. When that happens, the security lead decides on containment and evidence; the incident commander coordinates service restoration within those limits.

Use objective declaration criteria

The worst time to debate whether something is "major" is during it. Agree criteria in advance, write them down, and make declaring cheap: it is easier to stand down an over-declared incident than to recover time lost to an under-declared one. Google's SRE book suggests treating an event as an incident if a second team is needed, if the outage is visible to customers, or if it is still unsolved after about an hour of focused analysis[Google (sre.google)]. Example criteria to adapt:

• A service on your critical list is unavailable or unusable for a significant share of users.
• No safe workaround exists, or the workaround cannot be communicated quickly.
• More than one team is needed to restore service.
• A contractual, regulatory or safety obligation is at risk.
• Anyone on call believes it is major. Allow anyone to declare.

The severity classifier shows one transparent way to turn these questions into a priority.

Roles: one owner per responsibility

Roles exist so that nobody has to do two jobs at once under pressure. On a small team, one person may hold two roles, but say so explicitly. Google's SRE book describes a similar division into incident command, operations, communication and planning[Google (sre.google)].

Incident commander

Owns coordination and decisions. Sets priorities, assigns roles, keeps people focused, decides when to escalate and when the incident is over. The commander holds any role not yet delegated, which is a strong reason to delegate early. The commander should resist the pull to debug.

Technical lead

Owns diagnosis and changes to production. Proposes options to the commander, coordinates the engineers doing the work, and makes sure every change is recorded.

Communications lead

Owns updates to stakeholders and customers. Works from facts confirmed by the commander, publishes on the agreed cadence, and shields responders from status requests.

Scribe

Owns the shared timeline: times, observations, decisions and changes. This record makes the later review accurate and is often the most undervalued role.

The first 15 minutes (illustrative)

1. Acknowledge the alert or report and confirm the impact is real.
2. Declare the incident using your criteria. Announce it in the agreed channel.
3. Name the incident commander. Until someone else takes over, it is the person who declared.
4. Open a dedicated coordination channel or bridge and post the link.
5. Start the shared timeline with what is known so far, in one timezone.
6. Name a technical lead and a communications lead.
7. Ask the security question: is there any sign of malicious activity or data exposure? If yes or unsure, page security now.
8. Check recent changes: deploys, config, infrastructure, third-party status.
9. Publish a first stakeholder update with the time of the next one.
10. Agree the first mitigation to try and who will do it.

Status update cadence: a framework, not a number

There is no universal correct update interval. What matters is that people know when the next update is coming and that you keep that promise. Choose a cadence by considering:

• Severity and audience: executives and customers in a full outage need more frequent updates than an internal team during a partial degradation.
• Rate of change: update more often while the situation is changing, less often during a long, stable recovery.
• Commitments: contracts or regulations may set required notification times. Those override your defaults.
• Responder load: each update costs the communications lead time. Too frequent becomes noise.

Every update should answer four questions: what is affected, what we are doing, what users can do now, and when the next update will be.

Escalation and service restoration

Escalate early and without apology. Paging another team that turns out not to be needed is cheap; paging them an hour late is expensive. Prioritize restoring service over finding the cause: rollback, failover, scaling, disabling a feature or routing around a dependency are all legitimate. Record every change in the timeline so it can be reviewed or reverted. Before declaring recovery, verify from the user's point of view, not only from internal metrics.

Customer updates

• Say what users experience, in their language, not your internal service names.
• Do not speculate about causes in public. "We are investigating" is honest; a wrong guess is hard to retract.
• Give workarounds if they exist and are safe.
• For any suspected data exposure, coordinate wording with security, legal and privacy before publishing.
• Send a resolution notice, and later a summary if your policy calls for one.

Handoff and post-incident review

Long incidents need handoffs. A good handoff is a short written summary: current state, what has been tried, what is in progress, open questions and who holds each role now. When the incident closes, record final times, assign an owner for the review and schedule it. The post-incident review guide covers what comes next.

Example: a hypothetical major incident

The organization and details below are fictional.

At 14:02 a fictional payroll provider's web app starts timing out for all customers. The on-call engineer acknowledges at 14:05, sees the database connection pool exhausted, and declares a major incident at 14:08 because the service is on the critical list and customers are affected. She is commander until the platform manager takes over at 14:15. A technical lead and a communications lead are named. The security question is asked: no signs of malicious activity. The first customer update goes out at 14:18 with a next-update time. At 14:31 the team rolls back a query change deployed at 13:55; latency recovers at 14:36 and is confirmed by test logins. The incident is resolved at 15:20 after the query is fixed properly, and a review is scheduled for two days later.