IT incident management reference
The incident management process: eight practical steps
Incident management is how a team restores normal service as quickly and safely as possible when something unplanned breaks or degrades, then learns from it. This page is for IT operations, site reliability, DevOps and service desk teams that want a clear, repeatable process without buying a framework first.
By the Incident Management Process Editorial Team · Published
About this framework
These eight steps are our own practical editorial framework. They are not quoted from NIST, ITIL or any other standard, and no standards body has reviewed them. They line up with common practice, such as the incident command roles described in Google's SRE book[Google (sre.google)]. For cybersecurity incidents specifically, NIST SP 800-61 Rev. 3 is the primary public reference[NIST].
The process, step by step
- 1. Prepare. Ownership, on-call, channels and a severity policy, agreed before anything breaks.
- 2. Detect and log. Capture the alert or report with timestamps and the affected systems.
- 3. Assess and prioritize. Judge impact and urgency, and route possible security events to specialists.
- 4. Declare and coordinate. Name an incident commander, assign roles and start one shared timeline.
- 5. Contain and mitigate. Stabilize the service and stop the harm from spreading.
- 6. Recover and verify. Restore service and confirm it from the user's point of view.
- 7. Communicate and close. Send accurate status, close cleanly and hand off open work.
- 8. Review and improve. Run a blameless review and assign follow-ups that get finished.
1. Prepare
Most of an incident's outcome is decided before it starts. Write down who owns each production service, who is on call and how to reach them, where incident coordination happens (a dedicated chat channel, a bridge line, or both), and how severity is decided. A one-page severity policy beats an argument at 2 a.m. Keep a short template for the live incident record so nobody has to invent a format under pressure.
- A service ownership list with an escalation contact for each service.
- An on-call schedule people can find in under a minute.
- A severity policy with examples. Our severity classifier shows one transparent approach.
- Agreed channels for internal coordination and for customer updates.
2. Detect and log
An incident starts as an alert, a customer report or someone noticing something odd. Record it immediately with a timestamp that includes the timezone, what was observed, and which systems or users seem affected. Write observations, not theories: "checkout returns HTTP 500 for card payments" is useful, "the database is broken" may send people in the wrong direction. Good timestamps here are what make metrics like MTTA and MTTR meaningful later.
3. Assess and prioritize
Decide how bad it is and how fast it needs attention. Impact (what users experience), breadth (how many are affected), whether a workaround exists and how time-sensitive the situation is together drive priority. If there is any reasonable suspicion of a security breach, privacy exposure or physical safety risk, bring in your security team now. Operational priority and security response are related but separate decisions.
4. Declare and coordinate
For anything beyond a routine ticket, declare an incident and name an incident commander. The commander coordinates and decides; they usually do not debug. One widely used rule of thumb is to declare when a second team is needed, when customers can see the problem, or when the issue is still unsolved after about an hour of focused work[Google (sre.google)]. Start one shared timeline that everyone writes to.
5. Contain and mitigate
Stabilize first, investigate second. Rolling back a change, failing over, shedding load or disabling a feature flag often restores service long before anyone understands the cause. Write down every change made during the incident so it can be reviewed or reverted later. For suspected security incidents, preserve evidence before making changes that would destroy it, unless the change is needed to stop active harm.
6. Recover and verify
Recovery means users can do what they came to do, not that a dashboard turned green. Confirm with a synthetic check, a test transaction or a customer report. Record two separate times: when users regained service, and when the incident was fully resolved (cleanup done, temporary fixes replaced or tracked). Those two times answer different questions, which is why the different kinds of MTTR should never be blended.
7. Communicate and close
Keep stakeholders informed on a predictable cadence and say only what you know. A good update states current impact, what is being done, and when the next update will come. When service is restored, send a closing note, hand any remaining work to an owner, and record the final times.
8. Review and improve
Hold a blameless post-incident review while memories are fresh. Focus on how systems and processes allowed the failure, not on who to blame, and finish with a small number of specific follow-up actions, each with an owner and a due date. Our post-incident review guide covers the full method, and the postmortem generator turns your notes into a clean document.
Example: a hypothetical incident, start to finish
This example is invented to show how the steps fit together. The company, times and numbers are fictional.
| Time (UTC) | What happened | Step |
|---|---|---|
| 09:58 | Login error rate starts climbing after a certificate rotation. | Starts |
| 10:03 | Alert fires. On-call engineer acknowledges at 10:06 and logs the observation. | 2 |
| 10:09 | Most sign-in attempts failing, no workaround: assessed as P1. | 3 |
| 10:11 | Incident declared. Commander, technical lead and communications lead named. | 4 |
| 10:15 | First status update published with the next update time. | 7 |
| 10:24 | Previous certificate restored on the load balancer. | 5 |
| 10:31 | Sign-in success rate back to normal; test logins confirmed. | 6 |
| 11:40 | New certificate chain fixed and deployed; incident resolved and closed. | 7 |
| Two days later | Blameless review: three follow-ups assigned with owners and dates. | 8 |
In this example, time to acknowledge is 3 minutes (10:03 to 10:06), time from impact to recovery is 33 minutes (09:58 to 10:31), and time from detection to resolution is 97 minutes (10:03 to 11:40). Three different durations, each answering a different question.
Who does what
Small teams often combine roles. What matters is that each responsibility has exactly one owner during the incident.
| Role | Owns | Does not |
|---|---|---|
| Incident commander | Overall coordination, priorities, decisions, role assignment | Get lost in hands-on debugging |
| Technical lead | Diagnosis and changes to the affected systems | Write customer updates while fixing |
| Communications lead | Stakeholder and customer updates on a set cadence | Speculate about causes publicly |
| Scribe | The shared timeline: actions, decisions and times | Filter out awkward facts |
Google's SRE book describes a similar split between incident command, operations, communication and planning[Google (sre.google)]. See major incident management for how the roles work under pressure.
P1 to P4 in plain language
There is no universal priority scale. Every organization defines its own, so treat this as an example to adapt, not a standard.
| Priority | Typical situation | Typical response |
|---|---|---|
| P1 | Critical service down for most users, no workaround | Declare a major incident, all hands, frequent updates |
| P2 | Major degradation, or a critical outage with some workaround | Dedicated owner, incident commander if teams are many |
| P3 | Limited impact with a workable mitigation | Normal on-call queue with tracked follow-up |
| P4 | Low urgency, no meaningful current impact | Routine queue |
The severity classifier applies a transparent version of this table to five questions and shows its reasoning.
Free tools
MTTR calculator
Upload an incident CSV and get MTTA, resolve and recovery MTTR, and MTBF when your data supports it. Nothing is uploaded.
Severity classifier
Five questions to an illustrative P1 to P4, with every rule shown and a security escalation flag.
Postmortem generator
Turn incident notes into a structured blameless postmortem. It never invents causes or impact.
Free: Post-Incident Review Starter Kit
An editable one-page review template (DOCX and Markdown), a meeting checklist and a filled fictional example. No signup required.
Need more than a starter template? The template packs list every file, format and license term before you decide anything.
Frequently asked questions
Is incident management the same as problem management?
No. Incident management restores service as quickly as possible. Problem management investigates underlying causes so incidents stop recurring. A post-incident review often feeds problem management with its follow-up actions.
Are these eight steps an official standard?
No. They are this site's editorial framework, written to be practical. They are consistent with common practice but are not quoted from NIST, ITIL or any other body, and no such body has endorsed them.
How is an IT incident different from a security incident?
An IT operations incident is any unplanned disruption or degradation of service. A security incident involves a possible compromise of confidentiality, integrity or availability through malicious or unauthorized activity. They can overlap, but security incidents need specialist handling, such as evidence preservation and legal or regulatory steps. NIST SP 800-61 Rev. 3 covers that side [NIST].
When should we declare a major incident?
Agree the criteria in advance. Common triggers include customer-visible impact on a critical service, the need for more than one team, or no progress after a set period of focused work. See the major incident guide.
Which MTTR should we report?
Whichever one you define precisely and measure consistently. MTTR can mean time to repair, recover, respond or resolve, and these give different numbers [Atlassian]. Our MTTR guide explains how to choose.
Sources
- Google (sre.google): Site Reliability Engineering, Chapter 14: Managing Incidents (accessed 2026-10-18)
- NIST: SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile (accessed 2026-10-18)
- Atlassian: Common incident management metrics (accessed 2026-10-18)
Links point to the original publishers. Citing a source does not mean that publisher reviewed or endorses this site.