The hardest part about an outage isn’t when things are going wrong, customers are calling you non-stop and your app is unresponsive to anyone and anything that breathes on it.
Believe it or not, this is the easy part — primarily because everyone who has ever been near an emergency of any kind in their life knows what to do.
- Stop the Bleeding.
- Triage.
- Patch.
- Next.
In software speak…
- Reduce the impact on other customers.
- Identify what we have to do now to get through this.
- Make and apply patches to the system.
- Move on to the next issue that you know is going to arise and hit you smack dab in the face the moment you deploy the first patch.
And that is the easy part, everyone is gathered around, they have been told to drop everything, come together as a team, work on whatever areas they have been assigned, triage, patch, next, move on.
The hard part comes after you’ve deployed that last fix and your customers have started to calm down, the cases are dropping, things are getting back to normal and you are trying to figure out what exactly happened, what went wrong, and most importantly what do you need to avoid this from happening…
