Playbooks and Response Plans: Striking the Right Balance

I happened to see a Techcrunch article this morning titled “US cybersecurity agency CISA had to build its incident playbook during the incident, agency reveals”. What piqued my interest is that the article’s focus seemed to be on incident playbooks - a favorite topic of mine! For those who didn't click, the thrust of the article is that CISA "Had no playbook" for an incident and that not having had a playbook was an indisputably bad thing.

Disclaimer: I don’t know the full specifics about what went on at CISA and “You posted a credential to github” is a very basic and easily imaginable scenario, so it is one of those things you should have a playbook for. However this quote from CISA stuck out to me: “The agency said it is important to prepare playbooks for all anticipated needs to ensure that organizations are ready to respond in the event of a security incident rather than scrambling to improvise one in real time.”

I think "all anticipated needs" pushes the principle a little too far.

Playbooks are useful but they're one means to an end.

Should we have playbooks? Yes - for some things, playbooks are very useful. For example, playbooks that cover a complex technical topic (i.e, how to do a key revocation) are highly useful because that thing might need to be done by anyone in the team, and missing a step or getting a command wrong can turn out very bad.

More than that, we should have response plans for situations. A response plan covers the broad “If X happens, how do we want the team to respond and coordinate? Who needs to know? What do they need to know and when?” aspects of an incident.

Should we have playbooks and response plans for ALL the things we could possibly anticipate, though? I think not, and here's why..

Predicting your next incident is terribly difficult.

IT environments today are often in a constant state of "Change is normal". Systems change, threats change, attackers adapt and you can't spend all your time imagining the future, nor can you predict what will actually hit you when it hits. For years I’ve been running tabletop exercises and “what if” scenarios as training for my team and two things stand out as extremely common after-effects:

1. My imagination consistently fails to actually land on the next major threat we have to face, and
2. When I DO successfully land on the right threat, our tabletop or exercise often fails to anticipate some major system change which takes place between the exercise and the event.

This isn’t to say that these tabletops and hypotheticals aren’t impactful - they’re incredibly useful! But their utility lies more in exposing the changes from the previous exercise and keeping us practiced at pivoting and working “in the moment” - they’re much less useful at producing a playbook that will “solve it when it happens”.

If you build it, you must maintain it!

In preparation (or postmortem after an incident) it's easy to proclaim "If you can imagine it, you should write a playbook for it" - but this doesn't take into account how much operational work then needs to go into monitoring, maintaining and updating those playbooks.

As noted earlier, IT environments are in a constant state of change. A specific playbook must be reviewed and updated regularly to account for those changes. If your team’s policy is to imagine future incidents and write playbooks for them but you’re not devoting significant time to then regularly reviewing and updating them, you’re likely making things worse for your future incident and not better; There’s a fire alarm and extinguisher on the wall but the alarm doesn’t alarm and the extinguisher won’t extinguish.

Striking the right balance & what TO rely on

Nothing above is to say “Don’t write playbooks and response plans” - you need them, for sure.

You should have a broad response plan that guides your team on how to approach problem-solving and assigning roles during incidents. You should have specific playbooks for high-likelihood things if there are steps that need to be performed a certain way or in a certain order for them to be effective. You should have as many playbooks and response plans as you will routinely need and a crisis plan for when everything is on fire - but not more than that.

To paraphrase the old Mastercard commercials: "There are some things imagination can predict. For everything else, there’s Incident Management Protocols.”

Beyond the responses that are clearly anticipatable you should be cultivating a team culture of trust, professionalism and accountability. These are key pillars of an effective response team.

Trust: With team members who trust one another (including management layers!) and a high-level set of expectations, you’ll have the flexibility for the responder who is there at the time to do what’s needed with what guidance exists quickly and effectively whether the specific event they’re looking at has a playbook or not.

Professionalism: I have written a lot more extensively on this topic in another blog post - but the core concept here is that your team should be performing as a team, united in a clear mission, and should be bought into that mission. The team needs to want to fix the problem and want to be the ones to fix it - and they need to dedicate some amount of time to training and practicing their skills.

Accountability: The way to perform is to constantly improve and the way to improve is to be accountable for your performance. A team that does postmortems and can recognize their mistakes and gaps, improve them, and be better prepared for the next event will perform whether there’s a specific written instruction for the event or not.

Tending to these key pillars will allow the team to work through today's problem under today's environment and with today's tools - not whatever you thought those might be back when the playbook was last touched. So build that team and write whatever playbooks you need and whatever response plans you need - then stop writing and start training.