The best DFIR metrics are derived, not collected

In professionalizing versus proceduralizing security operations I talked a bit about an experience early in my EMS career where a data-collection exercise intended to improve patient care actually ended up getting in the way instead. That story is very relevant to this topic and I recommend reading the earlier post. The very short summary is: Years ago, as a volunteer EMT, I was part of a pilot program in collecting healthcare metrics. Paramedics and EMTs were given iPads (new at the time!) and asked to document every aspect of our patient care during the call. The medics hated this; paying attention to collecting data during the call directly distracted from actually taking care of the patient, resulting in worse health outcomes.

In technology, If you’ve worked in an operations role of any sort (Helpdesk, sysadmin, datacenter operator, System Reliability Engineering (SRE), etc.) you’ve probably had at least a few run-ins with metrics. While metrics are an essential part of professionalizing an operations team, bad metrics and bad methods^[1] of collecting them are the fastest way to proceduralize your responders into burnout, bore-out, and worse security outcomes.

Why Metrics Matter

When you’re on the ground doing operational work, metrics can easily feel like pointless bureaucracy. After all, you don’t need a dashboard to tell you that there are too many cases, that a critical log source is broken, or that the team has been buried in complex investigations all month. It’s right there in your face.

At their heart, metrics are a translation layer between the subjective experience of doing your operational work and your leadership need to have solid facts to anchor their decisions. As any organization grows, leadership becomes further and further separated from the operational day-to-day. They can’t truly know what’s happening, how it feels, and which work is critical vs. futile. All the same, they’re the ones that have to align with one another on resource allocation, funding, headcount, etc.

Your vice-president (VP) can’t walk into a budget meeting and say "the team feels really slammed right now" and walk out of that meeting with more hiring authority. They need to know, and be able to explain, why hiring an incident responder is more beneficial to the business (or organization) than hiring an engineer to build tools or capabilities that might be able to reduce your team’s workload overall - which, by the way, may be a better outcome for you than obtaining that extra hire!

Finally, you need metrics on the ground as well. A DFIR team (or function) of any size is going to end up performing some critical work, some important work, and some work everyone wishes we could just not do. Measuring and being able to explain the impact of each of those is key to your own arguments about whether that non-critical busywork could and should be automated away by someone else, or even written off as “risk accepted”.

When Metrics Lead You Astray (e.g., Goodhart’s Law)

The collection and presentation of metrics is an age-old source of agony for folks on the ground who are trying to do the job. There’s many ways that metrics collection goes wrong and I’ll only cover some of them here - these are the ones I see most often.

Spending operational time on collecting metrics

I put this one first because it’s been the most consistently frustrating over time for me. It’s essentially the medics-with-iPads story above: Manually collecting data to produce metrics is the most frustrating form of metric collection. This happens when a partner team, manager in another organization, or some regulation comes down and says “We need to know how often X happens”. There’s always a seemingly good reason for this request, so the team acquiesces.

When there’s no good way to derive that answer from existing data, someone will say “OK, let’s create a data point in our tracking system and start recording that.” Pretty soon you have a dropdown box for “root cause”^[4], a dropdown for “product affected”, one for “Number of systems”, one for “Operating system impacted” and before long you’re spending more time filling out the data fields than you spent on the case. I watched this trend play out in real-time over a years-long period at Google and by the time the team finally threw in the towel there were sixty-seven^[5] fields to fill out for each case!

Not only was that too many fields, they weren’t being filled out consistently and so the data analysis all that collection was meant to be driving hadn’t been occurring due to poor data collection. Everyone was literally wasting their time on metrics collection. We eventually declared “Checkbox bankruptcy”, deleted all but a dozen of those data collection boxes, and implemented a strict gatekeeping of any new data collection. I called it “Metrics Triage” and before we would collect any new data as a matter of routine ops, I insisted that the person asking us to collect it do the following:

• Specifically explain what the answers provided by tracking that metric were intended to illuminate (so I could assess whether the thing being tracked would actually answer that question)
• Specifically commit to using those answers to drive a specific structural improvement which would reduce the number of incidents or cases driven by it, and
• Acknowledge that if these structural improvements weren’t begun, with committed personnel, within a reasonable time period, that my team would no longer collect the data.

After this change, we managed to keep the number of manually-tracked fields down to an essential minimum - those fields that we were collecting because they were necessary to track and assign work (e.g., which team is handling the ticket) or were key to moving the case along in the case management system (e.g., status: open, pending, closed, etc).

Time-analysis as a proxy for responder skill/efficiency

Measuring how long it takes responders to do their jobs is another seemingly-obvious, easy metric to collect. You just measure the time the case was opened, and the time it was closed, and track how long it takes Sam to do their work. Then, you try to make that amount of time go down to show improvement! The problem here of course is that Goodhart’s law will always rear its head. Goodhart’s law is often summarized as “When a measure becomes a target, it ceases to be an effective measure.”

Simply put: If you want the time per-ticket to go down, and your responders know it, then the time per-ticket will go down. Not because everyone got better at the job or because the cases became simpler, but because you’re incentivizing the cases to get closed more quickly. You’ll likely be incentivizing things you don’t want, such as:

• Cutting corners in the investigation to get it done quicker (but less thoroughly).
• “Accidentally” hanging up on the customer (for front-line folks) or responders manipulating the queue so that they only get assigned easier (thus, faster) cases.
• People splitting tickets into multiple sub-tickets so that each ticket can be closed immediately upon completion of a small task.

Tracking MTTR (Mean-Time to Resolve) is a surefire way to ensure that anything more complex than average isn’t actually as resolved as you want it to be.

Counting the volume of cases / per-responder equity in numbers

As a manager of a team, part of my job is to make sure resources address bursty operational needs and that there’s equity in workload. I want to make sure a few hard-working people aren’t pulling a lot more weight than others. Among other reasons (basic fairness, knowing who to reward for their work, etc), unaddressed inequity is one of the key drivers of burnout among your high performers. The sense that they’re unfairly treated and that management won’t or can’t fix it, drives a huge attrition risk.

The wrong way^[2] to approach this, though, is counting the number of cases/tickets that are being worked and trying to ensure everyone is working on the same number of things. I spoke a little bit in my other blog, Solving for Negatives, about the amount of time that can be spent on some investigations. A few simple cases that take 20 or 30 minutes each to resolve simply don’t stack up against a three-week-long investigation into a single complex issue.

As noted earlier in Goodhart’s law - your staff are very aware of what this kind of “per person math” means to their workload. I once watched a peer manager implement a new policy that a certain type of unpleasant ops work would be done by “Whoever had the fewest open cases” on his team, in order to make sure work was fairly distributed. The immediate effect of this rule: His team stopped helping anyone else with small questions, little tasks, or performing any assistance to others unless a ticket were opened and assigned to them so their absolute full workload would be reflected in their number of open cases. His intent was to drive better equity; the effect was to incentivize his team to not voluntarily help anyone with anything.

Measuring busy-ness instead of readiness

A manager who expects their response team to always be busy, and begins tracking numbers to ensure “efficiency” in the work, is already beginning to set the team up to fail at a critical moment in the future.

As I noted in About Burnout in Cybersecurity, DFIR is a stressful and bursty job and there’s almost always something big and difficult looming in the future. You need to account for and plan rest, readiness, and training into the team’s time. Tracking measures that attempt to figure out how busy the team is, with the intention of ensuring that they’re always working, is setting your team up to fail on the next php-pocalypse, log4j, or heartbleed because you’ll have an already-exhausted team about to begin a marathon of work.

Deriving the right metrics

To paraphrase one of my recent managers “The best metrics are derived, not collected.”^[3][7]

That fundamental principle is an excellent way to bring the act of metrics collection back to the core business need: Managers need data to understand where to drive resource-allocation, and the data needs to be about the actual challenges the business is facing. That data exists, scattered across all the errata of your day to day operations. The trick is how you find it, aggregate it, and report it.^[6]

The good news is that advancements in AI have made this once-devilish problem much more approachable. In 2020, if I had asked someone “Write me a script that will crawl through all of my google docs, the case tracking system, my emails, my chats, my notes, my command-line history and all the servers I logged into this month and tell me how many hours I spent working on malware execution cases” I would be laughed out of the room, or quoted a year of effort for a senior SWE. Last week, Antigravity+Gemini Argon did exactly that for me in about 45 minutes.

Given this new capability being broadly accessible, here are some things you can attempt to derive from your everyday metadata already being produced, that might help you measure operations.

Response and Time-to-Triage SLAs

Measuring your time to respond and how quickly cases are triaged is highly effective at measuring whether your load has exceeded your capabilities. In particular, measuring the delta between when a problem was detected and how quickly it was mitigated (account keys/credentials revoked; system quarantined, etc.) can show your DFIR team acting as an effective control of last measure.

Why Goodhart’s Law is less disruptive here: While you don’t want to incentivize rushing to close tickets (thereby doing less investigation) it’s totally appropriate to speed toward the initial response, and it’s likely a good decision to err on the side of mitigating an attack without a full investigation being completed - as opposed to closing one. Time-to-Mitigate is also tracked as a broad team-level impact vs being a responder-level performance judgement.

Measuring time spent on different categories of casework

I’ve recently started using AI models to group all our cases into categories that are human-understandable and labeled into “missions”, such as “Actual attack”, “Malware”, “Assisting Legal”, “Supporting other teams”, and “Accidental policy violation”. Tracking this on a monthly basis helps me identify where engineering controls could reduce our case load - which is essentially what leadership in the past were hoping the 67 manually-collected infoboxes could provide.

Another helpful thing to track here - are certain responders the only ones working on certain categories of cases, because they’ve become the expert over time? If so, you need to focus on cross-training to avoid single points of failure.

Note: LLMs can be and often are wrong about the categorizations. One way I’ve worked to improve accuracy is that our agent keeps a “Categorization override” file. Each time I run it, I review the results and any case whose categorization I change goes into the override file and is used as a reference to drive further refinement of the underlying prompt. Make sure a human always has the final say in your AI output!

Measuring wait-time

Tracking periods of inactivity on cases that depend on other teams to act has helped me identify when our team is held up by other teams being resource constrained. Removing those teams as bottlenecks, or reducing our own commitment to that work until those other teams are staffed may be appropriate responses.

Finding the “Hidden work”

One of the biggest gains I’ve gotten from having given AI models access to our team’s chat is that I can now measure the work that has never been tracked at all in our case management system. Lately I’ve been able to highlight how often our on-callers are asked by other teams to provide advice, guidance, peer reviews, etc. (~9 times per day on average, if you were wondering). This work is critical to our broader organization but until now has been impossible to really clearly show the value and extent to which it occurs.

Remember the manager whose team stopped answering questions without a ticket filed to track their time? They can reignite the spirit of collaboration now that this work can be derived instead of tracked.

Tracking Bursts not just averages

One of the things that really stresses a DFIR team are bursts of workload. Most ops teams aren’t blessed with a single consistent and unchanging workload over time; things ebb and flow and sudden bursts of major cases all seem to happen at once. You can track the number of concurrent high-severity cases on a daily basis, over time, and should attempt to plan your team’s capacity at the burst level, not the sunny-day-average level.

Tracking off-hours work

This one is often considered but becomes overlooked. You should be tracking how often cases are reported, assigned, and handled outside of usual business hours. Has your team’s “on-call weekends” slowly become “We work the weekends, too?” - you may need to realign with the business (or even local labor law!) on this one.

Note that there are security/operational security (opsec) and privacy tradeoffs that’s always involved when you allow AI access to sensitive data - such as your case management system, your team’s chats, etc. Don’t forget that a lot of this data is attacker-controlled initially, so it may also carry the risk of prompt injection. You’re going to want to ensure you’re operating your AI in strongly-protected and strongly-guardrailed environments. If you can arrange for them to have access only on-demand to run the statistics-gathering routines and be denied persistent access, that’s even better!

It’s time to derive some change!

I believe that the key role of a DFIR (or other ERT/IRT) lead is to ensure the team (responders) have all the support to do an excellent job so they are best to support the organization. To obtain this support you’re going to need data, facts, and strong arguments about why those resources will result in good business outcomes. Metrics are an unavoidable part of this process. Learning how to effectively measure your contributions - and where the pitfalls of ineffectively measuring them are - will help you achieve the best possible support for your team.

But whenever we design a process or a dashboard we have to remember who is holding the iPad in the back of the ambulance. Measure what your systems automatically produce. Measure what pushes your engineering teams to build safer architectures. Measure what protects your responders' capacity to think critically and do their job well in the moment. And whenever a metric gets in the way of taking care of the patient, put down the iPad.