I was trying to recall a famous quote related to “Metrics” for including here and below is what Mr. Google hints me:

The quote has a few variations, but that seems to be the most famous one. Perhaps now it will finally stick. So, does it make sense or is it just another unquestioned corporate adage?

Basically, the idea here is to give you more food for thought in case you are into this metrics thing and trying to apply it to Security Operations.

Actually, let me start by saying I like measuring data, therefore metrics is an interesting topic to me. Simply put, translating your effort and progress to management is way easier if you are able to come up with a metric from which they can understand what you are doing and why.

As usual, bonus points if a metric ties to a business goal (more info below). So working on a good, easily digestible metric also saves management time assuming this one is not there only for you, nor can it be allocated quickly. Therefore, selecting key metrics and meaningful charts is an opportunity security practitioners cannot miss in order to keep their budgets flowing in.

Many questions, few metrics

How do you evaluate the work done by your SOC or SecOps team? How to verify your MSSP is providing a good service?

Within Security Operations, and I dare using this term to refer to the tasks carried out by MSSPs, SOCs or CSIRTs, you should generate metrics that help or enable answering the following questions:

1. How many investigations ended up being a false positive (FP) or a real threat (TP)?
2. From above answers, what scenarios are seen or involved most often? Is there a technology, NIDS signature, correlation rule or process clearly performing better (or worse) than others?
3. Which analysts are involved in the process of developing or tuning signatures/rules that lead to real investigations?
4. In a multi-tier environment, which analysts were responsible for the triage of most FP cases?
5. MSSP only – Are customers responding or interacting with cases that are raised towards their security teams?

Linking Metrics to benefits

Now, read question #1 and ask yourself: Do you really believe a properly deployed security infrastructure will never, ever detect a real threat? So why are you still paying a MSSP to provide you with anything but FPs? Checkbox Security?

No wonder why your Snort/Bro guy, with a single sensor is able to provide 10 times more consumable alerts than your 5 super-duper Checkpoint NG IPS Blades? Track answers from questions #2 and #3 to find out.

From #4 you will have a better idea about where to invest your budget for training and which analysts might need some mentoring.

Many incidents evaluated doesn’t mean people are busy on analysis, nor does it mean good work. The higher the FP rate on the SOC escalations, the less interest your customer will have. That indicates less engagement on following up the investigations. Refer to #5.

And what about the relationship with business goals? That’s easier to exemplify for MSSPs: sounding metrics performing as expected are the best ammunition you can bring to the table for contract renewals or (ups!)selling.

Here are some (measurable) metrics examples:

• Alerts to Escalations ratio
• Escalations to real investigations ratio
• Alerts handled per shift/analyst
• Time to triage (evaluate a new alert)
• Time to close an investigation (by outcome)
• Number of FPs/TPs per rule, signature, use case

If you embrace Gamification, there are many more that might be interesting assuming the risks involved here, for example: Escalations to real investigations (TPs) ratio per analyst or shift.

No Case Management = No Game

An investigation must have a start and an end, otherwise it’s impossible to measure the output of it. Even if you want to monitor an attacker behavior for a while, this decision (observe, follow-up) was most likely the result of an investigation.

Now, scroll up to the list and ask yourself how many of those questions are easily answered by data mining the ticket or case management database. Doing Analytics on your case management DB might be challenging but definitely worth it.

“I don’t have a case management system!”, then, go get one before you start the metrics conversation. If you don’t have an incident workflow in place, those systems might even drive you towards designing one.

Happy to discuss that stuff further? Feel free to comment here or message me on Twitter.