Blame it on YOU for the damn false-positives!

indexBelow  is a list of 6 facts (and counting) you should know before whining and complaining around the infamous false-positive (FP) topic. If you’ve been there, feel free to comment and share your pain or your own facts.

As you know, the FPs are everywhere and multiplying just like Gremlins after a shower! [Misled Millennials, click here] In fact, that seems to be part of 12 in every 10 Security Monitoring projects out there, including those heavily relying on SIEM technology.

So here’s the list of facts and some quick directions:

#1 Canned content sucks

Bad news. Out-of-the-box things don’t go well with Security Monitoring, my friend. Your adversaries are quite creative and you need to catch up.

But before bashing  vendors, think about it from their perspective. Would you release a product without any content? They are there for providing an idea of its usage, they are meant for general cases. You need to RTFM and spend a great deal of time evaluating and testing the product features to enable customization.

If you are into #photography, let me ask you: are you still shooting on automatic, generating JPEGs, with lower resolution (more shots!) out of your fully customizable camera? If you want to boost your chances of having that perfect shot, you need to go manual and experiment with all possible settings.

The technology is just another tool. Regardless of the feature set, you should be able to get the best out of it by translating your knowledge into a tailor-made rule set.

#2 You don’t have a scope

Since it’s pretty much utopia consuming a risk analysis result as an input for your Security Monitoring program, you need to find a way to define an initial scope. This will drive the custom rules development cycle and define the monitoring coverage (detective controls).

We tend to go with low hanging fruits or quick wins, but even then, you still need to spend some time defining your goals. And here the “Use Cases” conversation starts, perhaps one of the most important from the program!

Actually, it deserves a full article on its own, but I will leave you with a simple, yet interesting approach delivered by Augusto Barros (Gartner). I liked it when he pretty much defines the scope as the interception between Importance (value) and Feasibility (effort).

#3 People > Technology

No matter how cool and easy to use the technology is, if you don’t have a skilled team to work on #1 (RTFM and experiment) and #2 (define goals), you will likely end up with a bunch of unattended alerts.

Therefore, don’t hire deployers, hire Security Content Designers, Security Content Engineers, enable them to extract the value of your security arsenal (investment).

#4 Exception handling is not optional

When I say tailor-made, that means your rule should already address exceptions to a certain extent, that’s also called whitelisting.

If during  development you realize a rule is generating too many alerts, try to anticipate the analysts call, and filter out scenarios that are not worth investigating.

The technology MUST provide easy ways to quickly add/modify/remove exceptions. Bonus for those products providing auditing (who, when, why).

#5 Aggregation is key

Have you noticed how many alerts are related to the very same case or target?

Some technologies (and security engineers), especially applicable to SIEMs, dare generating the very same alert – multiple times – simply because the time range considered for processing the events is overlapping, longer than the rule’s execution interval itself. “Run every hour, checking the last 5 days”. WTH?

What about aggregating on the target (or victim, if you will)? Not all products provide such functionality, but most SIEMs do. So instead of generating multiple alerts for every single hit on your data stream, why not consolidating those events on a single alert? Experiment aggregating on unique targets to start with.

#6 Threat Intel data != IDS signature

Here, there’s not much to say, with all that Threat Intel porn hype nowadays, just go ahead and read Alex Sieira’s nice blog post: Threat Intelligence Indicators are not Signatures.

Help spread the love for FPs and share your thoughts!


Splunkers on Twitter

Below is a list of Splunk users I am following on Twitter, including Splunkers, partners and awesome users. Most of them are also into #Infosec. The list is not sorted in any particular order.

Missing someone, maybe you?! Please feel free to contact me for adding more. In case you want to follow a list, it is also available via Twitter here.

Ryan Kovar @meansec
Staff Security Strategist @Splunk. Enjoys clicking too fast, long walks in the woods, and data visualizations.

Holger Sesterhenn @sesterhenn_splk
Sales Engineer, CISSP, Security Know-How, Machinedata, Security Intelligence, IoT, Industrie 4.0, BigData, Hadoop, NoSQL, User Behavior Analytics

The Dark Overlord @StephenGailey
Towering intellect; effortlessly charming…

Cédric @_CLX
Let me grep you. #infosec and useless stuff. Using security buzzwords since 2005.

Brad Shoop @bradshoop
Security Onion for Splunk app developer, infosec, devops, infrastructure, cloud and homebrewer.

monzy merza @monzymerza
Chief Security Evangelist @Splunk. Thoughts are my own.

Damien Dallimore @damiendallimore
Splunk Dev Evangelist, Golfer, Rugby Player, Musician, Scuba Diver, Thai linguist, Chef.

Adam Sealey @AdamSealey
Information security, both applied and research. CSIRT, DFIR, and analytics Generalist geek. Husband & father of 3. Tweets are my own.

Hacker Hurricane @HackerHurricane
Austin TX. area Information Security Professional

Mika Borner @my2ndhead
Splunk Artisan. Because Splunking is an art.

David Shpritz @automine
I Splunk all the things. Blieve, hon. Splunk, Web App Sec, Open source, EDC

Dimitri McKay @dimitrimckay
Glazed donut connoisseur, plus size hand model, technologist, splunker, replicant, security nerd, CISSP, MMA fighter, zombie killer & lover of pitbulls.

Luke Murphey @LukeMurphey
Developer of network security solutions at #splunk. Founding member of Threatfactor ( ) and Converged Security (acquired by GlassHouse).

Sebastien Tricaud @tricaud
Principal Security Strategist @Splunk. Playing with data, binary-ascii-utf16-whatever. Opinions are my own, not my employers. Re-tweeting != Agreeing

Dave Herrald @daveherrald
dad | husband | splunk security architect | GIAC GSE | tweets=mine

Ryan Chapman @rj_chap
Security enthusiast. Incident response analyst. Malware hobbyist. Retro game lover. Husband and father. TnVsbGl1cyBpbiB2ZXJiYS4=

Michael Porath @poezn
Product Manager for Data Visualization @splunk. Bay Area based Swiss Information Scientist

skywalka @skywalka
my daughter, basketball, hip hop, film, comics, linux, puppet, splunk, nagios, and sensu keep me awake

James Bower (Hando) @jamesbower
Pentester / Threat intelligence / #OSINT / #Honeypots / #Bro_IDS / #Splunk | #Python / Follower of Christ and occasional blogger –

georgestarcher @georgestarcher
Information Security, Log analysis and Splunk, Forensics, Podcasting. Photography and OSX Fan. GnuPGP key ID: 875A3320BD558C9E

Brian Warehime @brian_warehime
Security Analyst | Threat Researcher | #Honeypots | #Splunk | #Python | #OSINT | #DFIR

Michel Oosterhof @micheloosterhof
Splunk // My opinions are my own.

Hal Rottenberg @halr9000
I am the Lorax. I speak for the Developers! @Splunk, Author, Podcaster @powerscripting , Speaker, #PowerShell MVP, #CiscoChampion, husband, father of four!

Jason McCord @digirati82
Security analyst, software developer, #Splunk fan. Log everything. #WLS #DFIR


Challenge your MSSP/SOC/CSIRT: what metrics can they provide you?

I was trying to recall a famous quote related to “Metrics” for including here and below is what Mr. Google hints me:

The quote has a few variations, but that seems to be the most famous one. Perhaps now it will finally stick. So, does it make sense or is it just another unquestioned corporate adage?

Basically, the idea here is to give you more food for thought in case you are into this metrics thing and trying to apply it to Security Operations.

Actually, let me start by saying I like measuring data, therefore metrics is an interesting topic to me. Simply put, translating your effort and progress to management is way easier if you are able to come up with a metric from which they can understand what you are doing and why.

As usual, bonus points if a metric ties to a business goal (more info below). So working on a good, easily digestible metric also saves management time assuming this one is not there only for you, nor can it be allocated quickly. Therefore, selecting key metrics and meaningful charts is an opportunity security practitioners cannot miss in order to keep their budgets flowing in.

Many questions, few metrics

How do you evaluate the work done by your SOC or SecOps team? How to verify your MSSP is providing a good service?

Within Security Operations, and I dare using this term to refer to the tasks carried out by MSSPs, SOCs or CSIRTs, you should generate metrics that help or enable answering the following questions:

  1. How many investigations ended up being a false positive (FP) or a real threat (TP)?
  2. From above answers, what scenarios are seen or involved most often? Is there a technology, NIDS signature, correlation rule or process clearly performing better (or worse) than others?
  3. Which analysts are involved in the process of developing or tuning signatures/rules that lead to real investigations?
  4. In a multi-tier environment, which analysts were responsible for the triage of most FP cases?
  5. MSSP only – Are customers responding or interacting with cases that are raised towards their security teams?

Linking Metrics to benefits

Now, read question #1 and ask yourself: Do you really believe a properly deployed security infrastructure will never, ever detect a real threat? So why are you still paying a MSSP to provide you with anything but FPs? Checkbox Security?

No wonder why your Snort/Bro guy, with a single sensor is able to provide 10 times more consumable alerts than your 5 super-duper Checkpoint NG IPS Blades? Track answers from questions #2 and #3 to find out.

From #4 you will have a better idea about where to invest your budget for training and which analysts might need some mentoring.

Many incidents evaluated doesn’t mean people are busy on analysis, nor does it mean good work. The higher the FP rate on the SOC escalations, the less interest your customer will have. That indicates less engagement on following up the investigations. Refer to #5.

And what about the relationship with business goals? That’s easier to exemplify for MSSPs: sounding metrics performing as expected are the best ammunition you can bring to the table for contract renewals or (ups!)selling.

Here are some (measurable) metrics examples:

  • Alerts to Escalations ratio
  • Escalations to real investigations ratio
  • Alerts handled per shift/analyst
  • Time to triage (evaluate a new alert)
  • Time to close an investigation (by outcome)
  • Number of FPs/TPs per rule, signature, use case

If you embrace Gamification, there are many more that might be interesting assuming the risks involved here, for example: Escalations to real investigations (TPs) ratio per analyst or shift.

No Case Management = No Game

An investigation must have a start and an end, otherwise it’s impossible to measure the output of it. Even if you want to monitor an attacker behavior for a while, this decision (observe, follow-up) was most likely the result of an investigation.

Now, scroll up to the list and ask yourself how many of those questions are easily answered by data mining the ticket or case management database. Doing Analytics on your case management DB might be challenging but definitely worth it.

“I don’t have a case management system!”, then, go get one before you start the metrics conversation. If you don’t have an incident workflow in place, those systems might even drive you towards designing one.

Happy to discuss that stuff further? Feel free to comment here or message me on Twitter.