Posted by anton on April 8, 2011 under SIEM | 
One of the ugliest and most painful issues with SIEM is resourcing. Yes, that SIEM appliance might set us back $75,000 in hard earned security budget dollars, but how much more will we have to spend in the next 3 years deploying, integrating, using, tuning, cursing, expanding the thing? How much manpower will the new operational procedures (example) cost us? And if we actually build a SOC or “a virtual SOC”, how much will we have to spend on an ongoing basis to get the value and benefits? In fact, how much will the coffee cost if we have to work 20 hours in a row recovering that crashed SIEM database partition?
These and other questions are super-important for every SIEM and log management project. So:
NEWSFLASH! SIEM costs money. “Free” SIEM (example) costs money too, BTW. Let’s try to delve into what those costs are. I will be not-quite-scientific in regards to real “hard money” costs (e.g. software license purchasing) and “soft costs” costs (e.g. staff time costs), but I will try to clearly mark each kind of SIEM cost below.
First, assumptions and limitations:
- This is NOT “SOC staffing” , but simply “running a SIEM” staffing. SOC implies more processes and more tasks and a broader mission.
- Assumes in-sourced, traditional “buy and run” SIEM; outsourced, co-managed/co-sourced cost model would look different.
Here is the rough cost model for some of the most common SIEM cost categories:
- Initial “hard” costs
- SIEM license costs: base price +per user, per node, per EPS, per CPU (and per CPU core), etc costs – however your favorite vendor charges
- For software SIEM, also hardware, OS, database costs for as many servers as you need
- If any, mandatory 3rd party software license costs (occasionally, agents, reporting tools, etc)
- If chosen, vendor or consultant deployment services costs. If not chosen, staff time for deployment will pop out in soft costs below!
- Vendor training or 3rd party training on logs, log management, SIEM, etc
- Additional external storage (in most cases)
- SIEM ongoing, operating “hard” costs
- Various SIEM vendor services: support (typically mandatory), ongoing professional services costs
- Personnel to operate a SIEM: from part of FTE (very small scale, few use cases for a SIEM) to 1 FTE (small appliance deployments) to many FTEs of various roles (+much more for SOC staffing if live monitoring is implemented). 0 FTE for SIEM = SIEM project FAIL with 100.00% probability.
- Periodic or occasional “hard” costs
- Various SIEM vendor services: professional services, custom development work for device integration (some of these may go into soft costs if done internally – for advanced organization or those experienced with SIEM already)
- Periodic staff training on SIEM operation and tuning
- Specialty staffing: DBA, sysadmins, node sysadmins, in-house developers for custom connectors, Crystal Reports administrator (yuck!), etc – some of these might go into “soft” costs if “poaching” existing personnel time
- Deployment expansion costs: same as initial costs, but for additional systems, software, hardware, etc as you grow; these sneak up really fast if SIEM is licensed using many dimensions such as user+CPU+node+server+something else.
- External storage expansion costs – yes, you will buy more storage if your volume grows, and log retention time stays the same (e.g. 1 year for PCI DSS)
On the other hand, here are some of the “soft” costs, such as time expenditures by existing resources:
- Initial “soft” costs
- Deployment time for the SIEM project – allocate more time if deploying purely using internal personnel, not vendor or consultant
- Log source configuration and integration – this will likely take way more than simply installing the tool. This is what makes SIEM deployment projects go for months in complex, distributed organizations with many silos.
- Initial tuning, content creation and adapting the tool to your environment (however lightweight it may be)
- Training and other staff time costs to jumpstart the operation (Congratulations! You bought ta SIEM. Now you need to operate it…)
- SIEM ongoing, operating “soft” costs
- Report review and other ongoing monitoring tasks – from 24/7 to daily to weekly
- Alert response and escalation; SIEM implies correlation and automated alerting
- Other “using SIEM” tasks such as reviewing the dashboards
- Uptime maintenance tasks i.e. caring for your SIEM as well as storage – backups, updates, minor troubleshooting, etc
- Periodic or occasional “soft” costs
- SIEM rule tuning, reports creation, dashboard customization, new log source integration, other ongoing SIEM tasks
- Periodic training and related staff time costs
- Expansion: same as initial soft costs
As was suggested here on our SIEMusers.org mailing list, it is useful to separate soft costs that are mandatory FOR SIEM operation from those which commonly arise FROM SIEM operation. The most obvious example is incident response due to increased awareness of network and system activities, delivered by your SIEM.
”Soft” costs that commonly result from SIEM:
- Added cost of incident response: more incidents are likely to be detected
- Resulting incident remediation costs and even cost of new technologies deployed for preventing the discovered issues
- Other department personnel time for dealing with issues revealed by SIEM – the soft costs do leak out of the security department to IT and even beyond (legal, HR, etc).
Anything big I missed that you experienced? BTW, in my experience, I have seen the total cost of a SIEM project (hard + soft) range from 10% of SIEM license costs (for shelfware SIEM “deployments”) to a mind-boggling 20x of license cost…
Posted by anton on March 1, 2011 under SIEM |
Quick question: How do I go about writing a SIEM or log management RFP?
Quick answer: don’t. This “purchase method” is probably equally hated by vendors and end-users. As somebody who was “volunteered” to help sales folks with 1600 page (yes, really!) and smaller RFPs more than once during my [i.e. Anton's] “vendor years,” I can tell you that – with a tongue firmly in cheek:
a) if you ask a vague question in your RFP, you will get either a “Yes” or a nice blurb taken from a random location in vendor datasheet
b) if you ask a question starting with “How…?”, you will get a nice blurb taken from a vendor datasheet
c) if you ask a silly question (“do you have an Albanian language interface?”), you will get either a “Yes” or a nice blurb taken from a random location in vendor datasheet
d) if you ask a question that is impossible to answer (“Can your product handle the high load?”), you will likely get a “Yes” – surprise!
e) if you ask an honest question that might cast a product in a negative light (“will you every lose log data?”), what do you think you will get….?
See a theme emerge here? Note that I am not trying to imply that any particular vendor would lie in their RFP responses – the term here is “defensible creative exaggeration.” BTW, what do you think happens when a standard enterprises RFP template collides with a standard vendor RFP “boiler plate” response? Boom! The explosion of high-grade concentrated idiocy… And if you think that I am a bit cynical about this whole thing, than maybe you are correct… making sausage for a long time does distort one’s personality a wee bit 
Despite the above, there are two exceptions to this rule of not doing RFPs:
- You are obligated to do a RFP (government, etc)
- You’d like to use your RFP as a chance to distill and focus your SIEM/LM requirements.
Let’s address them both at the same time. If you are case #1 above, you should really turn it into case #2. As you recall (if not, review these posts here), one of the most important things an organization should do before buying a SIEM is to set its own goals, requirements, use cases, etc. BTW, this recent SIEM presentation stresses the same point – esp. see slide 16 and around. This older presentation has some things to avoid at the product selection stage – see “worst practices” 1-4.
So, based on my experience on both sides of the RFP “interface”, here are some of my SIEM RFP tips:
- Keep it short! If you cannot express what you need in under 10 pages, go back and rethink it. “Every time an organization releases a 500+ page RFP, God kills an intern.” Yes, that very intern who is tasked with responding to that monster, of course.
- Start from your REAL main reason for getting a SIEM, your problem statement – monitor PCI DSS CDE, perform IDS/IPS alert analysis, monitor servers for suspicious logins, protect web applications via log correlation, etc
- Include your use cases – which simply means to describe how you plan to use the system and what you expect the system to do for you. Some examples are shown here (more high level) and here (more detailed inside the whitepaper).
- Based on your goals and use cases (and that is important!), describe SIEM product functionality that is essential for your mission: agentless collection, bandwidth throttling, rule-based correlation, visual dashboards, trend reports, whatever…
- Include log sources / devices that you absolutely need supported and what you mean by “supported” (e.g. parsed, normalized, categorized, suitable for correlation, covered by default correlation rules, updated promptly when log source changes, etc). This area is notorious for extra-high volume of “creative exaggeration” (“of course we support VidgetMaster 7.2! – via our generic LogMahgic 1.0 (TM) collector … which dumps log files right into storage without analysis … and then rotates them to oblivion within 7 days”)
- Avoid or reduce the usage of vague terms: ”scalable”, “high”, “flexible”, “effective”, “advanced”, “automatic”, “proper”, etc. Why tempt the other side unnecessarily?
- Clarify most other terms, even those that look clear to you: “correlation”, “reporting”, “keyword search”, “trend”, “responsive”, etc
- Size the environment before writing an RFP, as we discuss in LogChat #2. Baseline your log sources for 2-4 weeks to get your average EPS rate then include both the volume of data and number of log sources that you absolutely need supported. Also, specify response time for reports and searches while you are at it.
- Make phases of your SIEM project clear up front – don’t say “400,000 devices and 4,000,000 EPS enterprise-wide.” I got news for you – you probably will never get there… Be very clear about your Phase 1-2 and simply keep later phases in mind for the coming years.
- Try hard to avoid idiotic statements (sorry!): “Vendor MUST specify their efforts and processes to guarantee that products and services provided will completely satisfy us or exceed our expectations” (quote from a real RFP)
And – hold on to your pants – despite the above effort you should be prepared to take the responses with a HUGE grain of salt. One of my contacts on the enterprise side put it simply: “of course we ignore all the specifics in RFP responses.” 
With this approach to RFP writing, you WILL still benefit even if you don’t read the responses…
Finally, a more useful question than “how do I write a SIEM RFP?” is “how do I buy the right SIEM for my organization?” Keep this in mind while tuning your RFP. We at SIEMusers.org plan to cover the subject of SIEM RFPs in more depth in the future, stand by. Care to help with a community SIEM RFP template?
Posted by anton on November 5, 2010 under Announcements, Log Management, Logging |
Knowing how much people love checklists, here is one more: a checklist for comparing log management tools.
It is being released at the new log management related site, Log Management Central (subscribe to RSS, follow on Twitter).
- The announcement and brief description is here.
- Printable PDF version is here.
- Spreadsheet XLS version with adjustable criteria scoring is here.
Disclosure: creation of this checklist was funded by a vendor, but it did not affect my choice of criteria or any other content decision. It also does not reduce awesomeness in any way! In other words, it is up to you how to use it (and whether to use it) and what decision to make after evaluating the tools. Just don’t make a decision of letting your logs rot 
Please feel free to make suggestions to make the checklist more useful! Is anything missing? Worded in a non-vendor neutral way? Anything else?
Possibly related posts:
Posted by Joe on October 13, 2010 under Announcements, Netwitness |
Hello all,
Just a quick announcement that the Netwitness User Conference is just around the corner!
Registration is free for this event and closes on October 18th.
You can register here:
http://www.netwitness.com/userconference/register.html
Posted by anton on September 3, 2010 under Log Management, Logging |
As promised, here is another “Top 11 Reasons” which is about log analysis. Don’t just read your logs; analyze them. Why? Here are the reasons:
- Seen an obscure log message lately? Me too – in fact, everybody have. How do you know what it means (and logs usually do mean something) without analysis? At the very least, you need to bring additional context to know what some logs mean.
- Logs often measure in gigabytes and soon will in terabytes; log volume grows all the time – it passed a limit of what a human can read a long time ago, it then made simple filtering ‘what logs to read’ impossible as well: automated log analysis is the only choice.
- Do you peruse your logs in real time? This is simply absurd! However, automated real-time analysis is entirely possible (and some logs do crave for your attention ASAP!)
- Can you read multiple logs at the same time? Yes, kind of, if you print them out on multiple pages to correlate (yes, I’ve seen this done
). Is this efficient? God, no! Correlation across logs of different types is one of the most useful approaches to log analysis.
- A lot of insight hides in “sparse” logs, logs where one record rarely matters, but a large aggregate does (e.g. from one “connection allowed” firewall log to a scan pattern). Thus, the only way to extract that insight from a pool of data is through algorithms (or, as some say, visualization)
- Ever did a manual log baselining? This is where you read the logs and learn which ones are normal for your environment. Wonna do it again? Log baseline learning is a useful and simple log analysis technique, but humans can only do it for so much.
- OK, let’s pick the important logs to review. Which one are those? The right answer is “we don’t know, until we see them.” Thus, to even figure out which logs to read, you need automated analysis.
- Log analysis for compliance? Why, yes! Compliance is NOT only about log storage (e.g. see PCI DSS). How to highlight compliance-relevant messages? How to see which messages will lead to a violation? How do you satisfy those “daily log review” requirements? Thru automated analysis, of course!
- Logs allow you to profile your users, your data and your resources/assets. Really? Yes, really: such profiling can then tell you if those users behave in an unusual manner (in fact, the oldest log analysis systems worked like that). Such techniques may help reach the holy grail of log analysis: automatically tell you what matters for you!
- Ever tried to hire a log analysis expert? Those are few and far between. What if your junior analysts can suddenly analyze logs just as well? One log analysis system creator told me that his log data mining system enabled exactly that. Thus, saving a lot of money to his organization.
- Finally, can you predict future with your logs? I hope so! Research on predictive analytics is ongoing, but you can only do it with automated analysis tools, not with just your head alone (no matter how big ) …
Posted by anton on under Log Management, Logging |
Following my now-famous Top 11 Reasons to Collect and Preserve Computer Logs and Top 11 Reasons to Look at Your Logs, here is the promised “Top 11 Reasons to Secure and Protect Your Logs”
- Let’s review why you are reviewing logs. Will logs that might have been changed by somebody, somewhere, somehow still be useful for items 1-11 from here? No? Secure them!
- Oooh, logs in court? Challenges abound! To respond to them, one needs to protect the logs so you can claim that they are both authentic and reliable.
- A human error still beats an evil hacker as the main cause of IT problems. Are your logs safe from it? Available when needed? Protect them from crashes and other faults!
- PCI DSS just says so: “Secure audit trails so they cannot be altered.” Wonna do it- or pay the fines?
- Do you protect financial records? Identity info? Passwords? Some of it ends up in logs – thus making them more sensitive. Secure the C-I-A of logs!
- Do you look at logs during incident investigation? Do you want them to be “true” or full of random (if creative…) cr*p, inserted by the guilty party? Secure the logs!
- Think that “attacks vs logging” are theoretical? Think again. Are your logs safe or vulnerable? Is your logging tool 0wned?
- Syslog + UDP = log injection. Are you protected (reliable TCP, confirmed delivery, encryption – SSH, SSL, VPN)?
- Why change logs? No, really, why change logs? If you never change logs – and you never should – hash them right away after collection to make them immutable.
- Logs are backed up on tape – who will see them? Well, whoever restores the tape, that’s who! Encrypt them to protect them from accidental and malicious disclosure if tape is lost.
- Why log access to logs? Same reason why you had the logs in the first place – to review who did what. Who broke through and stole the logs? Who browsed them without permission? Only logs will tell – if you have them!
Overall, one need to strive for having no holes in log safeguards from log birth to analyst conclusion based on log information…
Posted by anton on under Log Management, Logging |
As promised, I am following my Top 11 Reasons to Collect and Preserve Computer Logs with just as humorous and hopefully no less insightful ”Top 11 Reasons to Look at Your Logs.”
- The first reason is again disarmingly simple (is it ). Read PCI DSS lately? Glanced at HIPAA? Suffer underFISMA? Yup, all of the above say that you must not only have, but also review logs periodically.
- Are you 0wned? How do you know if all your logs are stashed on a tape in a closet? Look at them! Now!!
- An incident happens. Really, who needs extra motivation to look at logs in this case? Duh! Logs for incident response is a “no-brainer” use case for log review.
- Users – from CEO to a janitor. You might have to know what they do on your IT systems! How? Read the logs! Everybody leaves tracks.
- Logged system errors. Sometimes they are stupid, sometimes – benign. However, often they mean that “stuff” is about to hit the fan. Periodic review of logs reveals them and saves the day.
- Network slowed to a crawl? Applications are slooow? Server is not … well, serving? Where is the answer? In the logs, but you need to read them and understand them.
- That policy you wrote a few months ago. Anybody following that? Anybody remembers that? Halloooo! Check the logs and you’d know.
- You know your auditor might check your logs. But did you know they might also check whether you looked at them? Did’ya? Review the logs and leave the record of this activity!
- Change can be good. But then again, it may be the sign that your controls are lacking. Who changes what and when? From what and to what? Just review the logs.
- Now, you hate looking at logs. You have too many! In this case, look at a specific subset of logs that you never saw before- NBS. Or just deploy log management that can do it for you.
- Logs can help you predict the future (if you review, know and love them ). Don’t believe it? If you read them for long enough, you develop an ability to – gasp!- predict the future, albeit mostly future problems
Posted by anton on under Log Management, Logging |
I’ve been wanting to create those for a loooooong time and finally – here they are (you can guess I’ve been on a long flight ). Some are admittedly tongue-in-cheek, but useful nonetheless. So, enjoy Anton’s ”Top 11 Reasons to Collect and Preserve Computer Logs”, presented in no particular order:
- Before anything else, do you deal with credit cards? Patient info? Are you a government org under FISMA? A financial org? You have to keep’em – stop reading further.
- What if there is a law or a regulation that requires you to retain logs – and you don’t know about it yet? Does the world “compliance” ring a bell?
- An auditor comes and asks for logs. Do you want to respond “Eh, what do you mean?”?
- A system starts crashing and keeps doing so. Where is the answer? Oops, it was in the logs – you just didn’t retain them …
- Somebody posts a piece of your future quarterly report online. Did John Smith did it? How? If not him, who did? Let’s see who touched this document, got logs?
- A malware is rampant on your network. Where it came from? Who spreads it? Just check the logs - but only if you have them saved.
- Your boss comes and says ‘I emailed you this and you ignored it!!’ – ‘No, you didn’t!!!’ Who is right? Only email logs can tell!
- Network is slow; somebody is hogging the bandwidth. Let’s catch the bastard! Is your firewall logging? Keep the info at least until you can investigate.
- Somebody added a table to your database. Maybe he did something else too – no change control forms were filed. Got database log management? How else would you know?
- Disk space is cheap; tape is cheaper still. Save a log! Got SAN or NAS? Save a few of them!
- If you plan to throw away a log record, think – are you 100% sure you won’t need it, ever? Exactly! Keep it.
Have more? Feel free to suggest your own reasons below!
Posted by Joe on March 26, 2010 under Uncategorized |
Welcome to the SIEM Users Website and Mailing List. This site is dedicated to discussion and idea sharing around Security Information Event Management (SIEM) products and their applications. Please sign up for our Mailing List by clicking the link in the upper right hand cover of the site.
