In today’s scenario, log monitoring is no more a compliance requirement. It is required for being proactive in detecting real-time threats and to prevent any damage to reputation, service or financial loss. While some of the compliance requirements define what must be logged or specify the log retention period, there is no standard solution or framework for minimum logging. Predominantly the log management solutions are driven by ‘logging everything ‘, available storage or use cases.
While ‘log everything and store everything’ is seen as a better option to manage worst scenarios, we also have to look at the basic purpose of log monitoring which is ‘finding a needle in haystack’.
Organizations trying to stabilize and mature in log collection and analysis could run into any of the following issues.
- An unexpected log volume growth and storage becoming a major bottleneck
- Inability to process large volume of logs or peak events per second, because of inappropriate sizing
- Unused capacity with large capital investments either because of less understanding on the business requirements or lack of expertise
- Store logs which are never used and unnecessarily occupying usable space
- Performance and stability issues due to insufficient log collection strategy e.g. Start collecting everything from ALL devices in one go instead of log collection in multiple phases
- Challenges in identifying real threats since the log size is just too much to handle
- Logs for incident investigation are ‘missing’ when needed. This is like searching for permit events in firewall traffic logs to determine the source IP address of the machine that was used to hack VIP’s mailbox and you only get a shock that you never logged it!!!
With growing trend in log monitoring solutions, log collection and storage in terabytes is easily achievable and big data analysis is already incorporated in most of them. Some Organizations look for other options including outsourcing log monitoring to service providers so that they can bring in more meaningful and efficient solutions to overcome some of the issues stated above.
So where do we start? One of the common and most suitable approach is to align with organization’s risk management plan and gradually mature on log monitoring. Define the scope of assets covered in log monitoring to include business critical areas like public facing network segments, e-commerce applications etc... Customized and relevant use cases used in conjunction with out-of-the-box use cases have proved to be more efficient as compared to simply using what is provided by vendors. Setup a minimum base line for logging and ensure that it is followed for all assets covered in log monitoring scope. Start rolling out log management strategy for one location or department or network segments and further expand. You will also find most organizations starting with log collection for perimeter devices like firewalls, gateways, routers etc. and then implementing for other server farms, applications, and databases.
Log monitoring can be successful only if we know what to log and what log to look for proactive incident identification and analysis. So it is not the question of how many logs do we have, it is all about how much meaningful logs we have!!!
Authored By - Rathnamala Rajaram
TCS Enterprise Security and Risk Management