Log Management Best Practices: The Benefits of Automated Log Management

(1)

Alert Logic, Inc.

1776 Yorktown, 7th Floor, Houston, TX 77056 • 877.484.8383 • alertlogic.com

Alert Logic and the Alert Logic logo are trademarks, registered trademarks, or service marks of Alert Logic Inc. All other trademarks listed in this document are the property of their respective owners. Documents are the property of their respective owners.

Log Management Best Practices:

The Benefits of Automated Log Management

To comply with today’s government and industry mandates, such as PCI, Sarbanes-Oxley, HIPAA and GLBA, log data must be collected, regularly reviewed and archived. In addition, regular analysis and forensics can also be performed on the same log data to enhance overall security and availability.

This paper discusses the challenges associated with effective log management and enables you to better define best practices and requirements for log management projects, as well as log management and review solutions.

THIS DOCUMENT AND THE SOFTWARE DESCRIBED IN THIS DOCUMENT ARE FURNISHED UNDER AND ARE SUBJECT TO THE TERMS OF A LICENSE AGREEMENT OR A NON-‐DISCLOSURE AGREEMENT. EXCEPT AS EXPRESSLY SET FORTH IN SUCH LICENSE AGREEMENT OR NON-‐DISCLOSURE AGREEMENT, ALERT LOGIC, INC. PROVIDES THIS DOCUMENT AND THE SOFTWARE DESCRIBED IN THIS DOCUMENT "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. SOME STATES DO NOT ALLOW DISCLAIMERS OF EXPRESS OR IMPLIED WARRANTIES IN CERTAIN TRANSACTIONS; THEREFORE, THIS STATEMENT MAY NOT APPLY TO YOU.

This document and the software described in this document may not be lent, sold, or given away without the prior written permission of Alert Logic, Inc., except as otherwise permitted by law. Except as expressly set forth in such license agreement or non-‐disclosure agreement, no part of this document or the software described in this document may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, or otherwise, without the prior written consent of Alert Logic, Inc. Some companies, names, and data in this document are used for illustration purposes and may not represent real companies, individuals, or data. This document could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein. These changes may be incorporated in new editions of this document. Changes or improvements may be made to the software described in this document at any time.

U.S. Government Restricted Rights: If the software and documentation are being acquired by or on behalf of the U.S. Government or by a U.S. Government prime contractor or subcontractor (at any tier), in accordance with 48 C.F.R. 227.7202-‐4 (for Department of Defense (DOD) acquisitions) and 48 C.F.R. 2.101 and 12.212 (for non-‐DOD acquisitions), the government's rights in the software and documentation, including its rights to use, modify, reproduce, release, perform, display or disclose the software or documentation, will be subject in all respects to the commercial license rights and restrictions provided in the license agreement.

Alert Logic is a trademark or registered trademark of Alert Logic, Inc. or its subsidiaries in the United States and other jurisdictions. All other company and product names mentioned are used only for identification purposes and may be trademarks or registered trademarks of their respective companies.

(3)

Why Log Management?

Today most organizations have tighter budgets and fewer resources than ever, yet they are experiencing ever-‐increasing pressures to improve security, comply with regulations, and continuously improve availability.

Governmental and industry regulations have become better defined in recent years with significant fines or even incarceration facing senior executives who fail to comply. With decreasing staff, IT organizations are now being forced to commit resources toward compliance initiatives while also continuing to ensure security and meet service level agreements.

In the past, a network administrator or security analyst would collect log data from a few select systems in the event that the data might be needed for a specific search later. Today, log management is an organizational requirement, demanding comprehensive functionality that extends beyond data collection to encompass normalization, analysis, reporting, and disaster-‐proof archival.

The number, variety, and volume of log data and network infrastructures have created a massive challenge. In addition, the expansion of IT infrastructure into hosted and cloud deployments means that there is not only more data to manage, but that it resides in a variety of environments. Trying to collect and manage a continuous supply of distributed log data can quickly overwhelm at IT organization; adding storage sounds simple in concept, yet the costs of purchasing and managing terabytes of storage can be staggering.

With all of these challenges in mind, this paper will discuss best practices for log management in the current environment. Best practices for log management center on several key areas:

• Collecting the appropriate data. Consider all the sources of log data in your environment and which are required to meet compliance mandates, alert you to suspicious activity, and provide valuable forensic data.

• Making log data usable in a normalized, searchable format.

• Reviewing and analyzing log data regularly. Log data will not help you achieve your goals if it is not examined regularly; for compliance purposes, this is a requirement.

• Ensuring secure transmission and storage of log data. Log data is as sensitive as any of your other enterprise data and the same care you exercise with other types of data should be exercised with your log data.

• Archiving data according to relevant data retention policies, including provisions for the appropriate level of data protection – for example, off-‐site storage.

(4)

Collecting Logs for Best Practice Reports

With a multitude of systems generating log data within a typical business environment, many organizations struggle to determine which log sources should be collected. This challenge should be viewed from the perspective of which logs would translate to the most immediate value. When an organization is unsure how to attribute value, it is best to reference what the industry would determine as best practice reports associated with log data.

The following list outlines a list of best practice reports that should be available in a log management solution.

Active Directory

Active Directory Global Catalog Change – The Microsoft Active Directory Global Catalog provides

searchable information about every object controlled within your AD forest. Additionally, it provides the ability to search across multiple different domains without being required to access the AD for each domain directly. This report should identify log messages that indicate all changes to the AD Global Catalog.

Active Directory Global Catalog Demotion – The Microsoft Active Directory Global Catalog provides

searchable information about every object controlled within your AD forest. Additionally, it provides the ability to search across multiple different domains without being required to access the AD for each domain directly. This report should identify log messages that indicate each time a domain controller in your AD forest has been demoted and can no longer serve the global catalog.

Databases

Database Failed Logins– This report should identify log messages that indicate database login failure log

messages received from all monitored hosts.

Network Devices

Network Device Failed Logins– This report should identify log messages that indicate network device

login failure log messages received from all monitored hosts.

Network Device Policy Change – This report should identify log messages that indicate when a policy is

added/changed/removed on network devices.

Windows Server (2008 R2, 2008, 2003)

Excessive Windows Account Lockouts– The messages indicate that Windows user accounts have been

locked out. This report should identify log messages that indicate when a threshold of 2 log messages has been exceeded.

(5)

Excessive Windows Account Lockouts by Administrative User – The messages indicate that the Windows

Administrator account has been locked out. This report should identify log messages that indicate when a threshold of 2 log messages has been exceeded.

Excessive Windows Failed Logins– This report should identify log messages that indicate excessive

Windows login failure log messages received from all monitored hosts with a threshold greater than 5 messages.

Excessive Windows Failed Logins by Administrative User– This report should identify log messages that

indicate when an excessive amount of Windows login failure log messages are received from a single host for the Administrator account. The threshold is messages greater than 5.

Windows FTP Failed Logins – This report should identify log messages that indicate when accounts have

failed to successfully login to IIS.

Windows User Account Created– This report should identify log messages that indicate when user

accounts have been successfully created.

Windows User Account Modified– This report should identify log messages that indicate when user

accounts have been modified (changed, created and deleted).

Windows User Group Created– This report should identify log messages that indicate that a user

group has been created.

Windows User Group Modified– This report should identify log messages that indicate that user

groups have been modified (changed, created and deleted).

UNIX/Linux

Failed UNIX Switch User Command -‐ This report should identify log messages that indicate all

recorded failed uses of the UNIX switch user (su) command.

UNIX Account Created– This report should identify log messages that indicate the creation of UNIX

accounts.

UNIX Failed Logins) – This report should identify log messages that indicate local and remote accounts

have failed to successfully login.

UNIX Group Created– This report should identify log messages that indicate a UNIX user group

was added.

UNIX SSH Failed Logins– This report should identify log messages that indicate SSH login failure log

messages received from all monitored hosts.

(6)

UNIX Sudo Access– This report should identify log messages that indicate when a user has executed

the UNIX sudo command.

UNIX Switch User Command Success – This report should identify log messages that indicate a user has

(7)

Other Log Sources to Consider

The best practice reports described above provide the most immediate value to most organizations. However, there are other log sources that should be considered for collection for other operational goals, such as optimization health checks. In addition, some compliance and regulatory standards may require that additional log data be collected.

For example, operating system logs and application logs often contain security-‐related information as well as information about events that may not initially appear security-‐related. Organizations must consider the potential value of each and every potential log source.

In addition, log collection must be enabled in a growing variety of types of environments. In the past, log data typically resided in an in-‐house environment, or in traditional managed hosting deployments. As more infrastructure moves into the cloud, log collection projects must contend with data from virtual servers, elastic cloud environments with instances that are launched for days or hours, and hybrid environments. Along with the tremendous flexibility and efficiency that these deployment options bring come new challenges for IT managers.

The following log types should also be considered for collection:

Anti-‐Malware Software

Examples of anti-‐malware include anti-‐virus, anti-‐spyware, and rootkit detectors, to name just a few. These logs may include information indicating that malware was detected, disinfection attempt results, file quarantines, when file-‐system scans were last performed, when anti-‐virus signature files were last updated, and when software upgrades have taken place.

Applications

Organizations typically utilize a wide variety of applications to support business processes, including supply chain management, financial management, procurement, resource planning, customer relationship management, email and voice communications, web and ecommerce applications, and file and document management systems. Some of these applications are purchased from vendors and others are developed and maintained internally.

The information logged by various applications can vary wildly and may include account changes, user authentication attempts, use of privileges, usage details, client and server activity, configuration changes, major system failures, etc. Application logs can be more valuable when network communications are encrypted. However, application logs are often proprietary formats.

Authentication Servers

Directory servers and single sign-‐on servers will typically log each and every authentication attempt showing the originating user ID, destination system or application, date and time info, and success/failure details.

(8)

Firewalls

Some firewalls are perimeter-‐focused and general in nature and others are very application-‐specific or single-‐host (personal) focused. Firewalls not only block activity based on policy, they can inspect content and ensure the state and integrity of permitted connections. As such, their logs can be very detailed and informative.

Intrusion Detection and Protection Systems

These systems record detailed information about suspicious behavior and detected attacks as well as actions taken to halt malicious activity in progress. Some intrusion protection systems, such as file integrity systems, run periodically instead of continuously and thus they generate logs in batches rather than on an ongoing basis.

Network Access Control Servers

Network access control can operate for both internal and external hosts connecting to the internal network. At the time of connect, the hosts’ security posture is determined and hosts failing to adhere to the defined policy are quarantined onto a separate VLAN (Virtual Local Area Network) segment. NAC servers log a great deal of useful information about both successful/permitted and unsuccessful quarantined network connections.

Network Devices (Routers, Switches, etc.)

Routers can be configured to block certain types of traffic. Network devices can be configured to log very detailed connection activity but typically are configured to log very lightly. These logs can contain very informative network communication activity.

Operating Systems

There are many varied operating systems on servers, workstations, and assorted network devices. Logging is typically controlled by the host administrator. The types of events, as well as whether to log only successful or only failed events, or both, can be controlled.

These log entries typically contain information about service starts and stops, authentication attempts, file accesses, security policy changes, account changes, permission and privilege changes, and use of privileges. Operating System logs can also contain information from security software and system applications and are often beneficial for identifying suspicious activity involving a particular host.

Virtual Private Networks

Virtual Private Networks (VPNs) are the most popular type of secured remote access solutions and they log both successful and failed connection attempts. They record details such as the date and time each user connects and disconnects, as well as the types and amount of data sent and received during the connected session.

(9)

Vulnerability Management Software

Included here are both vulnerability scanning and patch management software. These typically run on an occasional basis and log batches of log entries that include information about scanned hosts/devices including: configuration, missing software updates, vulnerabilities identified, and patch/scan currency downloads, among other things.

Web Proxies

Web proxies are the intermediate hosts through which Web sites are accessed and can be used to restrict Web access as well as add a layer of protection between the user and external Web sites. Web proxy logs record user activity and URLs accessed by specified users.

(10)

Log Management Challenges

Recent compliance mandates require not only that you collect all logs, but also that they be reviewed regularly, searchable, and stored in their original, unaltered, raw form for mandate-‐specific timeframes. Logs can also be extremely useful in identifying security incidents, policy violations, fraudulent activity, and operational problems shortly after they occur. They are also valuable when performing audits, forensic analysis, internal investigations, establishing baselines, and identifying operational trends and long-‐term problems. However, the infinite variety of log data formats makes it impossible to utilize the data without data normalization.

It is reasonable to assume that the variety of log data sources and the volume of data will always increase. Compounding this challenge is the variability of data formats and distributed nature of these sources; in addition, every network infrastructure is in a constant state of change, with new systems, applications, users, and devices every day of the year.

This creates a variety of specific challenges for log management efforts. These challenges can be broken down into three areas: collection, analysis and review, and archival.

Collection

When we discuss log data, we are discussing a wide range and ever-‐changing range of data sets that must be accounted for.

• Log data is varied. Not only do systems, applications, and network devices have their own logs with varying types of specific data which are captured, but a single log source can have multiple logs to be captured. For example, applications often have multiple log files, each containing a specific type of data.

• Log data sources are distributed. Data sources may be located within internal on-‐premise infrastructure, collocated in a data center, hosted with a managed hosting provider, or in the cloud. This infrastructure may be managed separately or in a hybrid environment. Log collection must span all of these environments.

• Log data sources change constantly. At any time a new system, application, or network device may be brought online and begin generating new log data. Cloud instances may be launched for days or hours and then terminated. A log management solution must account for these changes, or else 100% log collection will not be possible. Otherwise, an organization risks discovering that a log source has not been collected after weeks or months, possibly in response to an auditor’s questions.

• Log data may contain sensitive information, such as excerpts from emails, user names and passwords. This raises security and privacy concerns that necessitate proper log data security. Logs improperly secured when being transported to any centralized collection system are susceptible to intentional or unintentional alteration or destruction.

(11)

• Log data should be secured. If administrative privileges are not properly maintained and the logs secured, then the logs can be manipulated or altered. It is important to understand and limit such privileges and access to logged data as well.

Analysis and Review

Analysis and review of log data presents two significant challenges: regular review of log data, and the varying formats of log data.

Regular log review is a valuable practice for any organization, and is a requirement of many compliance mandates. Typically, system administrators have been responsible for reviewing and analyzing log data, but this has usually been a lower priority than other activities, such as more strategic business projects. Rapid-‐ response situations, such as performance issues, vulnerability remediation, and security incident response and investigation, also tend to take priority over log review. The result? Log management projects are never started or linger unfinished.

Log contents vary enormously. Some logs are designed for humans to read and others simply are not; some logs use standard formats, while others use proprietary formats. Some log formats are comma separated, some are space delimited, and still others use symbols or other character delimiters between the fields within a single log message.

Each log entry, or message, contains certain defined pieces of information, such as a host IP address or username. Each log source records the pieces of information that it considers important. Consequently, it can be extremely difficult to link different log sources because they may or may not contain common values.

Even when two sources record the same values, they may be recorded in different and varied log messages. Additionally, they may represent them differently. For example, a date may be formatted MMDDYYYY, MM-‐DD-‐YYYY, or DD/MM/YYYY.

Deciphering dates in various formats may be simple for a human reviewer, but consider the example of the use of FTP (File Transfer Protocol) being recorded by one log source as “FTP” and another as “21,” its well-‐ known port number. Very few analysts can easily distinguish the full 1,024 well-‐known ports by port number.

One approach to dealing with this complexity is to create PERL scripts to search and produce only those log messages matching the query. In concept this is a reasonable approach, but with the growing complexity and variety of sources, its ability to alleviate the problems of manual log review is limited.

Archival

Log data must be treated like any other organizational data, subject to security and retention policies, as well as compliance mandates. Because it often contains sensitive data (such as customer data), breach of log data is a serious problem. As a result, protection of log data both in transit to the log collection solution and when stored is an important concern.

(12)

This means that access to log data much be strictly controlled, and under no circumstances should log data be alterable.

In addition, storing log data centrally from distributed sources across an organization creates a massive storage management challenge. Purchasing and deploying the required storage consumes valuable real estate and power (both for operations and cooling) and must be managed, backed up, and included in disaster-‐recovery planning.

(13)

Automated Log Management

As the challenges of log management have grown, so have the benefits of automated log management solutions. An appropriate log management solution provides many benefits to an organization:

• Log collection across all IT infrastructure – on premise, hosted, and in the cloud

• Sophisticated parsing of logs to enable analysis of data from a widely-‐varying set of log sources • Reporting tools that provide insight into your organization’s security posture

• Tools to enable post-‐incident analysis of log data

• Reliable, regular review of log data that meets compliance mandates as well as security best practices.

The cost of log management tools and services must be weighed against the internal staff time required to attempt log management, as well as the cost of non-‐compliance, data loss, and security incidents.

Log management solutions should be evaluated against the practices described in this paper:

• Does the solution provide complete log collection across all sources, and in all environments? • Is log data parsed and normalized to support the required search and analysis functions? • Is regular log review provided that meets internal requirements and compliance mandates? • Is data transmitted and stored securely?

• Can data be archived according to organizational retention policies, with appropriate levels of data protection?

(14)

Summary

While compliance initiatives often drive the need for log management, there are a myriad of security and availability related benefits as well. As for compliance, there are many governing regulations and standards, most-‐notably PCI, Sarbanes-‐Oxley, HIPAA, GLBA, and FISMA, which require log collection, retention and access for forensic analysis. Each of these has varying levels of “key controls” that dictate the collection, analysis and secure archival of log data in sufficient detail for appropriate time periods.

Some of the other benefits achieved through routine log analysis are improved detection of security incidents, policy violations, fraudulent activities, and operational problems. Logs are also useful for establishing performance baselines, performing auditing and forensic analysis, supporting internal investigations and identifying operational trends and long-‐term problems.

Whether home-‐grown or purchased, in-‐house log management solutions consistently fall short due to a continuous supply of log data with definite resource limitations. In today’s environment, every organization is faced with the log management challenge, though no one has idle full time employees and hardware resources to apply to the challenge – not to mention unlimited capital budgets.

Even if you were able to collect, consolidate, and archive log data in an automated fashion, the data needs to be protected from malicious and accidental breaches of confidentiality and integrity – not to mention disasters whether they be natural, malicious, or accidental. Compounding this is that interpreting raw log data via views and reports as well as supporting forensic queries is no small undertaking. Hiring and retaining log knowledge experts is not only an impossible task, but having these experts available to efficiently and effectively review log data on a regular basis is simply not feasible.

Considering the breadth of servers, operating systems, databases, applications, and network infrastructure components that produce log data, coupled with the lack of standardized log formats, a vendor managed solution is the best choice for most companies.

(15)

About Alert Logic

Alert Logic, the leading provider of Security-‐as-‐a-‐Service solutions for the cloud, provides solutions to secure the application and infrastructure stack. By integrating advanced security tools with 24×7 Security Operations Center expertise, customers can defend against security threats and address compliance mandates. By leveraging an “as-‐a-‐Service” delivery model, Alert Logic solutions include day-‐to-‐day management of security infrastructure, security experts translating complex data into actionable insight, and flexible deployment options to address customer security needs in any computing environment. Built from the ground up to address the unique challenges of public and private cloud environments, Alert Logic partners with over half of the largest cloud and hosting service providers to provide Security-‐as-‐a-‐ Service solutions for business application deployments for over 1,700 enterprises.

Alert Logic is based in Houston, Texas, and was founded in 2002. For more information, please visit

http://www.alertlogic.com.