Logs:
Data Warehouse Style
How the LogLogic
®Data Warehouse
™Can Streamline
Logs: Data Warehouse Style
How the LogLogic
®Data Warehouse
™Can Streamline
Your Log Management Needs, Now and for the Future
Once a revolutionary concept, data warehouses now are the status quo, enabling business users and IT professionals to collect, store, manage and report on data originating from diverse sources throughout the enterprise, all from one central location, thereby saving time, resources and money. But what about log data — data accounting for 25% of all enterprise data?
Historically log data has been collected, analyzed and reported on manually or through slow, cum-bersome legacy tools or applications, placing a huge burden on systems, administrators and users alike. The staggering volume of data generated per second from firewalls, routers, servers, applica-tions and more presents organizaapplica-tions with significant problems in managing, analyzing and storing this information, and creates compliance, security and operational concerns.
The LogLogic Log Management Data Warehouse changes all of this, providing secure, distrib-uted, efficient storage and analysis of your log data. Based on the same ideological framework as traditional data warehouses, the LogLogic Log Management Data Warehouse enables organizations to centralize log data for easy, efficient access — ensuring IT security, regulatory compliance, and operational efficiency.
The LogLogic Log Management Data Warehouse enables organizations to centralize log data for easy, efficient access — ensuring IT security, regulatory compliance, and operational efficiency.
So — What is a Data Warehouse? And Why is it Important?
In essence, a data warehouse represents the memory of a company, providing a centralized, histori-cal view of an organization’s data. Through the collection, retention and analysis of operational and informational data, companies can easily access and act on this data to address security concerns, compliance issues, and general systems management.
Defined by Bill Inmon, the “father of data warehousing,” a data warehouse is*:
• Subject-oriented: Data is organized by subject within the database
• Time-variant: Changes to data are tracked, recorded and stored, enabling analysis
and reporting that illustrates changes over time
• Non-volatile: Data in the database is never over-written or deleted, providing
immutable, read-only data for future auditing and reporting
• Integrated: Data collected from a variety of sources throughout an organization is
given a cohesive structure
*Source: Wikipedia: http://en.wikipedia.org/wiki/Data_warehouse
Typically, a data warehouse environment consists of the following elements:
• Database optimized for querying and reporting
• Extraction, transportation, transformation, and loading (ETL) solution to collect
information from throughout the organization, transform it according to pre-defined business rules, and upload it to the data warehouse
• Online analytical processing (OLAP) engine to perform complex analysis and ad hoc
queries on the data
• Other custom Business Intelligence (BI) applications
Through the use of data warehousing, organizations can access and report on data with unpar-alleled speed and efficiency. Rather than manually searching through data across multiple data-bases on multiple machines, data is centralized for access by anyone who needs the information, and allows personnel to view data from multiple sources simultaneously. User errors are diminished because all the data is being viewed and reported on from a single cohesive vantage point. Additionally, operational systems are relieved of excessive processing burdens for querying and reporting, which when dealing with large volumes of data can significantly slow systems down. A data warehouse is a persistent repository
of historical, time-variant data, created for decision support purposes and optimized for reporting and querying.
LogLogic — A Data Warehouse for Logs
As illustrated above:
A data warehouse is a persistent repository of historical, time-variant data, created for decision support purposes and optimized for reporting and querying.
When comparing this definition against logs, there are immediate and obvious parallels:
• Historical, timed data — logs are time-stamped data
• Persistent storage — logs are never modified
• Storage for decision support — logs are used for security, compliance and operational
decisions
• Optimized for reporting — using a log data warehouse approach, you no longer have to
wait hours for log reports, as you would using legacy tools
The LogLogic Log Data Warehouse centralizes the collection, storage and access of log data across the enterprise, freeing organizations from a device-by-device approach to log management. Providing a central repository for all log data, LogLogic enables you to easily query and report on this data with unparalleled speed and efficiently manage the massive amounts of log data generated through network devices, security gear, operating systems, network servers, databases, and more. Additionally, the LogLogic LDW goes beyond simple storage, allowing you to discover and act on relationships between data from these heterogeneous data sources.
LogLogic eliminates log silos in the enterprise with a single, distributed, efficient platform.
Not Just Storage — Storage You Can Access and Use
LogLogic provides a platform that allows you to do more than just store log data, enabling you to store it in a way that provides easy access to the data and the ability to use the data to spot trends over time.
Immutable storage: Unlike data in operational systems that is written over or changed over time, log data stored in an LDW is never changed — by design. This provides a unique historical view of your organization’s log data, a useful tool for compliance, surely, but also gives IT a view into how systems run over time.
Centralized control of log data retention: Centrally determine how long to store logs to ensure effort-less compliance and to avoid errors. Using traditional log tools across multiple systems, it would be difficult if not impossible to ensure that all logs were either saved for the appropriate amount of time, or deleted when necessary to adhere to privacy standards. Using a centralized LDW ensures these decisions are made once for all data sources.
Correlated data store: Because data is structured within the data warehouse, you can easily see relationships between data sources that would be difficult to see across multiple systems and machines in an operational environment, making your data more actionable.
An Essential Part of the Log Management Lifecycle
Collect. Alert. Report. Store. Search. Share. This is the log management lifecycle.
The log management lifecycle is built on central business policies and IT controls, using log data to gain insights into the business. A log data warehouse is at the heart of this lifecycle, allowing you to centralize, aggregate and store log data from across your enterprise and use it more effectively later. Because the task of collecting the volumes of generated data is not trivial, organizations need a powerful tool to accomplish this. LogLogic trumps legacy systems in its ability to easily and efficiently collect this data and make it available for a myriad of uses by constituents throughout the organization.
COLLECT: LogLogic allows you to easily centralize and aggregate data across the enterprise from devices including databases, servers, homegrown applications and more in a log data warehouse.
ALERT:LogLogic delivers the industry’s first smart behavioral alerts, which can be set by device, device group or network. LogLogic delivers adaptive baseline alerts, network policy alerts, ratio-based alerts — all powered by artificial intelligence and machine learning technology.
LogLogic provides centralized, easy access to your log data, enabling you to spot trends over time, and making your data more actionable.
REPORT: Report on your log data easily and quickly. Easy to use templates make this simple, and because the data is centralized and optimized for reporting, your reports will be done quickly, without slowing down operational systems.
STORE:LogLogic’s flexible storage options for log data give you unparalleled possibilities. Providing up to 34 terabytes of compressed storage, LogLogic delivers a highly secure platform for storing vast amounts of raw log data with plug-and-play deployment and a maintenance-free database and operating system. Need more storage? Simply add another appliance. Additionally, LogLogic con-nects to external SAN and NAS storage networks and supports WORM drives with certified integra-tion for NetApp Snaplock, EMC Centera, and Nexsan Assureon.
SEARCH: LogLogic supports keyword and regular expression search functionality, making sifting through the vast amount of data fast and painless. And because the log data warehouse is opti-mized for queries, you’ll get results fast — in just seconds even when searching through terabytes of data.
SHARE: Share log data, reports and alerts easily with other applications and services. Deliver insight to coworkers and partners with complete chain of custody over data.
Once log data is collected and stored, it can be used to better understand user access and activity across the enterprise. The centralized access and storage provided by the LogLogic Log Data Ware-house makes your log data truly actionable. Because all of the log data is at your fingertips, real-time alerts, ad-hoc queries and reports, complex searches, or sharing information becomes simple.
Log Management Lifecycle
A log data warehouse is at the heart of the log management lifecycle — enabling the centralized collection and storage of all log data.
Why Warehouse Log Data?
Through innovation and lowered hardware costs, log data is being generated by more and more devices, from routers and firewalls to operating systems and databases, resulting in the generation of up to millions of records per second that an organization must contend with. Extracting log data from disparate organizational sources poses huge challenges to IT and business users alike, impact-ing the ability to quickly and reliably detect and assess risk, ensure regulatory compliance across the enterprise, and maintain smooth IT operations.
Additionally, in this era of increased attention to risk mitigation and regulatory compliance, log data management has become an essential part of the equation, as compliance initiatives, including PCI DSS, HIPAA, FISMA, etc., demand specific methods of storing and reporting on log data.
By applying a data warehousing methodology to log data, organizations are given fast access to the massive amounts of log data generated by their enterprise. Because log data is centralized in one cohesive source, queries and reports can be completed quickly and efficiently, without affecting the performance of operational systems. This speed of access is simply not possible using legacy tools. Additionally, because the system is subject-oriented, trends and relationships between diverse data sets can easily be observed. Users can view data across the entire IT infrastructure and glean answers about security, compliance and general IT operations. Information has already been collected and aggregated, meaning that users don’t have to spend time organizing the data, but can focus on interpreting data and using the results to build domain-specific applications that effectively use the data, or proactively prevent security breaches or systems problems before they occur. This centralized approach also means that searching across data sources becomes simple. Powerful search functions, by keyword or regular expression, can access results quickly from data coming from multiple systems across the enterprise. The same searches without a log data ware-house would take much more time and effort, the user manually searching each source, one by one.
A log data warehouse provides an access and activity record of your enterprise, enabling you to easily see who has accessed what systems, when they did so, and what they did. You can easily monitor users across your network and prevent unauthorized access.
Whereas operational data is often over-written or deleted over time as it becomes unnecessary from a day-to-day business perspective, log data stored in a log data warehouse is immutable, never over-written or deleted until specified. This provides an invaluable historical view of a company’s systems. Whether for security purposes, compliance audits, or general systems health, seeing infrastructure activity over time helps prevent breaches and strengthen systems over time. Also, as the preserva-tion or destrucpreserva-tion of log data is centralized, there are no forgotten silos of log data, and instances of error are reduced. Data is kept as long as needed for audit and evidentiary needs, and then deleted to ensure privacy.
By using a LDW your organization gains:
• FAST access to LOTS of data
• Single decision support tool for security and compliance issues at all levels • Immutable repository of data for compliance and evidentiary purposes
• Foundation for building domain-specific log analysis tools for various organizational
units (networking/security/servers/databases/desktops/etc.)
• Centralized control over all log data retention and destruction
“But I can access and view all my log data now, why do I need a log data warehouse?”
The answer is that a log data warehouse gives you fast, intelligent access to your data, not just logs and a crude way to get to them. Rather than burdening multiple operational systems with queries, or digging through archived DVDs or tapes, all of your historical log data is in one place, and is organized and optimized for querying, reporting and in-depth analysis.A log data warehouse provides enhanced user access to key log data, freeing users to more effec-tively use the information collected from routers, firewalls, databases, etc., for security, analysis and decision support, rather than spending time organizing the data.
TYPES OF LOGS GOING INTO A LOG
DATA WAREHOUSE
Following the natural flow of log manage-ment, these are the typical types of log data going into log data warehouses:
Network devices: routers, switches, etc. Security gear: firewalls, IDS, IPS,
anti-malware, etc.
Operating systems: Unix, Linux, Windows Network servers: email, DNS, web, FTP,
VOIP, etc.
Databases: Oracle, MS SQL, IBM DB2,
others
Applications: Everything else in the
organization!
Case Study:
Payment Card Industry Data Security Standard Compliance
LDW as a Platform for Log Analysis, Audit and Compliance Applications
In recent years major credit card companies have taken a hard-line stance on protecting consumer credit card data and usage and preventing fraud. To this end they established the Payment Card Industry (PCI) Data Security Standard (DSS), which outlines specific security and compliance measures for all companies that process credit card payments. PCI requirement 10 stipulates that organizations must be able to track and monitor all access to network resources and cardholder data. Log data warehousing provides a platform that allows companies to easily automate compliance with this PCI requirement. Because information gathered from across systems is centralized and immutable, log analysis, audit functions and regulatory compliance become simple and automatic.
In preparation for a PCI DSS audit, a major retail chain chose to implement a LogLogic log data warehouse solution at the suggestion of their auditor, who believed it would help the organization satisfy PCI requirement 10 by giving them a centralized view of their systems over time.
Working from the outside in, they began with the DMZ firewalls, moving on to payment processing applications, process server logs, and select internal firewalls, incorporating data from all Internet DMZ firewalls and intrusion prevention systems. In later phases of the project routers and network gear, servers and databases involved in payment processing were added to the project. Using a phased approach to the project based on risk assessment and complexity of log collection, they ended with centralizing access to log data from legacy operating systems and retail-specific applications throughout the enterprise.
LogLogic provides the only scalable, extensive and powerful log management solution that remains easy to deploy, manage and run, making it the obvious technology choice for this challenging retail environment.
The project was successful in more ways than anticipated. Not only did the company pass the PCI audit, but the IT group discovered that the LDW solution helped them to easily address a number of other compliance mandates, such as the Sarbanes-Oxley Act. Additionally, it helped to strengthen their operational troubleshooting capabilities and improved overall IT efficiency.
LogLogic’s Log Data Warehouse solution helped this retail chain to easily pass its PCI audit and other compliance mandates, while improving overall IT efficiency.
THE LOGLOGIC SOLUTION PROvIDED:
Control over the alignment of log data collection, reporting, and alerting Visibility into undesired access to confidential records
Rapid remediation of threats
Alerts to malicious content that could alter, damage or contribute to the theft of sensitive information
Awareness of rate-based attacks that could reduce or impede the availability of critical resources and information Proper auditing, monitoring, logging, and reporting of security events for rapid iden-tification and response to a material event Forensic analysis of suspicious or material events
Detailed archiving of network logs in a legally acceptable and easily managed form
The Future of Log Data Warehousing
Implementing a log data warehouse will not only allow organizations to better deal with log data today, but will prepare them for the future. As bandwidth continues to increase, and the cost of hardware comes down, more and more log data will be generated. As organizations adopt this meth-odology, and as compliance requirements become more stringent, it is likely that users will want to look at logs not currently deemed important, such as application logs.
The flexibility and scalability of a log data warehousing solution, and specifically that of the LogLogic Log Data Warehouse, will ensure that your enterprise can swiftly integrate new types of log data and accurately access and respond to the information found in your logs.
Conclusion
Log data is becoming more and more abundant, and more and more relevant to your enterprise. From operational issues to risk mitigation and compliance mandates, you need quick, easy and effective access to your log data. Implementing a log data warehouse solution, such as LogLogic, provides just that. By centralizing log data in an environment optimized for querying and reporting, business and IT users alike will be able to view, analyze and report on the data with unparalleled speed and efficiency. You’ll also future-proof your enterprise, ensuring that regardless of which system or application logs you need to access in the future, you’ll be able to integrate them into your LDW solution. Mitigate risk, ensure compliance, and share this information all from one source. The future is clear. Increased log volume, greater log diversity and ever more logs from esoteric and custom sources will be a reality, while regulations mandating the collection, storage and analysis of these logs will simultaneously increase. By using a log data warehouse you’ll be in control of your logs, and tame the future.
Future-proof your enterprise by centralizing log data in an environment optimized for querying and reporting. A log data warehouse will put you in control of your logs, and the future.
About LogLogic
LogLogic, the market visionary and leader, provides the world’s leading enterprise-class platforms for high-performance aggregation, retention and analysis on 100% of log data from virtually any device, operating system or application. LogLogic series 3 LX and ST appliances address the compliance and risk mitigation needs of the most demanding enterprises. LogLogic’s ST appliances for high-performance log data capture and storage were named winner of a Best of Interop Award at NetWorld+Interop 2005 Las Vegas. LogLogic is backed by the world’s leading venture capital firms and serves Fortune and Times 1000 companies globally.
