3
State of Affairs
The number of applications and devices on corporate networks is growing at a rapid rate. Each of these sources generate a stream of log data (also known as machine data) which is a hot commodity for different business functions—from security to compliance to IT operations to analytics. The applications that use machine data typically pull the information directly from the log sources with few or no limits on the log data they copy and store. Each of the uses has different requirements for which log data should be retained and for how long.
Because there is no centralized management of the machine data generated by these log sources, basic questions like which systems get what machine data are difficult, if not impossible to answer. For example, credit card logs from a database may be collected by many systems, creating a PCI compliance nightmare. The uncontrolled distribution of log data also means that there are few or no guarantees that at-risk data is kept only for the minimal amount of time possible. The systems that use machine data (such as compliance applications) often are not geared toward long-term archiving, which creates the opposite problem: long-term data retention requirements may not be met.
The Problem
The amount of machine data being generated imposes significant storage costs. The volume also creates challenges in meeting user expectations when searching (often to address critical security issues) and performing other operations. The tools in use may not be designed for long-term retention and may modify data in order to meet performance goals or to cut storage costs, which violate the requirement to have log data in its original format for compliance and forensics. Management is at an increased risk of non-compliance and possible resulting financial penalties, legal repercussions, and loss of business. All the while, administrators struggle with how to archive the data, accommodate growing storage requirements with finite storage resources and budgets, and ensure access to this critical information.
The Solution
What is needed is a solution that can address log storage needs more economically and judiciously. The solution should provide storage of critical data for the mandated time periods and of non-essential data for shorter periods, freeing up valuable disk space for additional critical data. TIBCO LogLogic® Log Management Intelligence addresses this
need with granular data retention.
Granular data retention supplies policies specifically constructed to address these problems. The flexibility of data retention policies assist administrators in leveraging storage and keeping essential data only for the mandated time periods while assigning less critical log sources shorter retention periods. This intelligent storage capability allows more economic use of a costly and finite resource without compromising availability and access to the data.
Login Attempts Financial Data Access Router Configuration Change Print Spoolers High-Speed Index
180 days 365 days 90 days 30 days
Long-Term Archive
3 years 7 years 3 years 1 year
Table: Business Requirements For Data Retention Policies
How it Works
TIBCO LogLogic comes with a pre-defined set of data retention policies for different types of log data. These data retention policies provide the ability to define both how long to keep archives of raw data and the period of time data should be indexed to improve search performance. One of the many standard data retention policies available in LogLogic is calls for archiving raw data for one year and keeping one month’s worth of data indexed. Short data retention policies like this would be assigned to less critical log sources. Long-term data retention policies might be applied to critical log sources like a general ledger system. Data can be archived with TIBCO LogLogic for up to 10 years and is available for searches, reports, and alerts for the full retention period.
Create custom policies
5
Create custom groups
To facilitate the assignment of data retention policies, log data can be grouped.
Administrators can organize subsets of log sources for even easier assignment to specific data retention policies, ensuring intelligent and efficient disk utilization. TIBCO LogLogic data retention policies allow the most fine-grained control of log data. Log sources can be easily rearranged into groups specific to organizational or retention requirements. New groups can be created using a few mouse clicks, allowing devices to be added via pattern matching, dynamic rules, and other methods.
Set data retention priorities
The facility for grouping log data is quite rich and flexible, and sometimes data belongs to more than one group. For example, an administrator might define a data retention policy for CISCO PIX routers (30 days indexed/1 year archived) and a separate policy for “Severe Errors” (90 days indexed/3 years archived). These are not mutually exclusive groupings. When this situation occurs, TIBCO LogLogic uses the highest priority data retention policy. If the priority of the “Severe Errors” data retention policy is higher than the “CISCO PIX” policy, severe errors on CISCO PIX routers will be retained according to the 90-day/3-year policy, and all other data will be retained according to the 30-day/1-year policy.
Step 3: Assign data retention policies to sets of log data
Step 2: Review data retention policies and create custom policies Step 1: Create and organize sets of log data
Firewall Firewall Financial Systems Standard: 90/1 INDEX: 90-days RAW: 1-year Financial
Systems PCI SevereErrors
Standard: 90/1
INDEX: 90-days RAW: 1-year
Custom: Financial Systems
INDEX: 90-days RAW: 10-years
Custom: Financial Systems INDEX: 90-days
RAW: 10-years
Applies to
Summary
Granular data retention policies give administrators the ability to create individual polices for specific data types as required by the log source, operational guidelines, or regulatory requirements. Administrators can assign critical log sources to pre-defined or custom data retention policies quickly and easily, ensuring that log data will be retained for the long term. Granular data retention policies also allow administrators to fine-tune the amount of data kept online to optimize searching, reporting, and alerting. Benefits include the ability to:
• Respond faster
