Chapter 5 Centralized Log Management
This chapter presents the following topics:
Overview ... 60 vCenter Log Insight remote syslog architecture ... 62 Centralized logging integration ... 64 Content packs for vCenter Log Insight ... 66 Configuring alerts ... 69 Summary ... 71
Chapter 5: Centralized Log Management
Overview
Many key solution resources constantly record operational and security-related events to a local log. When a security incident occurs, log files can help you track down the root cause. However, without log file consolidation, those investigative tasks can be laborious. Running a reliable and secure data center is a continual process of planning, delivering, and operating. Without a consolidated view of your infrastructure’s system log data, your data center is incomplete and at risk. The risks include:
Lack of central and holistic visibility into security-related events
Inability to easily correlate events that would indicate a security breach
Log files are overwritten causing you to lose log entries that are critical for security, compliance, and troubleshooting
Increased downtime for applications and servers, because more time is needed to locate and search system log files when trouble occurs
Security risks such as malicious attacks or unauthorized logins that could be occurring without your knowledge
Loss of historical system logs, leaving you unprepared to report local authentications or maintain compliance
Consolidated system logging is a critical data center feature that is commonly left unimplemented because of its complexity. Many IT organizations rely solely on data center monitoring tools, which, while useful, mostly focus on raw metrics—such as CPU utilization, memory consumption, and storage I/O—but completely ignore log files and security events. When system log files are ignored, valuable security information is overlooked.
To address these challenges, the Federation Software-Defined Data Center uses VMware vCenter Log Insight to deliver real-time log management and log analysis with machine-learning-based Intelligent Grouping and a high-performance search, enabling better visibility across the entire Federation Software-Defined Data Center solution.
Chapter 5: Centralized Log Management
Figure 13. Centralized logging of software-defined data center components with vCenter Log Insight
Log Insight is tightly integrated with vCenter Server and vSphere ESXi and comes with built-in knowledge and native support for vCenter Operations Manager. Alerts are configured to notify security administrators by email or through the vC Ops
dashboards. vCenter Log Insight works in multiple ways to ensure greater visibility into your cloud operations and achieve the security compliance that your company requires.
Log Insight can analyze log events from the entire Federation Software-Defined Data Center by configuring each solution component to forward logs to Log Insight. Some of this configuration is achieved using Log Insight’s native capabilities, while the remainder is done by manually configuring syslog operations.
Chapter 5: Centralized Log Management
Figure 14. Searching for security events with vCenter Log Insight
Log Insight allows you to search for security events across all consolidated data, as shown in Figure 14. For example, to search for failed logins across the infrastructure, you can search across all the components that make up the Federation Software- Defined Data Center. Log Insight provides a powerful security tool that consolidates and analyzes logs and enables high-speed interactive queries. In addition, you can create your own custom queries to save and create your custom security dashboard.
vCenter Log Insight remote syslog architecture
Remote syslog is used for various reasons, including: Aggregation
Querying
Correlation
Retention
Chapter 5: Centralized Log Management Log Insight instances, as shown in Figure 15. This is referred to as client-server architecture.
Figure 15. Log Insight client-server architecture
This client-server architecture is suited to environments, which:
Are greenfield, with no syslog operations to date
Use automation or configuration management
Have fewer than 750 devices sending remote syslog data
For larger instances of this software-defined data center solution, a distributed Log Insight deployment with a master node and up to five worker nodes can be deployed in a cluster configuration. To ensure high availability in such a configuration, you must deploy the cluster in an N+1 configuration and use a load balancer in front of the cluster to load-balance connections and handle node failures. With this configuration, if any node goes down, the load balancer can redirect traffic to the remaining nodes. Note that the Web UI access is limited to the master node, as shown in Figure 16.
Note: A worker node stores forwarded syslog events and processes queries against log data it stores on behalf of the master node.
Chapter 5: Centralized Log Management
Figure 16. Log Insight distributed architecture
For more information on remote syslog architecture for vCenter Log Insight, refer to
VMware vCenter Log Insight: Getting Started Guide.
Sizing information for VMware vCenter Log Insight for this software-defined data center solution is documented in the Federation Software-Defined Data Center:
Foundation Infrastructure Solution Guide, in the Resource Sizing Guide chapter.
Centralized logging integration
Unlike many syslog implementations that only support UDP, Log Insight supports receiving syslog-formatted events over UDP, TCP, and SSL protocols. In high-volume environments, the inclusion of TCP support provides a significant performance improvement over a UDP-only-based system, because more events can be channeled through fewer connections. This ensures that events are not lost as they would be with UDP-only based log servers. Additionally, the support for receiving syslog events over SSL ensures that the event details are transmitted over the network in a
Chapter 5: Centralized Log Management
Identification of security incidents and policy violations as they occur
Performance of auditing and forensic analysis
Establishment of baselines that can be used to detect future anomalous behavior
When you have collected data, using Log Insight you can perform ad-hoc searches across all the event data. Figure 17 shows an example of a failed login query.
Figure 17. Sample vCenter Log Insight dashboard for vCenter Server
You can save queries you perform often as Favorites and also use them to create charts, dashboard widgets, and alerts. In large environments with numerous log messages, you can use runtime field extraction with Log Insight to instantly locate and extract the most important data fields using regular expressions.
Chapter 5: Centralized Log Management
The following components of the software-defined data center management platform should be configured to forward the application logs to Log Insight:
EMC Avamar
EMC Unisphere for VMAX
EMC ViPR
EMC VNX
VMware vSphere ESXi hosts
VMware IT Business Management
VMware NSX for vSphere
VMware vCloud Automation Center
VMware vCenter Orchestrator
VMware vCenter Server
All physical compute, fabric, and network devices
Content packs for vCenter Log Insight
Analysis of the forwarded events can be enhanced using pre-packaged VMware, EMC, partner, and community-provided content packs, which are available on the VMware Solution Exchange. Currently available content packs that relate to components in the Federation Software-Defined Data Center are:
EMC Avamar content pack
EMC VMAX content pack
EMC VNX content pack
VMware vCAC Log Insight content pack
VMware vCenter Log Insight content pack for vCenter Operations Manager (bundled with Log Insight)
VMware vSphere content pack (bundled with Log Insight)
Additional content packs: Available for Microsoft Windows, Microsoft Active Directory, and other partner solutions
Chapter 5: Centralized Log Management The vSphere content pack provides important operational information about the vSphere environment using several dashboards that contain a comprehensive list of security events and event types such as:
ESX/ESXi connections by source
ESX/ESXi logins by type, source, and user
vCenter Server authentication attempts by type, source, and user
Events, tasks, and alarms
When integrated with Log Insight, EMC content packs for Avamar, VNX, and VMAX provide dashboards and user-defined fields specifically for those EMC products that enable administrators to conduct problem analysis on their VNX and VMAX arrays or backup infrastructure. Many of these content packs include dashboards that include security-related charts and widgets that provide at-a-glance visibility into security- related events, shown in the example in Figure 18.
Figure 18. Customized software-defined data center security dashboard using multiple content packs
Content packs are read-only plug-ins to vCenter Log Insight that provide predefined knowledge about specific types of events, such as log messages. The goal of a content pack is to provide knowledge about a specific set of events in a format easily understandable by security administrators, monitoring teams, and auditors. Each content pack is delivered as a file, and can be imported through the Log Insight web UI. The custom Log Insight dashboard in Figure 19 shows EMC Avamar backup, vCenter and Windows authentication, failures, and ESXi host firewall changes.
Chapter 5: Centralized Log Management
Figure 19. Custom Log Insight dashboard
Dashboards and widgets can be manually created for those components for which content packs do not already exist. Each widget provided by a content pack can be cloned and added to a personalized dashboard that can be shared to contain only the views required by the user. Figure 19 provides an example of this, showing a partial view of the software-defined data center dashboard that contains widgets from the content packs installed for this solution.
The content pack for vCenter Operations Manager presents its log data in a more meaningful way and analyzes all of the logs that are redirected from a vCenter Operations Manager instance.
The vCenter Operation Manager content pack provides:
A collection of logs from all vCenter Operations Manager servers
Default queries to expose key fields and events
Pre-configured dashboards to make troubleshooting quick and easy The content pack provides 6 dashboard groups, 32 dashboards, 24 queries, 11 alerts, and 40 extracted fields. You can use these queries and dashboards to monitor and troubleshoot various issues in the vC Ops environment. The queries and
dashboards can be used to monitor and troubleshoot issues in the vCenter Operations Manager environment.
Chapter 5: Centralized Log Management
Configuring alerts
The Federation Software-Defined Data Center solution uses vC Ops to monitor the cloud management platform, compute resources, and workloads used in production. vCenter Log Insight integration with vCenter Operations Manager enables you to raise alerts for Log Insight queries and send notifications to vC Ops based on a
configurable threshold, as shown in Figure 20.
Figure 20. Example of a Log Insight alert configured to send a notification to vC Ops
You can also configure predefined alerts that are installed when content packs are imported to Log Insight. An example of a number of security-related alerts imported by the Microsoft Active Directory content pack is shown in Figure 21.
Chapter 5: Centralized Log Management
Figure 21. Examples of security alerts installed in Log Insight
In addition, the integration between vCenter Log Insight and vC Ops enables a Launch in context menu in the vC Ops dashboard that can be used to launch a vCenter Log Insight interactive analytics dashboard to display events related to the selected vC Ops object.
The example in Figure 22 uses the integration between Log Insight and vC Ops in which the Actions menu in vC Ops triggers a search of all relevant Log Insight information on the selected item.
Chapter 5: Centralized Log Management The launch-in-context functionality filters the logs using the constraint hostname equals <each hostname>, which displays only events that match the criteria, as highlighted in Figure 23.
Figure 23. vCenter Log Insight filtering logs for the management cluster components
For a more detailed discussion of vC Ops and the role it plays in this solution, refer to
the Federation Software-Defined Data Center: Foundation Solution Guide.
Summary
The integration of vCenter Log Insight in the Federation Software-Defined Data Center solution enables greater visibility into operational and security-related events. We demonstrated how each component can be configured to forward events to Log Insight to provide a single point of visibility into the environment for administrators and configure alerts to notify through email or vC Ops. Where an organization already has a Security Event and Incident Management (SEIM) system in place, Log Insight can act as an aggregator to forward events to the SEIM, providing the security team with a single integration point for the whole solution.