Logging and Alerting
for the Cloud
What you need to know about monitoring and tracking across your enterprise
The need for tracking and monitoring is pervasive throughout many aspects of an organization: development, operations, security, auditing and compliance, to name just a few. Logs contain important information on performance, security and user activities, and provide critical information about what the system is doing. With migration to the cloud, logging and monitoring become even more important since you are giving up some control to the cloud services provider.
This paper provides best practices for logging and auditing to track and report on activities occurring within your enterprise, including your private and public clouds.
The importance of logging
requirements for logging are important to any organization regardless of whether they are articulated:
• Compliance
• Change control
• Auditing
• Improvements to operations and development processes
Compliance
Many organizations require appropriate logging to demonstrate that they are in compliance with regulations. While the exact requirements vary from regulation to regulation, they all require that the organization be able to demonstrate that it has end- to-end proof that it can monitor and audit what is happening across all its systems.
2
Cloud providers
generally do
not provide their
customers with
logs of what actions
users have taken
via the provider’s
console or API.
As a result, the customer, by default, does not have any way of tracking which users performed what actions.
Fortunately, there is a workaround.
Organizations can purchase or build a cloud-aware proxy between themselves and the cloud provider and use that proxy to perform all tasks. The proxy should, of course, have the ability to log who each user is, what actions each user performed and when those actions were taken. Additionally, the proxy should be able to forward the logs to whatever centralized logging system the organization has in place, in a format that is easy for the system to understand and manage.
Change control
Change control is a key aspect of many compliance regulations, most notably Sarbanes-Oxley (SOX 404) and PCI.
Change control relies heavily on logging, since it provides evidence of when both unauthorized and authorized changes were made.
A well-architected logging system should integrate with the change management system in at least one — and ideally both — of these methods:
• The logging system should notify the change management system when a change occurs, so the administrator can either approve the change or record a violation and deal with it appropriately.
• The change management system (or an approved integrated automation system such as Chef or Puppet) should be able to initiate the change and include notes in the logs containing information such as the change control ticket number.
More granular access control lists = more meaningful logs
Dell Cloud Manager sits between the customer and their cloud, monitoring and controlling all user actions. Since it sees all management-level interactions between the user and the cloud, it can deliver role-based access control with far more granularity than any cloud
provider makes available. Additionally, it provides a consistent level of access control across all clouds, regardless of what the underlying cloud producer is capable of providing.
Auditing
A huge part of any compliance strategy is auditing, the process by which organizations are assessed by authorized third parties who certify that the organization is compliant with the relevant regulations. Auditors rely heavily on logging both as direct and indirect evidence of compliance. Directly, logs enable organizations to demonstrate that they are taking certain actions such as regularly evaluating who has access to what resources. Indirectly, the existence of logging demonstrates to auditors that there is sufficient evidence being gathered to detect and respond to security incidents when they happen.
Improving operations and development Logging does more than help
organizations track user behavior; logs can also be a rich source of information about what IT systems are doing and how well they are doing it. For instance, logs can be used to identify systems that are in need of maintenance. They can help identify when more systems are necessary, or when systems have been over-provisioned, allowing excess resources to be decommissioned. Logs are a great source of data for long-term capacity planning and forecasting. And logs provide valuable information to both developers who need to identify issues with applications and technical support staff who need to debug problems.
Improving operations and security with dynamic alerting
Of course, having logs isn’t very useful unless they are actually being reviewed and acted upon. Long-term analysis is a great use for the data. However, sometimes organizations need to be notified of possible issues or incidents right away. This is where alerting comes into play (although alerting may happen
It can be just as
important to receive
an alert when
expected actions
don’t happen as
when unexpected
events do happen.
based on the results on non-real-time analysis as well).
With a system monitoring the logs, automated alerts can be generated on conditions that are important to the organization. This can be very useful for the security, operations and applications teams. Sometimes, it can be just as important to receive an alert when expected actions don’t happen as when unexpected events do happen.
Types of notifications to consider When things happen that shouldn’t:
• Someone logging in outside of their normal hours
• Someone logging in from a dramatically different location than usual
• Outbound traffic from a key server to an IP address to which it doesn’t usually connect
• Unexpected changes to systems or networks (see the section above on change control)
• Systems unexpectedly go offline
• Additional systems or networks come online
When things don’t happen that should:
• Backups or other regularly scheduled jobs fail to happen or do not complete properly
• Network traffic drops below expected levels
• The number of users is much lower than expected
• Sales volumes suddenly drop
Alerting with Cloud Manager
Cloud Manager acts as a proxy between the enterprise and the cloud. This enables Cloud Manager to capture not only who performed what actions, but also when the actions were performed. All actions taken by Cloud Manager, whether via the console or the API, are logged. Alerts can be configured whenever certain actions happen (or fail to happen) and are sent via email.
Cloud Manager regularly polls the cloud providers and will also create an alert when it detects discrepancies between what it thinks is happening and what the cloud provider believes its current state to be. This is ideal for detecting when users are directly accessing the cloud
providers’ consoles instead of using Cloud Manager.
Logging and alerting in the cloud Logging and alerting in the cloud is very similar to logging and alerting in a traditional data center. Essentially, data is collected, sorted, consolidated and alerted upon.
There are, however, two key concerns to keep in mind when dealing with logging and alerting in the cloud:
• Lack of access to public cloud logs
• Transmission of logs from a public cloud
Lack of access to public cloud logs As noted earlier, by migrating to a public cloud, organizations are generating logging data to which they probably will not have access. Public cloud providers, as a rule, don’t make logs available to their customers showing what actions users have taken, whether via the provider’s console or the API. This can be a huge problem for any organization, but especially so for ones that have security or compliance requirements from regulators or customers.
Many regulations, such as PCI and HIPAA, require that organizations have control of the full lifecycle of the server, which includes controlling who launches or terminates a server or application. The lack of access to logs means that cloud- enabled organizations have created a gap in their logging of the server lifecycle, so while they may in fact be in compliance, they are unable to provide the proof to meet audit requirements.
Organizations can address this issue by deploying a proxy between themselves and their cloud provider. A proxy enables the organization to capture all console or API-level logs in a format that can be easily managed by that organization.
Transmission of logs from a public cloud Consumers of a public cloud need a secure mechanism for transmitting logs from their cloud instances back to a centralized log repository. Most
4
organizations don’t want to open up their firewalls to the incredibly broad range of IP addresses that cloud service providers have to get those logs. This can be handled in a couple of other ways:
• Use the aforementioned proxy along with firewall APIs to automatically open the necessary IPs and ports for instances as necessary.
• Use an existing operations or security agent to direct the logs to the correct place, using the agent’s central server. A pull mechanism could be used, where the log management server regularly polls the cloud servers and pulls the logs down.
Or a tiered log management structure could be built where logs are centralized into one or two instances on a cloud provider and then forwarded on to the log management server.
Transmitting and consolidating VM logs with Cloud Manager
Cloud Manager provides multiple mechanisms for consolidating virtual machine (VM) logs. Using Cloud Manager’s automation and systems configuration modules, you can configure VMs to direct their logs to the appropriate boxes. Alternatively, you can use those same modules to enforce the installation and running of both commercial and open source log management agents such as Splunk or logstash, and then use the Cloud Manager agent’s secure communications channel to transmit the appropriate logs to the correct location via API calls.
Either of these options will provide the desired result. An organization contemplating log transmission from within their public cloud should evaluate both and see which option will be easiest to manage with available resources.
Conclusion
Logging and alerting are key services needed by any organization, but especially ones that have to meet security or compliance regulations.
Alerting helps staff proactively resolve emerging issues, while logging ensures
that organizations can show that they are in full control of their systems, and can also be useful for long-term analysis.
Organizations collecting logs from their clouds need a strategy to get the logs from the cloud provider to their centralized log servers and cover the API logging gap. By managing these two concerns, organizations can ensure that they have the necessary logs to safely conduct business while taking advantage of the operations and economic benefits of the cloud.
About Dell Cloud Manager The enterprise cloud management
solution
Cloud Manager is a cloud infrastructure management solution for deploying and managing enterprise-class applications in public, private and hybrid clouds. The Cloud Manager multi-cloud architecture provides enterprises around the world with agility, governance and choice.
Agility — Easily deploy and manage cloud applications across public and private clouds. Developers can utilize self- service provisioning, a service catalog of pre-approved templates for easy re-use and automation capabilities to quickly deploy VMs, curated software stacks and complete applications.
Governance — Maintain control over cloud operations through integration with identity management systems, fine- grained access controls, financial tracking and logging/monitoring of all actions taken within your cloud environment.
Choice — Cloud Manager supports the leading public and private cloud platforms, plus leading configuration management tools, and integrates into your existing operations, such as billing and monitoring.
Cloud Manager can be delivered as software as a service (SaaS) or deployed on-premises. For more information, visit www.enstratius.com.
Consumers of a
public cloud need a
secure mechanism
for transmitting logs
from their cloud
instances back to
a centralized log
repository.
About Dell Software
Dell Software helps customers unlock greater potential through the power of technology—delivering scalable, affordable and simple-to-use solutions that simplify IT and mitigate risk. The Dell Software portfolio addresses five key areas of customer needs:
data center and cloud management, information management, mobile workforce management, security and data protection.
This software, when combined with Dell hardware and services, drives unmatched efficiency and productivity to accelerate business results. www.dellsoftware.com.
If you have any questions regarding your potential use of this material, contact:
Dell Software 5 Polaris Way Aliso Viejo, CA 92656 www.dellsoftware.com
Refer to our Web site for regional and international office information.
© 2014 Dell, Inc. ALL RIGHTS RESERVED. This document contains proprietary information protected by copyright. No part of this document may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose without the written permission of Dell, Inc. (“Dell”).
Dell, Dell Software, the Dell Software logo and products—as identified in this document—are registered trademarks of Dell, Inc. in the U.S.A. and/or other countries. All other trademarks and registered trademarks are property of their respective owners.
The information in this document is provided in connection with Dell products. No license, express or implied, by estoppel or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Dell products.
EXCEPT AS SET FORTH IN DELL’S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT,
DELL ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL DELL BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR INABILITY TO USE THIS DOCUMENT, EVEN IF DELL HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Dell makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves the right to make changes to specifications and product descriptions at any time without notice. Dell does not make any commitment to update the information contained in this document.
For More Information
