Cloud Computing 3D:
Seeing through the Clouds
Author: Dr. Rupert Ogilivie
Introduction
Virtualization is a vigorously expanding industry that enables enterprises to consolidate IT infrastructure and has now been embraced in some form by most enterprises. “The Cloud” is the de facto demonstration of virtualization as a service, however, there are still many concerns regarding the Cloud which are proving to be a bar to mass migration. Visualization is a technique used to express data sets. Visualization facilitates the extraction of meaning from the consolidation and mining of disparate and complex data-sources, but has yet to be widely applied to the problems surrounding migration to the Cloud. This paper aims to draw out some of these concerns and examine how visualization could be used to complement and help drive forwards virtual migration and innovation.
What is meant by Visualization?
Looking at a log file is a form of visualization. Not a particularly pleasant medium in which to troubleshoot, check performance or even predict events. Even the Visio diagram has its limitations as a tool to accurately represent complex IT infrastructure and relate information to the events happening in the network in real-time. When it comes to highly complex Cloud infrastructures, there is a need for a new paradigm to map and manage computing. In this paper, visualization refers to a new breed of visual techniques to see IT devices, their links and processes. These techniques are not ‘nice to have’; it is the only way of taking control of the constantly evolving Cloud environment.
Traditional visualization techniques do not allow easy layering of information. Data exists in flat, separated silos with little, if any, correlation between them. By layering information and bringing it up into three dimensions, data no longer has to remain in isolated pools but can be correlated and analysed like never before. The techniques that allow this correlation are referred to as 3D visualization and their implications for the Cloud are discussed next.
Meeting Challenges
Data Governance
The Cloud provides a flexible and easily scalable platform for deploying web-apps and other services. Many companies, and the countries which regulate them, have tight rules surrounding data access, for instance the US Patriot Act. It is not uncommon that local regulation requires that data is kept within safe harbours; typically with the consequence that confidential or personal data is not allowed on any server which physically exists outside of the governing bodies' jurisdiction.
Whilst the physical location of Cloud servers can currently be tracked, the nebulous nature of the Cloud means that many companies containing regulated information are unwilling to take the step to a Cloud-based infrastructure, as they are concerned about being able to meet the burden of proof required by legislation. For instance, if an employee accesses a data-store in another country, how much of their data can be collected and stored by external 3rd parties?
Clear visualization of the location of data sources provides a stepping stone towards mitigating that burden of proof. Any IT infrastructure existing fully or partially within the Cloud can assess the position and integrity of data owned at any moment. By providing a mechanism for “at-a-glance” checks on the global position of any data stores, a data
manager within any organisation can be reassured that they are operating within the governmental or departmental strictures placed upon them.
Visualization also provides enhanced insight into the transmission of information across the network. Using visual cues to represent the flow of information, sensitive data can be traced throughout a network and highlighted the instant it is accessed by unauthorised processes or traverses unsecured infrastructure. With the entire IT estate visible from just one screen, operators can monitor, shut-down and perform root cause analysis on rogue flows. Additionally, the Cloud is often viewed as an anonymous storage base of persistent information (e.g. Facebook). If the location of data, as well as backups and duplicates, can be tracked and monitored within the Cloud then the organisation owning them can be satisfied that when data needs to be purged, all copies of the data can be suitably cleansed.
Performance Monitoring
Service managers who have lost their locally hosted applications to the Cloud will naturally perceive that quality of service (QoS) could be compromised. They will not necessarily know what computing resources have been allocated to their applications, nor will they have quality of service control over the transmission of data through the Internet. Without these guarantees, it is impossible for any organisation to offer Service Level Agreements (SLA), and without SLAs, Cloud computing will not be able to attract large enterprises to their service offerings. “Amazon found every 100ms of latency cost them 1% in sales. Google found an extra .5 seconds in search page generation time dropped traffic by 20%. A broker could lose $4 million in revenues per millisecond if their electronic trading platform is 5 milliseconds behind the competition.” 1
With these statistics, no one would enthusiastically move to a platform where they don't have clear visibility of performance and where they are unable exercise strict control over all aspects of the IT estate. 3D visualization provides clarity through the representation of topology, enabling the delineation between where one area of responsibility ends and another begins; such as the inter-connect between two service providers. Once the areas of responsibility are established, the status of network or computing ‘health’ can be shown through visual cues against the limitations expressed in the SLAs. With delay, jitter, loss and bandwidth utilisation all shown in real-time, proactive steps can be taken such as identifying hot spots as they build up or providing data to assist in capacity planning decisions. 3D visualization provides a way to assess the overall “health” of the whole IT infrastructure, enabling operators to react to situations before they become problems. Additionally, more passive benefits can be also be gained by ensuring that responsibility is correctly assigned when SLAs are breached and providing cost estimates for interdepartmental cross-charging.
1
Security
Cloud based services are also often viewed with suspicion by those responsible for network security. Concerns are typically raised over the security of the virtualization platforms on which the Cloud runs. On top of this there are the usual security risks over any public facing service, as well as those surrounding the adoption of new technology and the creation of fresh avenues of attack and exploitation. Being able to correlate different data sources providing information on the current security and vulnerability status of the network is a challenging problem, typically resolved through the use of heat-maps. Whilst heat-maps can and do provide a way of correlating large quantities of information, the information is not usually placed in context of where events are happening in the network or tied to a sense of business criticality. Using 3D visualization, data is presented in a more intuitive way, allowing larger trends within the data to be spotted, business information to be overlaid and more security data sources to be included in the view.
Capacity Management
One of the touted benefits of the Cloud is its elastic properties. As a service provider the customer no longer has to worry about demand outstripping supply and the knock on effect that that has on the ability of their hardware to deliver. By correctly configuring their Cloud-based services, a customer should be able to theoretically handle any number of clients without the overheads associated with maintaining a large amount of hardware to cope with spikes in demand. Obviously, the key term here is “correctly configured”. Monitoring how the Cloud is responding to the demand placed on their services and the performance being experienced over time is an essential guide as to how close to optimal configuration the current settings are. It may also be the case that over time patterns of demand or the evolving nature of the service led to the need for a reassessment of the scaling settings. As these updates help pre-empt a loss in quality and the corresponding loss of custom or revenue through violated SLAs they are an essential part of keeping the service on the cutting edge. 3D visualization is an ideal method for tracking the scaling and trends of resources and demand across a large IT estate, as there will be many machines querying many different data sets at different times and rates. By being able to display all this information at once, 3D visualization provides an intuitive way of absorbing these complex data sets and by understanding them, optimal configurations can be reached.
Some companies may be approaching the point at which “Cloud Bursting” occurs. Cloud bursting is a term used to denote the point at which internal pressure on resources has reached critical levels and the next step is either to invest heavily in more internal hardware or to migrate services into the Cloud. Taking a visual approach to assessing current resource usage will allow a company to determine how close they are to capacity, where savings/efficiencies can be made in the short term and help to plan for a graceful migration of services.
Agility
Often internal projects can be hampered by lack of scalability and a scarcity of equipment to perform the testing and validation needed to convert valuable ideas into production quality services. The use of the Cloud allows many projects to be run quickly and in parallel with significant savings, compared to the “physical” alternative. This is often referred to as Platform as a Service (PaaS). These cost savings and the different virtual appliances available allow companies to easily try out ideas which might under normal circumstances be considered to be too risky – allowing new revenue streams and diversification of services. In these situations tracking and monitoring resources used and the effectiveness of the service is vital. 3D visualization provides an intuitive insight into resource usage over time, as well as allowing overlays of different results from the project. This help those at a technical level make presentations to management, and also to enable management to overlay their own business information to allow all involved to assess the outcome of the project.
Cost and Budget
One of the recurring themes of this paper so far has been the ability of 3D visualization to make computing cheaper, whether through the reallocation of resources, improved planning and decision making or faster identification/pre-emption of problems as/before they occur. Another useful facet of visualization is the ability to blend different data sets. Network problems are often not the result of a single event in a single area. When a network operator is faced with multiple issues arising from many segments of a company a decision has to be made as to which to prioritise.
When the technological big picture is overlaid with business information such as costs per minute of downtime, service dependencies etc., the decision making process becomes that much easier as the servers’ monetary value can be easily seen. In traditional solutions merging the virtual and physical views is a non-trivial challenge. A visualization solution easily scales into the Cloud, seamlessly merging the local network view with the Cloud infrastructure and services.
Optimising Infrastructure
Moving applications to the Cloud may also move them from centralised datacentres, which previously served large swathes of the company. The new location of the servers may also require the breakout points to external networks from within a company to be reconsidered as there may be requirements to reduce the response time across the infrastructure, in order to maintain performance SLAs. The possible network redesign needed and any additional planning required is much easier to do with visual cues to work from and in a large organisation, the existing Visio diagrams may be out of date or simply too large to comprehend in any one, easy, view. 3D visualization techniques can assist in this process by giving IT specialists a visual canvas to rapidly detect flaws in the infrastructure, such as single points of failure, overloading of switches, under-utilisation of resources etc.
Process Tracking
The Cloud is often marketed as a cure-all panacea for a company’s IT woes. But once a process has been migrated, it is important to maintain the same structured and rigorous approach to managing the processes as was previously applied in the local infrastructure. As this comic strip from geek & poke satirises, it is easy to lose track of what exactly is happening and when.2
Conclusion
Cloud migration can bring many benefits and savings to customers but currently there is a lack of techniques and tools to provide the visibility and reassurance that decision makers require. Uptake of Cloud based services would be accelerated through improved visualization techniques. Even day to day issues and events risk causing information overload as operators struggle to understand the complex play of elements within their networks. Visualization flips information overload onto its head by providing ways to further increase the quantity of data visible at any time, whilst maintaining the ability of IT operators to successfully interpret these data to make timely and effective decisions.
About the Author
Dr Rupert Ogilvie is an IT Optimisation Consultant within Intergence Systems. His particular areas of specialisation are WAN governance, Infrastructure optimisation and Cloud visualization. Rupert holds a PhD in QoS from Queen Mary College, University of London.
http://geekandpoke.typepad.com/geekandpoke/2009/03/let-the-clouds-make-your-<END PAGE>
About Real Status
Pioneers in IT modelling and visualization, Real Status make what was previously unseen visible through cutting-edge intuitive solutions, equipping organisations with the insight needed to deliver IT decision-making based on the big picture.
Until now, mapping and modelling the complex inter-dependencies between IT infrastructure, data and applications has relied upon disparate dashboards and limited methods of
visualization. As demands upon IT infrastructures within distributed and virtualized computing environments increase, the limitations of this silo approach become more apparent and detrimental. Effective IT decision-making demands the ability to see – and in seeing, understand – all dimensions of inter-dependence at the same time, in the same place.
