Enabling Trusted Multi-Tenancy with Vblock Systems

(1)

Enabling Trusted Multi-Tenancy

with Vblock

®

Systems

Version 1.0 March 2015 www.vce.com

(2)

THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." VCE MAKES NO

REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OR

MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

VCE believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

(3)

Introduction

Organizations are consolidating as much infrastructure as possible into the simplest to manage, smallest possible footprint, such as a single VCE Vblock® System. This consolidation frequently includes workloads that might have existed in physically discrete security boundaries supported by separate operational silos. This document explores variations on this consolidation scenario. Physically discrete infrastructure traditionally met organizational security objectives; however, it often led to inefficiencies in terms of underuse of resources and vendor sprawl, and adversely impact business agility in IT processes.

Trusted multi-tenancy (TMT), as implemented using Vblock Systems, addresses the mechanics of creating secure boundaries between tenants in different trust zones. Individual tenants—or workloads, application tiers, applications, DMZs, departments, partners, customers, missions, and so forth—likely have business

requirements mandating specific segmentation needs regarding administrative, network, compute, and data access. In some cases, parts of the management or infrastructure might be shared with other tenants. For example, a service provider (or perhaps enterprise IT) might handle all administration tasks with a common management framework, while other resources are largely dedicated to—and managed by—tenants (or departments). Meeting the differing segmentation needs of these diverse environments requires flexibility and design modularity. A range of capabilities is available to enable this modular trust model, providing controls supporting separation and protection at various levels of administration, communications, and access. The modular trusted multi-tenancy model is implemented using core Vblock System controls, along with optional technologies from partner products. In practice, similar products can be substituted for the optional products discussed in this document, although overall solution interoperability and/or security profile might be impacted.

Document purpose

This document examines controls both inherent to Vblock Systems and provided through partner products that enable secure or trusted multi-tenant deployments. As segmentation can mean different things, this document establishes segmentation boundaries and definitions, and explores risks and fundamental mitigation

considerations. Particularly important is an understanding of two major risk classes:

 _{Lateral tenant influence -- resource leakage between tenants, either with information flow}

consequences and the possibility of further resource allocation tampering (for example, cracking cryptographic keys through monitoring CPU activity), or as a volume issue, impacting service availability, variants commonly known as the “nosy neighbor” and “noisy neighbor” problems.

 _{Vertical tenant influence -- using one layer within an architecture as a privileged “jumping off point”}

(6)

Applying segmentation to Vblock Systems requires an understanding of the Vblock System, its components, and their capabilities. Vblock Systems are highly adaptable platforms, but the baseline must be well

understood for maximum effectiveness in deployment for multi-tenancy scenarios. This document examines some of the controls available in Vblock Systems networking, storage, compute, and management domains:

 Controls that provide separation that is physical, such as dedicated hard disks and dedicated servers

 Controls implemented at a logical level, such as virtual local area networks (VLAN), virtual storage area networks (VSAN), and virtual firewalls

 Common combinations of controls and how VCE has worked with customers to turn catalogs of capabilities into functional systems

This document also explores topics having either a variety of potential solutions at the physical or logical levels or design considerations that apply through the architecture assembly, such as encryption and scaling.

Scope

This document assumes a business decision has been made to leverage one or more Vblock Systems to provide services to multiple tenants. There are many important considerations in designing information systems and security requirements to consider early in the system development process. Designing a shared infrastructure solution for multiple tenants can be a complex task. This document provides high-level guidance to help understand and address key security concerns for a multi-tenant Vblock Systems solution. VCE recommends customers consider a VCE professional services engagement to address out-of- scope areas such as:

 Scale: One tenant, tens, hundreds, thousands? How will things look over time? How important is

flexibility?

 Tenant characterization: What do tenants need? Expect? What are their compliance requirements?

Resource expectations? Bursting needs?

 Supplemental services: What additional functionality are tenants going to want, such as

authentication, application delivery controller (ADC), name services, and so forth?

 Data protection (BC/DR): How do you plan to handle things going wrong? How do tenants expect

you to handle things going wrong? And what is the geographic footprint of your operations? Can you make more money off of a more comprehensive approach?

 Deployment models: Are tenants integrated manually or will there be an

infrastructure/platform/software/communications-as-a-service offering?

 Workspace management model: Following on with the deployment model, who manages what for

whom through what interfaces? How much do tenants expect to be able to do for themselves? Consider areas like infrastructure auditing, which is traditionally awkward for service providers.

 Infrastructure management: Patching and release management can be especially challenging,

particularly when coordinating with a large number of parties. VCE’s Release Certification Matrices (RCM) de-risk technical aspects of this, but procedurally it is potentially a complicated topic in single tenant environments and more so in multi-tenant settings.

(7)

This document does not address basic security controls—which are assumed to be applied as appropriate— unless specifically relevant to trusted tenancy. The document looks briefly at front-end aspects of multi-tenancy, such as portals and orchestration tools. As these products work as expected on Vblock Systems, the focus is on areas that might be new, such as multi-tenancy aspects of the core infrastructure. As this

document focuses on the core infrastructure components, it also does not address software-defined networking (SDN) elements.

Audience

The intended audience for this paper is data center and security architects tasked with implementing multi-tenancy upon an infrastructure built using one or more Vblock Systems. This document does not provide detailed implementation guidance. Readers are assumed to have a reasonable understanding of Vblock Systems architecture (as documented in the various VCE Vblock System architecture overview guides and related product documentation.).

Feedback

To suggest documentation changes and provide feedback on this paper, send email to

[email protected]. Include the title of this paper, the name of the topic to which your comment applies, and your feedback.

(8)

Secure segmentation

Examining multi-tenancy security requirements typically involves looking at the relationships between all parties in the infrastructure, including tenants and platform administrators. This document uses that perspective to address both secure segmentation within Vblock Systems and the concepts of tenancy and potential threat vectors.

Tenancy boundaries

Infrastructure resources can be characterized by the following:

 How the resource (including applications) is managed, monitored, and used, and any constraints on resource management

 How associated data (including backups, snapshots, business continuity/disaster recovery data, and so forth) is stored and any constraints on storing or accessing data

 How associated applications are consumed, in what trust zones consumers are located, protections regulating consumption, and what trust or protective elements need to be provided locally or assumed/inherited from the infrastructure

Multi-tenancy arises when the differences in characteristics for two or more blocks of infrastructure resources become too large to resolve through simple process. These resource blocks—frequently referred to in front-end orchestration platform as containers—become “tenants.” Think of each tenant as having some unique— although potentially similar—set of agreements or “contracts” with platform administrators and other entities relating to management, data, application access, and so forth, effectively regulating security, performance, and, possibly, scaling.

Note: This document uses the concept of a contract similar to how it is used by software-defined networking vendors: as a mutually agreeable high-level policy directive used to shape specific implementation details between two parties. Although not necessarily a security concept, this document focuses on contract areas related to trusted multi-tenancy.

The following are some examples of how blocks of resources might be designated as tenants based on characteristics:

 Different parties with financial responsibility for the resources; for example, different companies (traditional service provider scenario), different divisions in an enterprise

 Different categorizations within a compliance framework; for example, one block of resources hosts a PCI Cardholder Data Environment (CDE), while another block does not

 Different tiers within an n-tier application (dedicated tiers), or the more traditional legacy model of shared tiers for DMZ, application servers, databases, and so forth, each of which might have full set of compute, networking, storage, security, and, potentially, management, resources

(9)

Acting on behalf of the tenant, the data owner has the best perspective on the tenant’s security requirements— these requirements being the basis for a subset of the above-mentioned contracts. The data owner’s

responsibilities for championing the security interests of the tenant might be shared with—or delegated to— other entities, such as the corporate security organization in a private cloud multi-tenancy scenario. However, ultimate responsibility remains with the data owner for appropriate treatment of tenant resources.

Data owners—or their delegates—are strongly encouraged to challenge assumptions and resulting contracts that seem overly complex or where it is unclear how the technology can support the contract.

Risks, exposures, and controls

The following risk-related terminology is used in this document:

Term Definition

Risk The probability that a threat agent will (act upon a threat vector to) exploit a vulnerability Exposure The impact, typically expressed financially

Control The countermeasure or safeguard that eliminates, reduces, or transfers risk associated with one or more exposures

(10)

Tenants—specifically, the data owners—have a responsibility to identify and take appropriate steps to protect valuable information. As this information is typically the reason the infrastructure exists, it makes sense to first examine the risks and exposures inherent to tenant contracts. Management security also has a significant impact on infrastructure security. Therefore, this document explores the risks and exposures inherent to management contracts that do not directly involve tenants.

Note: Although the focus of this document is technical, there are areas where the appropriate path is not purely technical but rather an administrative or procedural control to better safeguard certain

exposures. While VCE can advise in creating and implementing such controls, they are not addressed in this document.

This section outlines multi-tenancy security expectations organized by relationship that loosely corresponds to key threats and threat agents or, in some cases, jumps right to countermeasures (typically for areas this document will not cover in more depth). However, without organization-specific risk factors and asset/data valuation, gauging the importance of specific risks and determining an appropriate investment strategy in compensating controls is an arbitrary exercise. VCE can work with you to help understand your risks in the context of Vblock Systems and to prioritize mitigation of those risks.

Tenant – platform administrator

The following table describes, in the form of contracts, some security relationships and expectations between tenants and platform administrators, such as service providers or centralized IT organizations.

Note: Use the contract tables in this section as a template when building a model for assessing specific multi-tenant deployments. By building out and adapting the contracts to specific deployments, the scoping details become easier to manage for compliance objectives such as PCI. The contracts template becomes a tool in the ongoing assessment process.

Contract Discussion Provider mitigates

against virtual machine escape

The targeted threat is based on the idea that, having gained a foothold within a virtual machine, an attacker could perform a privileged attack upon the hypervisor and use it to access other resources, including those belonging to other tenants.

There are trusted multi-tenancy architectures that remove a large portion of the virtual machine escape risk –specifically in the multi-tenancy context—simply by isolating tenants to their own hypervisors. Note that attacks over service networks and against storage would have different risk characteristics after a successful virtual machine escape, so the risk is not completely mitigated.

Provider implements robust administrative and procedural controls for infrastructure

Encompasses a range of threats; the specific concerns vary depending on the nature of the multi-tenant relationship, the assets being protected, and the applicable compliance frameworks. Key areas include:

 Physical security for shared infrastructure, management infrastructure, monitoring resources, business continuity and disaster recovery, and so forth.

 Background checks and appropriate hiring protections.

 Change management practices. Is the provider more “fail open” or “fail closed”? If something goes wrong how does it align with the tenant’s business posture?  Incident response plans. Such plans must include scenarios for malicious insiders.

(11)

Contract Discussion Provider mitigates

against lateral attack (or leakage) via storage

Storage sub-systems within Vblock Systems might include one or more highly available storage arrays connected via traditional fiber channel SAN switches. These storage sub-system(s) are heavily leveraged due to the availability, performance, and broad spectrum of data services offered by these solutions (for example, flash, auto-tiering, encryption, de-duplication, replication, and so forth.). As a result, storage sub-systems and their associated management infrastructure are high-value targets for attackers and careful consideration of threat potential must be taken in the solution design. When properly hardened, storage systems are difficult to attack, even after virtual machine control is obtained using a datastore hosted on the SAN.

Provider mitigates against lateral attack (or leakage) via network beyond what be possible from “outside”

Hosts within a private network (whether delivered within a traditional enterprise or a cloud) achieve efficient operations by relying on management networks for monitoring, backup networks (perhaps replication or similar networks) for high availability services, and potentially many special-purpose networks that might have large footprints within the data center. In a shared environment, an attacker might reasonably expect that these networks would exist and that one or more might bridge multiple tenants. The provider can mitigate risk by preventing traffic across any of the service networks between two tenants.

Provider takes appropriate efforts to mitigate against signals intelligence attacks

In the past, simply sharing a host with an application performing frequent cryptographic activities (for example, SSL encrypts/decrypts) has been sufficient to extract the secret keys, based on changes in access to the CPU. Elasticity exposed can also be revealing, perhaps showing business prematurely won or lost.

Tenant – tenant

The following table describes sample security relationships and expectations between tenants. Note that responsibility for implementing these expectations falls largely to the platform administrator. There are two clusters of scenarios to consider for tenant-tenant interactions:

 Traditional service provider environments, where each tenant is a unique paying customer

 Central IT environment, where a tenant is a more flexible concept Service provider scenarios

Contract Discussion Traditional tenant

cannot see evidence that other tenants exist

Management tools, disk, and resource contention should not reveal any form of data about another tenant unless it is an explicit business model; for example, multiple tenants under the control of the same consumer.

Special provider tenants such as service level agreement monitoring systems might have visibility requirements into tenant resources, but there must be tight constraints. The tenant should know what the provider has access to and under what circumstances.

(12)

Central IT scenarios

Contract Discussion

Requirements vary Unlike the provider model, tenants within the same company or organization may need to interact frequently. The degree of separation between tenants can vary widely, even within a single deployment. For example, a government entity might maintain a very high level of separation between missions while also treating the application tiers within a mission as sub-tenants with a lesser degree of isolation.

Special tenants, such as central security management systems or network management systems, need privileged access along certain vectors. Data owners or their proxies should ensure such access is compatible with mission requirements.

Platform administrator – supplier

The following table describes the security relationships and expectations between platform administrators and suppliers.

Contract Discussion Provider secondary

relationships are insulated from tenants

The platform administrator likely contracts with suppliers having network, remote, or physical access to the infrastructure or to secondary infrastructure critical to its operation (for example, power, cooling, network uplink, certificates).

The provider is responsible for implementing and maintaining the technical, administrative, and procedural controls that remove these risks. The tenant is responsible for assessing thoroughness against their requirements and ensuring that an appropriate validating entity can attest to implementation of the controls.

Controls

Controls are technical, physical, procedural, or administrative measures collectively intended to mitigate risks. In many cases, the alignment is not direct as a combination of controls might address a handful of identified risks. Additionally, controls from different categories might come together to reinforce one another, as in this example on operations hygiene:

1 Specific VCE technology choices (Release Certification Matrix (RCM), the related compliance testing capability in VCE Vision™ Intelligent Operations, and its RCM pre-positioning capabilities) reduce testing, clutter, web downloads, risky behavior, and potential bad hygiene on production management systems.

2 Processes encouraging testing outside of production, staging on transient systems, and so forth, reduce risk through behavior optimization.

Ideally, it would be enough to tell people to do things properly but in practice, simultaneously making it easier for them to do the right thing yields superior results. From the security perspective, making it harder for them to do the wrong thing reduces risk and increases trust.

(13)

VCE trusted multi-tenancy design

The VCE trusted multi-tenancy approach is based on the concepts that each organization is different with significant diversity in its threat environments and observed risk appetites. VCE uses a modular approach to multi-tenant security to help organizations identify the right path to their needs, balancing security, compliance, cost, business models, and other requirements to build a tailored solution using predictable building blocks that provide supportability and consistency.

VCE Vblock Systems are manufactured blocks of data center capacity supported by a single telephone call. Trusted multi-tenancy adds layers of capability reflecting an organization’s particular needs, but fundamentally remains the same set of platforms.

Segmentation options

Segmentation controls enforcing the divisions between tenants are critical to the trust aspect of trusted multi-tenancy. Various physical or logical mechanisms can enforce segmentation, making available pools of dedicated and/or shared resources. Due to the modular nature of trusted multi-tenancy, many organizations choose to combine options to build an overall solution. In fact, some of the individual technologies covered in this document such as disjoint layer 2, encompass elements of physical and logical segmentation.

Native physical segmentation options include discrete Cisco UCS domains, dedicated compute nodes, and dedicated storage pools. While there are only so many practical ways to insert air between two physical components, the range of possible logical segmentation options is considerable, particularly when considering the ecosystem. Native logical segmentation options include Cisco UCS profiles, VMware vCenter clusters and resource pools, Virtual Data Movers and VSAN, Cisco UCS Virtual Machine Fabric Extender (VM-FEX), VLAN, and port channel controls. This document addresses each of these in more detail.

The native capabilities of Vblock Systems are only the starting point as trusted multi-tenancy deployments often involve ecosystem partner and/or third-party activity to bring the platform to full operation. This document addresses some of the most frequently used segmentation controls introduced through the ecosystem, such as virtual firewalls.

Compliance regimes or best practices such as data-at-rest encryption might require that other areas be addressed. These might have complex answers, depending on specific needs, hardware preferences, ecosystem partnerships, and so forth. It might be possible to address an organization’s needs with native capabilities, but where that is not feasible, VCE can help provide a product or trusted partner to meet the needs.

(14)

Trusted multi-tenancy technology overview

The Vblock System from VCE is the world’s most advanced converged infrastructure—one that optimizes infrastructure, lowers costs, secures the environment, simplifies management, speeds deployment, and promotes innovation. The Vblock System is designed as one architecture that spans the entire portfolio, includes best-in-class components, offers a single point of contact from initiation through support, and provides the industry’s most robust range of configurations.

VCE offers a converged infrastructure portfolio that includes a variety of Vblock Systems, including the Vblock System 200 family, Vblock System 300 family, Vblock System 500 family, and Vblock System 700 family. Each family has varying compute and storage platforms and scalability. As engineered products, these systems use common architectures. The question of how the architectures handle storage networking is very helpful when designing a trusted multi-tenancy solution.

Note: VCE also supports larger architectures and SDN offerings from Cisco and VMware. As most controls implemented through these solutions focus on the upper—orchestration—level of the stack, these are not currently in scope for this document. Contact VCE for more information on multi-tenant solutions based on VCE Vscale™ Architecture, Cisco Application Centric Infrastructure (ACI), or VMware NSX-v.

Vblock System components

The following table summarizes the components used in the larger Vblock Systems families. Optional components are listed in italics.

Technology Vblock 300 family Vblock 500 family Vblock 700 family Network Cisco Nexus 9000 Series

Switches

Cisco Nexus 5000 Series Switches

Cisco Nexus 1000V Switch Cisco MDS Switches

Cisco Nexus 1000V Switch Cisco MDS Switches Storage EMC VNX EMC VPLEX EMC RecoverPoint EMC XtremeIO EMC VPLEX EMC RecoverPoint EMX VMAX EMC VPLEX EMC RecoverPoint

(15)

Technology Vblock 300 family Vblock 500 family Vblock 700 family Compute Cisco Unified Computing

System (UCS)

Cisco UCS 5000 Series Blade Server Chassis

Cisco UCS B Series Blade Servers

Cisco UCS 6000 Series Fabric Interconnect

Cisco UCS

Virtualization VMware vSphere VMware vSphere VMware vSphere Management VMware vCenter Server

Cisco UCS Manager EMC Unisphere Manager Cisco Data Center Network Manager

VMware vCloud Director VMware vRealize Automation Cisco UCS Director

VMware vCenter Server Cisco UCS Manager EMC Unisphere Manager Cisco Data Center Network Manager

Security Cisco Virtual Security Gateway VMware vCloud Networking and Security*

Cisco Virtual Adaptive Security Appliance (ASA)

Cisco Virtual Security Gateway VMware vCloud Networking and Security*

Cisco Virtual ASA

Cisco Virtual Security Gateway VMware vCloud Networking and Security*

Cisco Virtual ASA

*Included as a component of VMware vCloud Suite

VCE ships Vblock Systems in both traditional and unified storage networking configurations. The following table highlights the differences between these architectures. The Cisco MDS switch provides more flexibility in trusted multi-tenancy scenarios.

Architecture Features

Traditional modular  Uses Cisco MDS switches for SAN networking  Advanced security features available

 Simplifies role administration for IP and SAN management  Higher physical port capacity

Unified modular  Uses ports on existing Cisco Nexus switches for SAN networking  Fewer devices, fewer ports to manage

(16)

Each Vblock System ships with an Advanced Management Platform (AMP) that provides the resources needed to configure and troubleshoot a Vblock System in the event of a problem. The AMP helps mission-critical resources rapidly return to operation. It can be enlarged and used to run a more general set of management workloads, but it is usually recommended that anything beyond VMware vCenter Server be allocated its own space on a Vblock System rather than be placed on an AMP.

As the AMP hosts element managers, it is a security target and needs to be protected. Most AMP variants are locked down and cannot have additional tools installed, the exception being the high-availability (HA) AMP. However, many trusted multi-tenancy scenarios involve higher levels of interaction with AMP resources and the installation of additional security or monitoring tools that might need to be close to element managers, such as VMware vCenter. Therefore VCE strongly recommends using the HA AMP for Vblock Systems used for trusted multi-tenancy.

Vblock System layers and secure segmentation

VCE’s multi-tenancy approach addresses secure isolation and segmentation at each layer of the Vblock System, as shown in the following table.

Component Description

Network Multi-tenancy concerns can be addressed at multiple levels within the network infrastructure of the Vblock System. Various methods can enforce network separation, including disjoint layer 2, virtual and physical firewalls, and VLANs.

Storage By default, data within Vblock Systems is accessed through block-based mechanisms (storage area networks (SAN)). It can optionally be accessed through file-based network attached storage (NAS), such as CIFS or NFS. Segmentation can occur within the storage itself or in transit. Options for segmenting block storage include SAN zoning and LUN masking. Options for segmenting file-based storage include VLANs and Virtual Data Movers. Encryption is always an option as well.

Compute and virtualization

Multi-tenancy concerns are addressed at multiple levels within the Vblock System compute and virtualization infrastructure, including the Cisco Unified Computing System and the VMware vSphere Hypervisor.

Management The Vblock System AMP provides a single management point, enabling organizations to monitor and manage the health, performance, and capacity of Vblock Systems. It provides fault isolation for management, eliminates resource overhead on the Vblock System, and provides a clear

demarcation point for remote operations. The AMP hosts technologies such as VCE Vision™ Intelligent Operations, Cisco Data Center Network Manager, Cisco Nexus 1000V Virtual Supervisor Modules, EMC Unisphere, and the EMC Secure Remote Services storage monitoring system. VCE Vision Intelligent Operations is part of the management environment and provides essential secure operational hygiene functions such as backing up configurations, additional logging, discovery, and compliance reporting for firmware versions and hardening configurations.

(17)

Physical segmentation

Traditionally, data center architectures contain dedicated stacks of infrastructure providing isolated environments for individual tenants, each with their own dedicated physical resources. The physical segmentation options available on Vblock Systems allow a shared infrastructure to provide much of the isolation of a traditional standalone architecture while continuing to leverage the benefits of converged infrastructure.

Trusted multi-tenancy high-level design

VCE's multi-tenancy approach empowers IT administrators to choose from among a range of cost effective mechanisms to separate the tenants within a Vblock System while the tenants share a common infrastructure. The VCE multi-tenant architecture for physical segmentation provides:

 Increased dedicated hardware resources for each tenant

 Increased path isolation

 Increased resource usage at lower cost

 Ability to meet organizational requirements for separating data when physical separation is mandated

 Different tenants on the same multi-tenant platform

Design considerations

While physical separation options exist at multiple layers within Vblock Systems, there are trade-offs between physical separation and effective resource usage. Organizations with tenants requiring physical separation between network, data, compute, and applications require dedicated underlying IT resources for those tenants,

(18)

Choose a modified level of physical isolation based upon organizational policies and business objectives. Consider the following design decision points and the extent to which each is individually necessary to achieve overall segmentation objectives:

 Network layer separation

 Storage separation

 Compute separation

 Management

Network layer separation

Network layer separation is one of the first important steps to take to build a trusted multi-tenancy network. VCE's trusted multi-tenancy approach provides the same degree of tenant isolation as a dedicated infrastructure.

The following controls are available for physical network separation:

 Disjoint layer 2

 Dedicated port channel

Disjoint layer 2

Organizations can consolidate workloads on a single converged infrastructure using disjoint layer 2 networks. Disjoint layer 2 provides separation between compute, storage, and network resources. The VCE trusted multi-tenancy approach enables traffic isolation at either the Cisco Nexus switches or Cisco UCS fabric interconnects, depending on an organization's traffic isolation and traffic management requirements. The simplest approach is to use the Cisco Nexus platform as the demarcation point for traffic isolation using trunk connections to forward traffic to the different tenant networks. Although the VLANs cross a shared set of connections, traffic is not forwarded between VLANs unless a device external to the Vblock System is configured. The Cisco Nexus platform switches have a number of reserved ports that can be used for connectivity to different networks, which does not affect Vblock System scalability.

Another approach for traffic isolation close to the server is to configure trunk connections from the Cisco UCS to the tenant networks using discrete uplink connections per network. However, this approach requires that each additional network have connections at the Cisco UCS 6200 Series. This reduces both the number of available ports for Cisco UCS Blade Server Chassis connections and the number of servers the Vblock System can support.

(19)

Disjoint layer 2 network upstream

Dedicated port channels

Standard Vblock Systems have a single port channel for each chassis assigned to eight physical ports per chassis. Implementing the VCE trusted multi-tenancy approach, dedicated port channels can be assigned for each tenant, ensuring that the tenant's network traffic flows across one physical path as compared to a single port channel in the standard Vblock System network architecture.

Storage separation

The main objective of storage separation is to provide an isolated storage environment where no tenant can access another tenant's data. This principle includes but is not limited to separation of data at rest and separation of data in motion.

(20)

Dedicated disk pools

Although it is not possible to partition a storage array cache, it is possible to maintain separation within the overall storage array cache by using separate physical disk pools for different tenants or a dedicated VSAN for a tenant. The VCE trusted multi-tenancy approach addresses the concerns of secure data separation by providing a mechanism to isolate tenants at one or more layers of the infrastructure.

For example, on the EMC VNX 5400 in a default Vblock System configuration, all disks are assigned to a single tenant and there are only two VSANs for redundancy. However, in the VCE trusted multi-tenancy approach, the disks can be split between tenants, each tenant having its own dedicated disks, VSAN, and physical ports on the controller.

Dedicated disk array enclosures

EMC storage arrays offer resource isolation and secure segmentation at the storage layer. An organization can choose the dedicated location of tenant data on the storage array down to the disk array enclosures.

Compute separation

In standard Vblock System solutions, compute resources are aggregated into a single layer cluster. All local users of the single tenant share resources, as the standard configuration has one tenant. Physical cables carry both network and storage traffic and connect to the fabric interconnects.

VCE trusted multi-tenancy separation allows for each tenant to have dedicated compute resources. The following controls are available for physical compute separation:

 Dedicated blades

 Dedicated Cisco UCS chassis

 Dedicated Cisco UCS domains

Dedicated blades

Standard Vblock Systems configure all physical blades for a single tenant. With the VCE trusted multi-tenancy approach, dedicated physical blades can be assigned to each tenant. For example, the Cisco UCS chassis could be divided to serve four tenants, each of which could be assigned two blades per Cisco UCS chassis. Alternatively, each tenant could be assigned a dedicated physical port in which all traffic for that particular UCS blade resides on that port and cable.

(21)

Dedicated Cisco UCS chassis

Organizations with higher bandwidth and compute requirements can achieve multi-tenancy by dedicating a complete Cisco UCS chassis to a single tenant. This approach completely isolates each tenant’s traffic, even at chassis level, as every chassis carries traffic belonging to only one tenant. From blade level through input/output module, each chassis forwards and computes the traffic from the same tenant, resulting in a secure environment. Each dedicated chassis network and fibre channel traffic splits into separate network at the Cisco UCS fabric interconnect.

Dedicated Cisco UCS domains

Vblock Systems support multiple Cisco UCS domains. Although not required for secure operation, it is possible to isolate tenant processing and UCS administrative function to a completely separate fabric interconnect pair. In a standard solution design, upstream communications to a shared Cisco Nexus switch pair would continue to leverage logical separation.

Management and physical separation

Physical segmentation can introduce barriers to the flow of management traffic. It might be necessary to create additional instances of management tools, even when the tools are normally multi-tenant capable, due to the thoroughness of the separation. Technical necessity is not the only driver for management isolation. For many sites implementing physical segmentation elements, even when the management tools are capable of supporting multi-tenancy and the architecture is not an impediment to their use, it is not unusual for tenant-facing tools in particular to be implemented with discrete instances per tenant. This is less consistent in environments relying solely on logical segmentation.

(22)

Logical segmentation

Logical segmentation refers to the effective separation and isolation of shared virtual compute, storage, and network resources between multiple tenants.

Organizations can achieve segmentation as described in the Physical Segmentation section of this document; however, there are some shared components that cannot be physically partitioned or for which dedicating physical resources is not a viable business proposition. This section introduces strategies for logically separating these resources.

VCE’s multi-tenancy approach provides options for multiple control points to isolate tenant resources. When combined with a comprehensive security program, these controls provide an effective foundation for tenant protection. Although each tenant might have access to different amounts of network, compute, and storage resources in the shared infrastructure, a tenant sees only those resources allocated to them. The VCE multi-tenant architecture for logical segmentation provides:

 Shared resources to increase usage

 Service density and rapid elasticity; easier growth and scale using standard infrastructures

 Secure isolation of resources and data to meet tenant security requirements

 Reduced service costs and expenses

 Improved predictably around planning around capacity and workloads

 Service assurance and faster updates

 Appropriate data separation under most circumstances; use physical segmentation options for the remaining scenarios

Consider the following design decision points when architecting multi-tenant segmentation:

 Network separation

 Storage separation

 Compute separation

 Virtualization

Network separation

Multi-tenancy concerns must be addressed at multiple levels within the Vblock System network infrastructure. Methods available to enforce network separation include security zoning and VLANs. Each tenant requires logical resource separation and/or path isolation at the network layer. VCE’s trusted multi-tenancy approach provides the same degree of tenant isolation as a dedicated infrastructure, where each tenant receives appropriate separation, controls, and auditing.

(23)

 Virtual Machine Fabric Extender (VM-FEX)

 Virtual Device Context (VDC)

Note: Virtual firewalls and other traditional network security controls are addressed later in this document.

Virtual local area networks

VLANs provide a layer 2 option to scale virtual machine connectivity, providing multi-tenant isolation and application tier separation. By default, a VLAN does not allow direct communications between virtual machines on different VLANs and all VLAN traffic must cross a router or firewall, which can be used to enforce

connectivity policies. This provides a good degree of isolation between the virtual machines and allows for policy enforcement. It is a simple and common way to isolate network traffic across the layer 2 domains and shared links throughout the infrastructure.

In general, Vblock Systems use two types of VLANs:

VLAN Description

Routed  Includes management VLAN, virtual machine VLAN, and data VLAN  Traffic passes through layer 2 trunks and is routed to the external network

Internal  Carries VMkernel traffic such as VMware vMotion, service console, network file system (NFS), and high availability

 Vblock Systems explicitly do not allow these VLANs across the trunks

VCE’s trusted multi-tenancy approach is to associate each tenant’s resources with a different VLAN to ensure that the management, tenant, and Vblock System internal VLANs are isolated. VCE recommends always separating data and management VLANs. Since significant isolation can result in quick exhaustion of VLAN resources, the best practice is to allocate appropriately sized contiguous ranges with the flexibility to reallocate ranges if required.

Virtual Machine Fabric Extender

VM-FEX collapses virtual switches and physical networks into a single infrastructure. VM-FEX is a hardware-based alternative to the Cisco Nexus 1000V switch that allows separation of traffic from different servers through network management domain separation from server or virtual machine management domain. VM-FEX allows a virtual machine to bypass the hypervisor and send network traffic directly to a vNIC on the Cisco virtual interface card (VIC). All network switching and processing is redirected from the hypervisor to an external network switch, reducing overhead and freeing up the server CPU for application workloads. This is ideal for applications with high packet rates.

VM-FEX does not support N-Port Virtualizer (NPV), which would allow a per-virtual machine virtual host bus adapter (vHBA) and relies on VMFS for storage access. Instead, use VMware vSphere Raw Device Mode (RDM) with VM-FEX for a per-virtual machine vHBA identity.

(24)

Organizations planning to implement a VM-FEX solution in VMDirectPath mode to bypass the hypervisor entirely should consider that they are limited to the number of devices that ESXi can support per host; only 8 VMDirectPath PCIe devices per host and 16 per virtual machine per VMware vSphere documentation.

Virtual Device Context

Use a Virtual Device Context (VDC) to logically separate an optional Cisco Nexus 7000 switch into multiple device contexts (virtual switches) to provide management separation, change domain isolation, and provide isolated VLAN and overlapping IP address support. A Virtual Device Context can contain its own unique and independent set of VLANs. Each Virtual Device Context can be assigned to its own physical ports, allowing for virtualization of the hardware data plane as well. The Vblock System 300, Vblock System 500, and Vblock System 700 families support Virtual Device Context.

The Cisco Nexus 7000 switch natively supports four Virtual Device Contexts. A maximum of eight VDC can be supported with the Nexus 7000 Series Supervisor 2 Enhanced (N7K-SUP2E=) and the Cisco Nexus 7000 Incremental VDC license (N7K-VDC1-K9).

Storage separation

At times it is necessary to ensure that a specific dataset does not share spindles with any other dataset. This separation might be required between tenants or even within a single tenant’s dataset for organizations using the same shared service requirements. With VCE’s trusted multi-tenancy design approach, multiple features can be combined with standard security methods such as SAN zoning and Ethernet VLAN to separate, control, and manage storage resources among an infrastructure’s tenants.

The following logical controls are available for storage separation:

 VSAN

 Zoning

 Virtual Data Mover (VDM)

 Virtual Machine File System Management (VMFS)

 LUN masking

Virtual storage area networks

VSAN is a separation mechanism similar to the VLAN enabled by the Cisco MDS storage area network (SAN) implementation. VSANs enable logical separation of large groups of fabrics at no additional hardware cost. Fibre channel services are fully replicated for each new VSAN after setting up the VSAN profile to ensure any failures from fabric changes are limited to the impacted VSAN. Security is improved because the VSAN independence minimizes the total system's vulnerability.

(25)

VSAN physical topology example

Zoning

SAN zoning isolates resources on a per pWWN basis. Resources can communicate only with other resources in the same zone as the initiating HBA/pWWN. Incorporating zoning in a multi-tenant infrastructure restricts visibility and connectivity between devices connected to a common fibre channel SAN, preventing data leakage between zones. In Vblock Systems, Cisco MDS zoning occurs by default.

Virtual Data Mover

Virtual Data Mover (VDM) is an EMC VNX feature that allows CIFS servers and their associated environment to be grouped into virtual containers. Virtual Data Mover isolates CIFS and/or NFS servers to provide a higher level of security. VLAN and file system mounts for different home directories can be separated, isolating home directory databases and their associated users. Virtual Data Mover also allows the replication and movement of a CIFS and/or NFS environment into another local or remote data mover.

LUN masking

Logical Unit Number (LUN) masking is an authorization process that restricts storage LUN access to specific resources/hosts on a shared SAN. LUN masking is implemented mainly at host bus adapter (HBA) level but in Vblock Systems it can be implemented at the storage controller level. As the controller itself enforces the

(26)

Virtual machine file system management

VMware uses a cluster file system called a virtual machine file system (VMFS). An ESXi host associates a VMFS volume that is made up of a larger logical unit. The virtual machine disk (VMDK) sub-directory in the VMFS volume stores each virtual machine directory. The VMFS volume locks those files to prevent updating by other ESXi servers. One VMDK directory is associated with a single virtual machine and multiple virtual machines cannot access the same VMDK directory within the VMFS volume, thereby isolating each tenant's VMDK, snapshots, and virtual machine files. VMFS reinforces the effects of the tenant Datastore isolation from the zoning mechanisms and LUN masking within the SAN at the file system level, serving to limit the effect of virtual machine-based exploits or inadvertent disk corruption.

Compute separation

Virtualization introduces new security concerns to the traditional data center where security policies are implemented at the physical level. When multiple logical servers belonging to a variety of tenants exist on a single physical server—or single compute domain—it must be possible to use logical mechanisms to isolate the logical servers belonging to each tenant.

Secure separation at UCS blade level

VCE’s trusted multi-tenancy approach enables organizations to achieve secure separation at the compute layer by managing multi-tenancy concerns at multiple levels, including the CPU, the Cisco UCS server infrastructure, and VMware solution elements.

(27)

The following logical controls are available for compute separation:

 Cisco UCS service profiles

 Cisco UCS organizations

 VLANs

 Virtual NICs (vNIC)

Cisco UCS service profiles

Use Cisco UCS service profiles to ensure secure separation at the compute layer. The UCS architecture presents hardware in a stateless manner that is completely transparent to the operating system and the applications it runs. A service profile creates a hardware overlay containing specific information that identifies the particular “server”: MAC addresses, WWN values, UUID, BIOS, and firmware versions.

VCE’s trusted multi-tenancy uses server roles to segregate and classify Cisco UCS blade servers, helping to identify and associate specific service profiles. A sample role configuration is shown below:

Server role Description

Management Meant only for cloud management or any type of service provider infrastructure workload Dedicated Meant for use with servers—presumably, but not necessarily, in clusters—dedicated to individual

tenants

Mixed Meant for use with servers in shared resource clusters hosting multiple tenants

Cisco UCS organizations

Organizations are a multi-tenant construct within Cisco UCS that act as an attachment point for policies and hardware. Each organization can have its own pool of resources including resource pools, policies, and service profiles. Cisco UCS organizations are hierarchical with root as the top-level organization. System-wide policies and pools in root are available to all organizations in the system. Any policies and pools created in other organizations are available only to organizations below it in the same hierarchy.

Virtual local area networks

VLAN separation is essential in multi-tenant architecture. Consider functional uses of VLANs with appropriate security to provide operational stability and control.

Cisco UCS manager, by default, allows configuring service profiles to access any upstream VLAN independent of the service profile's organization. Define VLAN permissions and VLAN group permissions consistent with security practices to restrict the VLANs that service profiles can access. Cisco UCS provides the capability to restrict access to VLANs based on user-configured VLAN permissions and the organization

(28)

Cisco UCS Virtual Interface Card

Instead of using physical NICs per industry best practice, VCE’s multi-tenancy approach uses multiple virtual interface cards (VICs) to separate tenant traffic. VICs deliver 256 virtual interfaces, allowing the creation of virtual adapters and their mapping to unique virtual machines and VMkernel interfaces within the hypervisor. This approach provides logical separation of tenant data traffic from back-end management traffic and visibility to enable true workload mobility in virtualized environments. Cisco UCS VIC enables a policy-based stateless agile server infrastructure to provide acceleration for the operational modes introduced by server virtualization, high performance, simplified management, and network visibility.

Virtualization

Virtualization occurs when software creates application-hosting environments that provide logical boundaries between tenants. With Vblock Systems, network virtualization provides a single common network

infrastructure to carry traffic from multiple tenants; it also ensures the tenant has visibility to only its own traffic and helps maintain traffic separation. Server virtualization enables more applications run on fewer compute resources.

Note: This document addresses storage virtualization earlier in this section.

In a multi-tenant infrastructure, the administrators of one tenant should not be able to modify the assets or resources of any other tenant. In a default Vblock System configuration, the virtualization layer consists of a single tenant in a single data center and it is typically not intended to divide up dedicated resources, traffic, or workloads for multi-tenancy. Using VCE’s trusted multi-tenancy approach, assign each host to its own group within the VMware vCenter Server and enforce policies so that administrators from one tenant cannot access hosts or virtual machines belonging to another tenant. This also ensures that traffic remains separated between tenants.

Consider the following when designing multi-tenant separation:

 Cisco Nexus 1000V Switch

 VMware vSphere

Vblock Systems can also be deployed with Cisco Application Centric Infrastructure (ACI) and VMware NSX.

Cisco Nexus 1000V Switch

The Cisco Nexus 1000V distributed virtual switch acts as the virtual network access layer for virtual machines. Edge LAN policies such as quality of service marking and virtual network interface card (vNIC) access control lists (ACL) are implemented at this layer in Cisco Nexus 1000V port profiles. The virtual machine traffic for tenants traverses the network through different uplink port profiles, where port security, rate limiting, and quality of service apply to guarantee secure separation and assurance. The following table describes Cisco Nexus 1000V Switch features:

(29)

Feature Description

Port profiling Provides seamless network connectivity across the Cisco UCS domain and ESXi cluster. Port profiles extend VLAN separation to virtual machines in flexible ways. It provides separation of virtual machine at the port level and can also be used to define configuration options and security and service level characteristics.

MAC pinning Provides more granular load-balancing methods and redundancy. Use port profile definitions to pin virtual machine NIC to an uplink path. An administrator defines the preferred uplink path to use. If the uplinks fail, notification packets inform upstream switches of the new path required to reach destination virtual machines. Based on these notifications, Cisco UCS fabric interconnect updates its MAC address tables and sends gratuitous ARP messages on the uplink ports so that the Vblock System access layer network can learn the new path.

Virtual Supervisor Module (VSM)

Controls multiple Virtual Ethernet Modules (VEM) as a single logical modular switch; VSM supports multiple VEM running in software inside the physical servers. Define VSM in pairs for high

availability, one as the primary module and the other as secondary. The two VSMs run in

active/backup mode to provide high availability switch management. Since the Nexus 1000V VSM is not in the data path, even if both VSMs are down there is no impact on VEM and traffic continues to be forwarded. This helps ensure high availability if even one VMware ESXi server fails.

VMware vSphere

VMware vSphere helps optimize IT service delivery and deliver the highest levels of application service agreements with the lowest total cost per application workload by decoupling critical applications from the underlying. The following table describes some features of VMware vSphere:

Feature Description Cluster file system

management

Creates a unique virtual machine disk (VMDK) per virtual machine. This ensures that multiple machines cannot access the same VMDK sub-directory within the virtual machine file system (VMFS) volume, thus isolating one tenant's VMDK from another.

Hypervisor based virtualization

Enables the creation of virtual machines on physical servers by logically abstracting the server environment in terms of CPU, memory, and network touch points into multiple virtual software containers.

Cisco Cloud Services Router 1000V

The Cisco Cloud Services Router (CSR) 1000V is an optional virtual router running the Cisco IOS-XE

operating system. It can run many L3 features commonly used in service provider solutions, including support for multiple hypervisors, MPLS, and multiple types of VPN. The Cisco CSR 1000V is an edge-based solution that typically serves the needs of a single tenant. Its place in the network allows it visibility to bidirectional traffic entering and exiting a cloud environment. This placement allows the CSR 1000V to provide deep packet inspection for all cloud environment traffic.

(30)

Feature Description

Application visibility Provides application-level classification, monitoring, and traffic control. It uses network-based application recognition (NBAR) v2 to dive deep into a packet to gain insight into application level usage and traffic patterns. This enables a service provider to proactively identify security threats associated with application traffic anomalies, provide application usage metrics for use in billing, and automatically set different QOS priorities based on application type. Virtual private network Supports IPSEC and route-based VPNs. It can be used to terminate a full mesh VPN, which

enables an organization to securely interconnect private/ public clouds via VPN terminating at the CSR router deployed at the hypervisor edge. Organizations can offer multi-datacenter services so tenants can load balance between systems located at different locations hosted by the ISP. The CSR 1000V can host a client-based VPN, which allows individual users to securely connect for management and/or other purposes.

MPLS support Serves as either a customer-edge or provider-edge router. Services providers can use it to offer customers end-to-end managed connectivity. Providers can increase tenant scale by extending the MPLS WAN deeper into the cloud.

Multicast routing Supports multicast routing and zone-based firewall, providing an advantage in a multi tenancy environment requiring a perimeter firewall to isolate tenant traffic, some of which could include MC feeds between zones. Without the Cisco CSR 1000v providing these services, MC traffic would be sent outside the Vblock System to an L3 device that supports MC routing and then back into the other zones. The CSR 1000V provides MC routing support and firewall functionality at the hypervisor edge. It can scale to support up to 10 separate zones with a single appliance.

(31)

Additional considerations

This section provides additional guidelines for achieving trusted multi-tenancy with Vblock Systems:

 Security and compliance

 Data-at-rest encryption

 Management models

 High availability and redundancy

 Scaling

Security and compliance

Network security

The following table lists tools available for network security:

Technologies and

products Description Access control list

(ACL)

Filters network traffic based on a list of access control entries. It is typically implemented to protect undesired access to a network but can also be used to separate applications and entities based on functional requirements. VCE’s multi-tenancy approach recommends applying ACL and/or VLAN access control list (VACL) in trusted multi-tenancy layer 2 and layer 3 to allow only the desired traffic for an expected destination within the same tenant domain or among different tenants. Due to its performance, scalability and operational complexity limitations, use ACL only as a hard boundary between tenants at the network layer.

Cisco Virtual Security Gateway

Provides multi-tenant access with granular, zone-based, and context-aware security policies. On Vblock Systems, use Cisco Virtual Security Gateway to fill the intra-tenant firewall functional role to filter communication between and within application tiers and from client to server.

Virtual Security Gateway uses the virtual network service path (vPath) technology embedded within the Cisco Nexus 1000V Virtual Ethernet Module (VEM), which offloads the switching logic directly to the host providing seamless interaction with other virtual machines. Virtual Security Gateway deployment on Vblock Systems can improve performance, as most packets are offloaded to the hypervisor and processed by the fast path. The Cisco Nexus 1000V vPath is tenant-aware, allowing for security policies to be implemented within and across multiple tenants.

(32)

Technologies and

products Description VMware vCloud

Networking and Security

Provides basic networking and security functionality for virtualized compute environments, including a broad range of services delivered through virtual appliances such as a virtual firewall, virtual private network (VPN), load balancing, NAT, DHCP and VXLAN-extended networks. VMware vCloud Networking and Security includes perimeter and interior firewalling capabilities. Note that these capabilities are purchased as part of the VMware vCloud Suite, not as individual components.

vCloud Networking and Security also offers a data security module that helps with endpoint security concerns by scanning virtual workloads for sensitive data, such as credit card information, and reports violations of country- and industry-specific regulations.

Cisco Virtual Security Gateway and VMware vCloud Networking and Security both have offerings capable of providing micro-segmentation, which is the implementation of firewall policies regulating east-west traffic between virtual machines—even those on the same VLAN, on the same host, in the same vApp, and so forth. These products are intended to prevent the spread of malware and other peer attacks and offer greater flexibility than traditional edge firewalls, which sit at layer-3 boundaries.

Endpoint security

With endpoint protection, the balance of responsibility between the tenant and platform administrator varies widely from deployment to deployment. Without active participation from platform administrators, it is challenging to achieve an efficient, high-security solution.

From the trusted multi-tenant deployment perspective, areas of interest within endpoint security include malware management (including anti-virus), host intrusion prevention and host firewalls, admission control, vulnerability scan enablement over complex topologies, and patch management. No single technology accomplishes all of these, but there are some common building blocks or suite providers that address major portions. For example, VMware vCloud Networking and Security Endpoint offloads antivirus processing to a secure virtual appliance supplied by VMware partners and:

 Improves consolidation ratios and performance by eliminating anti-virus storms.

 Automates anti-virus and anti-malware deployment and monitoring.

 Satisfies compliance and audit requirements with anti-virus and anti-malware activity logs.

Role-based access control (RBAC)

Role-based access control (RBAC) is a method of restricting or authorizing system access for users based on roles. A role can contain one or more system privileges where each privilege defines an administrative right to a certain object or object type. When assigned a role, a user inherits the capabilities of the privileges defined in that role. For example, a user assigned a server role can perform server-related operations, while a user assigned a network role manages network-related operations. The administrator role is the equivalent of the root user in a UNIX environment and has no restrictions in terms of privileges on resources.

(33)

Organizations can use RBAC to make efficient use of limited administrator resources. In an integrated management environment, RBAC enables network, storage, and server administrators to maintain responsibility and accountability for domain policies. Using Cisco UCS Manager, Vblock Systems support multi-tenant service providers and enterprise data centers serving internal clients as separate business entities. The system can be logically partitioned and allocated to different tenants to administer as their own. VMware also offers a full range of RBAC capabilities and multi-tenant management across a range of products; specific capabilities vary based on need and target market segments.

Data-at-rest encryption

Encrypting each tenant’s data is an excellent way to segment data and reduce the risks associated with lateral attacks in a storage system and some forms of hypervisor escapes. It can be challenging to implement, as there are many encryption options, each optimized towards very specific sets of expectations. If an

organization’s security model deviates from the expectations of its selected encryption, what might otherwise be an excellent, highly secure product may leave a large security hole or worse—a compliance finding. The following table lists encryption considerations for both tenant data and platform owners:

Owner Considerations

Tenant data  Whether encryption is needed

 Whether the sole objective is secure disposal of media (crypto-shredding)

 If encryption needs to conform to a certain standard, such as FIP 140-2 Level 1; whether there are any explicit standards requirements regarding key management or explicit requirements for using hardware security modules (HSMs)

 Whether to bring your own encryption or rely on what the platform administrator provides  Whether pre-boot unlocking of volumes is a requirement, an option, or an inconvenience Platform  Potential for orchestration and integration, including with business continuity/disaster recover

 Delegation of management and/or transparency – essentially how to keep out of the middle of encryption transactions

 Performance, scaling, granularity, licensing metrics, billing (or showback) units, and so forth A number of the encryption options complement one another. Consider a complex service provider scenario where the provider has a Vblock System 340 with a large EMC VNX array with controller-based encryption enabled. All data going in and out of the array is encrypted with the primary use case for the encryption being control of media/secure disposal. The provider offers an advanced encryption service in the form of encrypted LUNs that the tenant can also replicate to other sites. Looking at some of the tenants:

 Tenant A—uses the advanced encryption service

 Tenant B—does nothing, relying on the local encrypted volume

 Tenant C—turns on operating system encryption in virtual machines and manages them manually

Enabling Trusted Multi-Tenancy with Vblock Systems