How To Set Up A Virtual Network On Vsphere (Vsphere) On A 2Nd Generation Vmkernel (Vklan) On An Ipv5 Vklan (Vmklan)

Best Practices for Virtual Networking

Karim Elatov

Technical Support Engineer, GSS

Agenda

Best Practices for Virtual Networking

Troubleshooting Virtual Networks Virtual Network Overview

vSwitch Configurations

What’s New in vSphere 5.0 Tips & Tricks

Network Design Considerations

Virtual Switch

Virtual

Conventional access, distribution, core design Design with redundancy for enhanced availability

Under the covers, virtual network same as physical Access layer implemented as virtual switches

PhysicalSwitch PhysicalSwitch

Physical

Virtual Network Overview - Physical to Virtual

Physical

vNetwork Distributed Switch

Distributed:

1 or more per

“Datacenter”

- Expanded feature set - Private VLANs

- Bi-directional traffic shaping - Network vMotion

- Simplified management

Virtual Switch Options

Virtual Switch Model Details

vNetwork Standard Switch

Host based:

1 or more per ESX host

- Same as vSwitch in VI3

Cisco Nexus 1000V Distributed:

1 or more per

“Datacenter”

- Cisco Catalyst/Nexus feature set - Cisco NXOS cli

- Supports LACP

ESX Virtual Switch: Capabilities



> Uplink; No vSwitch <-> vSwitch or Uplink <-> Uplink

MAC address assigned to

vnic

VM0 VM1

vSwitch

Physical Switches

vSwitch

MAC a MAC b MAC c







Distributed Virtual Switch

vCenter vCenter

Standard vSwitch vNetwork & dvSwitch

Exist across 2 or more clustered hosts

•Provide similar functionality to vSwitches

•Reside on top of hidden vSwitches

vCenter owns the configuration of the dvSwitch

•Consistent host network configurations

Port Groups



• VLAN Assignment

• Security

• Traffic Shaping (limit egress traffic from VM)

• Failover & Load Balancing



• Bidirectional traffic shaping (ingress and egress)

• Network VMotion—network port state migrated upon VMotion

NIC Teaming for Availability and Load Sharing



• Availability—reduce exposure to single points of failure (NIC, uplink, physical switch)

• Load Sharing—distribute load over multiple uplinks (according to selected NIC teaming algorithm)

VM0 VM1

vSwitch

NIC Team



NIC Teaming Options

Explicit Failover Order

Highest order uplink from active list

Teamed ports in same L2 domain (BP: team over two physical

switches)

Best Practices:

•Originating Virtual PortID for VMs is the default, no extra configuration needed

•IP Hash, ensure that physical switch is properly configured for Etherchannel

*KB - ESX/ESXi host requirements for link aggregation (1001938)

*KB - Sample configuration of EtherChannel / Link aggregation with ESX/ESXi and Cisco/HP switches (1004048)

Name Algorithm—vmnic

chosen based upon:

Physical Network Considerations

Originating Virtual Port ID

vnic port Teamed ports in same L2 domain (BP: team over two physical

switches) Source MAC

Address

MAC seen on vnic Teamed ports in same L2 domain (BP: team over two physical

switches)

IP Hash* Hash(SrcIP, DstIP) Teamed ports configured in static 802.3ad “Etherchannel”

- no LACP (Nexus 1000v for LACP) - Needs MEC to span 2 switches

Cisco Nexus 1000v Overview



• Virtual Supervisor Module (VSM)

• Virtual Ethernet Module (VEM)

Things to remember:

• Virtual Ethernet Module (VEM)VSM uses external network fabric to communicate with VEMs

• VSM does not take part in forwarding packets

• VEM does not switch traffic to other VEM without an uplink

Cisco Nexus 1000v Modules

vCenter Server VMware ESX

Server 1

VMware vSwitch

VMware ESX

Server 2

VMware vSwitch

VMware ESX

Server 3

VMware vSwitch

VM

#1

VM

#4 VM

#3 VM

#2

VM

#5

VM

#8 VM

#7 VM

#6

VM

#9

VM

#12 VM

#11 VM

#10

Nexus 1000V

VSM

VEM Nexus 1000V vDS VEM VEM

Virtual Supervisor Module (VSM)

• Virtual or Physical appliance running Cisco OS (supports HA)

• Performs management, monitoring, &

configuration

• Tight integration with VMware Virtual Center

Virtual Ethernet Module (VEM)

• Enables advanced networking capability on the hypervisor

• Provides each VM with dedicated

“switch port”

• Collection of VEMs = 1 DVS

Cisco Nexus 1000V Enables:

• Policy Based VM Connectivity

• Mobility of Network & Security Properties

• Non-Disruptive Operational Model

vSwitch Configurations

Best Practices for Virtual Networking

Troubleshooting Virtual Networks Virtual Network Overview

vSwitch Configurations

What’s New in vSphere 5.0 Tips & Tricks

Network Design Considerations

Cisco ‘show run’ and ‘show tech-support’

The following is a Cisco EtherChannel sample configuration:

interface Port-channel1 switchport

switchport access vlan 100 switchport mode access no ip address

!

interface GigabitEthernet1/1 switchport

switchport access vlan 100 switchport mode access no ip address

channel-group 1 mode on

!

Obtain configuration of a Cisco router or switch

•Run commands in priviliged EXEC mode

•’show run’

•‘show tech-support’

KB - Troubleshooting network issues with the Cisco show tech-support command (1015437)

Traffic Types on a Virtual Network

Virtual Machine Traffic

• Traffic sourced and received from virtual machine(s)

• Isolate from each other based on service level vMotion Traffic

• Traffic sent when moving a virtual machine from one ESX host to another

• Should be isolated Management Traffic

• Should be isolated from VM traffic (one or two Service Consoles)

• If VMware HA is enabled, includes heartbeats

IP Storage Traffic—NFS and/or iSCSI via vmkernel interface

• Should be isolated from other traffic types Fault Tolerance (FT) Logging Traffic

Traffic Types on a Virtual Network, cont.



vMotion production

virtual switch

Service console/VMK Interface

virtual machines vMotion 106

storage 107

mgmt

108 management virtual switch

production management

storage

VLAN Tagging Options

vSwitch

Physical Switch

VST – Virtual Switch Tagging

VLAN Tags applied in

vSwitch VLAN

assigned in Port Group policy

vSwitch

Physical Switch

EST – External Switch Tagging

vSwitch

Physical Switch

VGT – Virtual Guest Tagging

VLAN Tags applied in

Guest

PortGroup set to VLAN

“4095”

DVS Support for Private VLAN (PVLAN)



• Between VMs on the same VLAN or network segment



• Community

• VMs can communicate with VMs on Community and Promiscuous

• Isolated

• VMs can only communicate with VMs on the Promiscuous

• Promiscuous

• VMs can communicate with all VMs

KB - Private VLAN (PVLAN) on vNetwork Distributed Switch - Concept Overview (1010691)

Allow devices to share the same IP subnet while being Layer 2 Isolated Benefits:

•Employ Larger subnets (advantageous to hosting environments)

•Reduce Management Overhead

application server Web

server

database server

email server

document server

isolated PVLAN

isolated PVLAN community PVLAN

DMZ network

router in promiscuous PVLAN

W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B

Distributed Virtual Switch

PG PG PG PG PG PG PG PG PG PG PG PG

TOTAL COST: 12 VLANs (one per VM)

W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B

PG (with Isolated PVLAN)

PVLAN Cost Benefit

Link Aggregation

EtherChannel

•Port trunking between two to eight

•Active Fast Ethernet, Gigabit Ethernet, or 10 Gigabit Ethernet ports

KB ESX/ESXi host requirements for link aggregation (1001938)

LACP (one of the implementations included in IEEE 802.3ad)

•Link Aggregation Control Protocol (LACP)

•Control the bundling of several physical ports into a single logical channel

•Only supported on Nexus 1000v

EtherChannel vs. 802.3ad

•EtherChannel is Cisco proprietary and 802.3ad is an open standard Note: ESX implements 802.3ad Static Mode Link Aggregation

Sample Link Aggregation Configuration

Failover Configurations

Figure — Using beacons to detect upstream network connection failures.

KB - What is beacon probing? (1005577)

Beacon Probing sends out and listens for beacon probes

•Broadcast frames (ethertype 0x05ff)

Beacon Probing Best Practice

•Use at least 3 NICs for triangulation

•If only 2 NICs in team, can’t determine link failed

•Leads to shotgun mode results

Link Status relies solely on the network adapter link state

•Cannot detect configuration errors

•Spanning Tree Blocking

•Incorrect VLAN

•Physical switch cable pulls

Spanning Tree Protocol (STP) Considerations



• Physical links put in “blocking” state to construct loop-free tree

VM0 VM1

vSwitch

Physical Switches

MAC a MAC b

Switches sending vSwitch drops

BPDUs

Recommendations for Physical Network Config:

1. Leave Spanning Tree enabled on physical network and ESX facing ports (i.e. leave it as is!)

2. Use “portfast” or “portfast trunk” on ESX facing



• ESX Uplinks will not block, always active (full use of all links)

Tips & Tricks

Best Practices for Virtual Networking

Troubleshooting Virtual Networks Virtual Network Overview

vSwitch Configurations

What’s New in vSphere 5.0 Tips & Tricks

Network Design Considerations

Tips & Tricks



• Dynamically balance network load over available uplinks

• Triggered by ingress or egress congestion at 75% mean utilization over a 30 second period

• Configure on DVS via “Route based on physical NIC load”

*LBT is not available on the Standard vSwitch (DVS feature for ingress/egress traffic shaping)



• DVS software scheduler to isolate and prioritize specific traffic types

Tips & Tricks

Tip #5 – Link aggregation is never supported on disparate trunked switches – Use VSS with MEC. (KB 1001938 & KB 1027731)

Tip #4 - Beacon Probing and IP Hash DO NOT MIX (duplicate packets and port flapping) (KB 1017612 & KB 1012819)

Tip #1 – After physical to virtual migration, the VM MAC address can be changed for Licensed Applications relying on physical MAC address. (KB 1008473)

Tip #2 – NLB Multicast needs physical switch Manual ARP resolution of NLB cluster. (KB 1006525)

Tip #3 – Cisco Discovery Protocol (CDP) gives switchport configuration information useful for troubleshooting (KB 1007069)

Using 10GigE

 2x 10GigE common/expected

• 10GigE CNAs or NICs

 Possible Deployment Method

• Active/Standby on all Portgroups

• VMs “sticky” to one vmnic

• SC/vmk ports sticky to other

• Use Ingress Traffic Shaping to control traffic type per

vSwitch

iSCSI NFS ^VMotion FT SC

FCoE FCoE

SC#2

FCoE 10

Gbps 10GE

10GE

Ingress (into switch) traffic shaping policy control on Port Group

1-2G Low b/w

High b/w Variable/high

b/w 2Gbps+

Tips & Tricks

Troubleshooting Virtual Networks

Best Practices for Virtual Networking

Troubleshooting Virtual Networks Virtual Network Overview

vSwitch Configurations

What’s New in vSphere 5.0 Tips & Tricks

Network Design Considerations

Network Troubleshooting Tips



• Physical NICs

• Virtual Switch

• Virtual NICs

• Physical Network



• vSphere Client

• Command Line Utilities

• ESXTOP

• Third party tools

• Ping and Traceroute

Capturing Traffic

ESXi uses tcpdump-uw (KB 1031186)

vSwitch must be in Promiscuous Mode (KBs 1004099 & 1002934)

Best Practice: create a new management interface for this purpose

What’s New in vSphere 5.0

Best Practices for Virtual Networking

Troubleshooting Virtual Networks Virtual Network Overview

vSwitch Configurations

What’s New in vSphere 5.0 Tips & Tricks

Network Design Considerations

What’s New in vSphere 5?

Monitor and troubleshoot virtual infrastructure traffic

• NetFlow V5

• Port mirror (SPAN)

• LLDP (standard based link layer discovery protocol) support simplifies the network configuration and management in non-Cisco switch environment.

Enhancements to the network I/O control (NIOC)

• Ability to create User-defined resource pool

• Support for vSphere replication traffic type; a new system traffic type that carries replication traffic from one host to another.

• Support for IEEE 802.1p tagging

What’s New in VMware vSphere 5.0 Networking Technical Whitepaper

Network Design Considerations

Best Practices for Virtual Networking

Troubleshooting Virtual Networks Virtual Network Overview

vSwitch Configurations

What’s New in vSphere 5.0 Tips & Tricks

Network Design Considerations

Network Design Considerations



performance and availability but maintain isolation between the various traffic types

(e.g. VM traffic, VMotion, and Management)?

• 2 NIC minimum for availability, 4+ NICs per server preferred

• 802.1Q VLAN trunking highly recommended for logical scaling (particularly with low NIC port servers)

• Examples are meant as guidance and do not represent strict requirements in terms of design

• Understand your requirements and resultant traffic types and design accordingly

• Starting point depends on:

• Number of available physical ports on server

• Required traffic types



• Team both NIC ports

• Create one virtual switch

• Create three port groups:

• Use Active/Standby policy for each portgroup

• Portgroup1: Service Console (SC)

• Portgroup2: VMotion

• Portgroup3: VM traffic

• Use VLAN trunking

• Trunk VLANs 10, 20,

vmnic0 vmnic1

Portgroup3 VLAN 30

VLAN Trunks (VLANs 10, 20, 30)

Portgroup1 VLAN 10

Portgroup2 VLAN 20

SC ^vmkernel

vSwitch

Example 1: Blade Server with 2 NIC Ports



• Create two virtual switches

• Team two NICs to each vSwitch

• vSwitch0 (use active/standby for each portgroup):

• Portgroup1: Service Console (SC)

• Portgroup2: VMotion

• vSwitch1 (use Originating Virtual PortID)

• Portgroup3: VM traffic #1

• Portgroup4: VM traffic #2

• Use VLAN trunking

• vmnic1 and vmnic3: Trunk VLANs 10, 20

• vmnic0 and vmnic2: Trunk VLANs 30, 40

VLANs 10, 20 vSwitch0 vSwitch1

VLANs 30, 40

Note: Team over dvUplinks with vDS

Active Standby

SC ^vmkernel

Portgroup1 VLAN 10

Portgroup2 VLAN 20

vmnic0 vmnic1 vmnic3

Portgroup3 VLAN 30 Portgroup4

VLAN 40

vmnic2

Example 2: Server with 4 NIC Ports



• Create one virtual switch

• Create two NIC teams

• vSwitch0 (use active/standby for portgroups 1 & 2):

• Portgroup1: Service Console (SC)

• Portgroup2: Vmotion

• Use Originating Virtual PortID for Portgroups 3 & 4

• Portgroup3: VM traffic #1

• Portgroup4: VM traffic #2

vmnic0

SC ^vmkernel

vmnic1

vmnic2 vmnic3

Portgroup1 VLAN 10

Portgroup2 VLAN 20 Portgroup3

VLAN 30 Portgroup4

VLAN 40

VLANs 10, 20 VLANs

30, 40

vSwitch0

Example 3: Server with 4 NIC Ports (Slight Variation)

Questions