How to Deploy a Nexus 1000v lab with VMware Workstation.
By Robert Burns – CCIE Data Center #37856
*Details of third party OS & application installation is beyond the scope of this guide.
*This guide will use a single flat private network. Advanced options can include separate (VLAN) networks.
Workstation Requirements: -Java installed
-Approx 8GB of memory -min 50GB of available storage -VMware Workstation 9 installed
-ESXi 5.x and vCenter Appliance ISO/OVA available Example Host List
Host IP VM Name Description
192.168.0.10 VMware Workstation
RHEL62-Test-1 Redhat test VM
192.168.19. vCenter-App vCenter Appliance
192.168.19.129 ESX5-Nested-1 Nested ESX 5.1 (VM 1)
192.168.19.130 ESX5-Nested-2 Nested ESX 5.1 (VM 2)
192.168.19.131 N1000v VSM Management IP
1. Nested ESX Installation
a. VMware workstation should be previously installed and working. You’ll want at least 75 GB of storage on your workstation for VMs.
b. Create a new Virtual Machine, using the Custom (advanced) option – Next. c. Set the HW compatibility to Workstation 9.0 – Next.
d. Set the path to your ESXi 5.x installation ISO - Next e. Name your VM and set the location – Next
f. Set 2 processors, 1 core – Next
g. Configure at least 2GB memory. (Depends on total available memory in system) – Next h. Set networking to use “Host-Only”. We don’t need external access for our VMs. i. Keep LSI Logic (default) storage controller – Next
j. Create new virtual disk – Next k. SCSI (default) – Next
l. Set disk size (min. 10GB) or more if you plan to host VMs within the VMFS of Nested ESX host – Next.
m. Keep default disk file name - Next
n. Review configuration check to power on VM – Finish
You may see warnings like these if you haven’t enabled Intel VT-d and Intel Virtualization support in your workstation’s BIOS:
o. Repeat for at least one more host. 2. vCenter Appliance Installation
a. In Workstation, go to File – Open, then browse to the vCenter Appliance OVA file. b. Give the VM a name and click Import.
c. Power on the VC Appliance and begin the installation process
d. Before you power it on, change the Network Adaptor binding to use the private “Host-Only” network.
e. After installation completes, log into the VCA using the IP provided from the console screen. Default credentials are root/vmware.
f. Accept the EULA – Next
g. Select Configure with default settings – Next h. Review and click Start
i. Once complete you can access the web client from https://<vCenter_IP>/vsphere-client/
with the default credentials root/vmware.
j. Add your two nested ESX hosts to vCenter under their own cluster.
3. Deploy the 1000v VSM
a. Download the latest 1000v SW from CCO.
*For ESX 5.1 and later you must use 1000v version Nexus1000v.4.2.1.SV2.1.1 or later.
http://software.cisco.com/download/release.html?mdfid=282646785&flowid=3090&softwa reid=282088129&release=4.2%281%29SV2%281.1a%29&relind=AVAILABLE&rellifecycle=&r eltype=latest
b. Unzip the bundle and navigate to the Install_App folder & launch the application.
c. Start with the VSM Complete Installation – selecting Custom install.
d. Review the pre-reqs & click Next.
f. Enter the appropriate info for your VSM. Since I’m hosting the VSM pair on my single bare metal ESX host, I’ve used the same ESX Host IP twice. Click Next when complete. Notes:
- Recommend setting your Domain ID to something other than default 1. - I’m setting my Management VLAN to what my physical switches use for
their native VLAN.
- I chose to use L2 mode for simplicity versus L3, but either will work. - Do not migrate hosts at this time.
h. Next step will prompt to add additional Modules.
i. Select the hosts you wish to have the VEM agent installed. Click Next.
Note: This method requires VUM to be previously installed. If not, you’ll need to manually install the VEM agent vibs.
j. Review and then click Finish to proceed.
k. The Install App hopefully completed successfully for all hosts.
**If the VEM installation fails, it likely points to a problem with VMware Update Manager (VUM).
l. From the VI Client go to Home -> Inventory -> Networking and you should see your two new hosts as part of the 1000v DVS. Ensure you click on the 1000v DVS in the left pane.
m. SSH into the VSM and and check the modules
N1000v(config)# show mod
Mod Ports Module-Type Model Status
--- --- --- --- ---
1 0 Virtual Supervisor Module Nexus1000V active *
2 0 Virtual Supervisor Module Nexus1000V ha-standby
3 248 Virtual Ethernet Module NA ok
Mod Sw Hw
-- --- ---
1 4.2(1)SV2(1.1) 0.0
2 4.2(1)SV2(1.1) 0.0
3 4.2(1)SV2(1.1) VMware ESXi 5.1.0 Releasebuild-838463 (3.1)
4 4.2(1)SV2(1.1) VMware ESXi 5.1.0 Releasebuild-838463 (3.1)
Mod MAC-Address(es) Serial-Num
--- --- ---
1 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA
2 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA
3 02-00-0c-00-03-00 to 02-00-0c-00-03-80 NA
4 02-00-0c-00-04-00 to 02-00-0c-00-04-80 NA
Mod Server-IP Server-UUID Server-Name
--- --- --- ---
1 10.85.49.220 NA NA
2 10.85.49.220 NA NA
3 10.85.49.218 422954ef-1f4d-f096-2a47-4b64cd67b932 10.85.49.218
4 10.85.49.219 42299b1d-8226-47bc-f375-2432c7cbe87e 10.85.49.219
4. Migrate Test VM & Test Connectivity
a. Cold migrate (powered off) your Test VM to one the Nested ESX hosts. If you get any errors, you’ve likely done one of the following:
- Didn’t modify your Nested ESX VM to Virtual Machine Version 9 prior to install - Didn’t enable the “Expose NX/XD flag to guest” option in the Nested ESX VM.
b. Before we power it up we’re going to create a Port Profile for it on the 1000v.
N1000v(config)# port-profile type vethernet rhel-pp N1000v(config-port-prof)# switchport mode access N1000v(config-port-prof)# switchport access vlan 711 N1000v(config-port-prof)# state enabled
N1000v(config-port-prof)# no shut
N1000v(config-port-prof)# vmware port-group
c. Now change the virtual network binding of your test VM from the vSwitch to the 1000v port profile.
d. Power up your test VM, and verify the interface on the 1000v.
N1000v(config)# show interface virtual
--- Port Adapter Owner Mod Host
--- Veth1 Net Adapter 1 RHEL62-Test-1 3 10.85.49.218
N1000v(config)#
5. Advanced Configuration (optional)
a. Now that we have basic connectivity, let’s add the remaining uplinks to your Nested ESX VEM hosts. Select the host – Configuration – Networking – vSphere Distributed Switch tab – Manage Physical Adapters
b. Find the uplink port profile and click “Add NIC”. Add each of the 2 remaining NICs from each host.
c. Verify the uplinks on the 1000v. Your uplink port profile should be configured for mac pinning in which case you should see two new Port channels automatically created.
N1000v(config)# show int brief
--- Port VRF Status IP Address Speed MTU --- mgmt0 -- up 10.85.49.220 1000 1500 --- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # --- Eth3/2 711 eth trunk up none 1000 1 Eth3/3 711 eth trunk up none 1000 1 Eth3/4 711 eth trunk up none 1000 1 Eth4/2 711 eth trunk up none 1000 2 Eth4/3 711 eth trunk up none 1000 2 Eth4/4 711 eth trunk up none 1000 2 --- Port-channel VLAN Type Mode Status Reason Speed Protocol Interface
--- Po1 711 eth trunk up none a-1000(D) none Po2 711 eth trunk up none a-1000(D) none <snip>
6. Exercise - Determine which uplink your Test VM is utilizing
i. Find which module the VM is hosted by.
N1000v(config)# show int virtual
--- Port Adapter Owner Mod Host
--- Veth1 Net Adapter 1 RHEL62-Test-1 3 10.85.49.218
ii. Identify the Sub Group IDs of all uplinks on that host.
N1000v(config)# module vem 3 execute vemcmd show port
LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type 18 Eth3/2 UP UP FWD 305 1 vmnic1 19 Eth3/3 UP UP FWD 305 2 vmnic2 20 Eth3/4 UP UP FWD 305 3 vmnic3 49 Veth1 UP UP FWD 0 2 RHEL62-Test-1.eth0 305 Po1 UP UP FWD 0 <snip>
You can see from the output, SGID 1 = vmnic1, 2 = vmnic2 and 3 = vmnic3 iii. Find the VMs pinned Sub Group ID from the same ouput.
N1000v(config)# module vem 3 execute vemcmd show port
LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type 18 Eth3/2 UP UP FWD 305 1 vmnic1 19 Eth3/3 UP UP FWD 305 2 vmnic2 20 Eth3/4 UP UP FWD 305 3 vmnic3 49 Veth1 UP UP FWD 0 2 RHEL62-Test-1.eth0 305 Po1 UP UP FWD 0 <snip>
From this we can see that the Test VM is assigned to SGID 2, which will use vmnic2 for external communication.
7. Exercise – Force your Test VM to utilize vmnic1.
a. First determine what the SGID of vmnic1 is. From our previous output, this would be SGID “1”. b. Configure either the port profile or the individual interface to “prefer” this Sub Group.
N1000v(config)# port-profile rhel-pp N1000v(config-port-prof)# pinning id 1
c. Verify the change.
N1000v(config-port-prof)# module vem 3 execute vemcmd show port LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type 18 Eth3/2 UP UP FWD 305 1 vmnic1 19 Eth3/3 UP UP FWD 305 2 vmnic2 20 Eth3/4 UP UP FWD 305 3 vmnic3 49 Veth1 UP UP FWD 0 1 RHEL62-Test-1.eth0 305 Po1 UP UP FWD 0 <snip>
