Hyper-Cluster
By John D. Lambert, Microsoft SQL Server PFE http://blogs.technet.com/fort_sql• Introduction
• Install and Configure Hyper-V Host
• Install WSS08 as Hyper-V Guest
• Install Node1
• Install Node2
• Configure iSCSI Targets
• Configure Node1
• Configure Node2
• Install and Configure Cluster
•
Appendix A: Sample Network ChartIntroduction
By "Hyper-Cluster" I mean a complete cluster, including shared disk space, all virtualized on a single physical computer. A hyper-cluster is not a recommended configuration for a production system for many reasons, but it's great for practicing with cluster-enabled services. This document describes how I created a Windows Server 2008 R2-based Hyper-Cluster, using Windows Storage Server 2008's iSCSI for shared disks. This minimum configuration uses a host and 3 virtual servers:
•
Host (for Hyper-V, Active Directory/Domain Controller, DNS) • Storage Server (for shared disk space and iSNS)• Cluster Node 1 • Cluster Node 2
Once you've got the Hyper-V host running, turn on the Remote Desktop feature, and you can log into it from somewhere else on the planet to create or play with your hyper-cluster.
Please bear in mind that I'm not a Hyper-V expert, and this isn't an official white paper on the best way to do everything, it's just a basic record of what I did to get my Hyper-Cluster working. I had a bunch of folks ask me how I did it under Server 2008 (not R2) and I didn't have any notes from that effort, so this time I figured I'd jot a few things down in case anyone's interested. I’ll also point out that this is not a comprehensive step-by-step guide, as I skip lots of common details, like clicking past splash screens and common Hyper-V management tasks.
TERMS
Target vs. Initiator:
An iSCSI Target is a shared hard drive or group of drives, and they are shared by an iSCSI Target service. iSCSI Targets are used by iSCSI Initiators, and the initiators are also called iSCSI clients or iSCSI target clients. It's that last one that causes a lot of confusion.
An iSCSI Qualified Name (IQN) is a globally unique identifier for both targets and initiators. In my Hyper-Cluster, Windows Storage Server 2008 (WSS or WSS08) provides virtual disk space as iSCSI targets which are used by the iSCSI initiators in my cluster nodes. The cluster nodes see them as if they were local physical disks.
TROUBLE AREAS
WSS installs with a default admin password but doesn't tell you. Once it boots up for the first time, you have to log in as the administrator, but it never told you the default password, which is buried in the release-notes of the Tools CD on MSDN. I list it below, but just in case you're reading this document only because you're in your own desperate hunt for it, it's wSS2008!. I also stumbled on the WSS firewall because I expected that enabling the iSCSI Service rules (inbound and outbound) would cover all relevant traffic, especially since they seem to be very loose rules. It didn't, however, and it wasted a lot of my time before troubleshooting pinned down my communication problem to the WSS firewall, and I had to add an inbound rule for the iSCSI TCP ports. Also, even though you define a virtual network as private in Hyper-V's Virtual Network Manager, that doesn't mean your guest machines will see it as private. To the virtual guests, that's a real network, and the Windows firewall will consider it a "Public" network as opposed to "Private" or "Domain."
One area I suspect might trip up some folks is an unavailable network in WSS. This could happen for 2 reasons: you haven't assigned a network to the virtual machine; or you haven't installed the Hyper-V integration services. With Server 2008 R2 as your node OS, the integration services are automatic, but you still have to make a network available in the virtual machine settings. Similarly, it's really easy to get fouled up on which Local Area Connection is which and assign the wrong subnets, so you really need to match the MAC addresses as I point out later on.
Install and Configure Hyper-V Host
This isn't intended as a basic Hyper-V document, but I'm listing the O/S, roles, hardware, and virtual networks I used. If you're new to Hyper-V you should find basic instructions elsewhere, but the general idea is to install Windows Server 2008 R2, then install the roles, and then define the virtual networks in the Hyper-V Manager. And in case you missed it elsewhere, your
microprocessor has to support virtualization for Hyper-V to work.
The hardware listed is what I'm running this on, but you can run a Hyper-Cluster on a single hard drive, and I think you could do it with as little as 2GB RAM although performance will suffer. I did this once before using Server 2008 (not R2) as the host and cluster nodes using only 4GB of RAM and just the 140GB drive, but of course performance is better with more memory and if the load is split between 2 physical drives. With 2 drives, I put the host Hyper-V on the C: drive and everything else on the other drive.
Roles installed:
• Active Directory Certificate Services • Active Directory Domain Services • DNS Server
• Hyper-V Hardware:
• 8 GB RAM
• 140GB SATA drive • 500GB SATA drive Hyper-V Networks:
• Public Network (connects to external network card, allows management OS to share the adapter)
•
iSCSI Virtual Network (private)•
Heartbeat Virtual Network (private)Install WSS08 as Hyper-V Guest
1. From host Hyper-V Manager, create a new virtual machine.Name: WSS08
Location: E:\ProgramData\Microsoft\Windows\Hyper-V\
Memory: 512MB
Connection: Public Network
Virtual Hard Disk: E:\ProgramData\Microsoft\Windows\Hyper-V\WSS08.vhd, 20GB
Install OS from ISO:
en_windows_storage_server_2008_embedded_basic_standard_enterprise_wor kgroup_dvd_x64_x15-49574.iso
2. From host Hyper-V Manager, right-click on WSS08 (under the list of Virtual Machines) and choose Settings to begin adding shared disks.
a.
Click on the SCSI Controller, then click the Add button, then the New button. I created 3 virtual drives for WSS to share, using Fixed Size, and the default location: E:\Users\Public\Documents\Hyper-V\Virtual Hard Disks\, but you can put them in any drive and folder that the Hyper-V host can see. After adding one virtual drive, you have to click on SCSI Controller again before adding another one.• Quorum.vhd, 1GB • SqlData.vhd, 20GB • SqlLog.vhd, 2GB
Remember that because this instance of WSS08 is a virtual guest, it can only see the drives provided by Hyper-V. If you want more than a C: drive, you'll need to add or more SCSI drives via Hyper-V (up to 64 virtual drives). You only need one virtual disk for WSS08, because you also create virtual disks in the iSCSI Target service, but I use multiple virtual disks at this point because I'm trying to emulate how I would be configuring things if I had a big honkin' SAN. 3. From host Hyper-V Manager, open the Settings on WSS08 and add the iSCSI Network.
b. Choose the iSCSI Virtual Network and click OK. 4. From host Hyper-V Manager, turn on WSS08 and connect.
5.
Install Workgroup or higher edition by using a key for the edition you want to run. The Basic edition of WSS08 does NOT support iSCSI targeting. And if you want to play around with clustering the iSCSI target service, then you have to use the Enterprise edition of WSS.6.
Log in as admin with the default admin password, which is wSS2008!. If you're new to Hyper-V, click the little button on the far left side of the button bar to send a Ctrl-Alt-Delete to the guest operating system.)7. On the Hyper-V container, click the Action drop-down menu and click Insert Integration Services Setup Disk (on type Ctrl-I) and install the Hyper-V integration services. I had to do this even though the service is supposed to be included already in WSS 2008, because it wasn't running and I couldn't find it after a little poking around.
8. Change the IP address, mask, and gateway for the public network on WSS08, and the IP address and mask for the iSCSI network, according to the chart in Appendix A or your own requirements. When you have more than one virtual adapter to configure, you have to figure out which adapter goes with which virtual network. Here's how I did it:
a. Go into the Hyper-V Manager and open the Properties for the WSS08 virtual server, click on the network adapter that says "Public Network" below it, and look at the MAC address (even though it's grayed-out).
b. Back in WSS08, open the Network Connections and double-click on either of either of the Local Area Connections, then click the Details... button and see if the MAC address from the previous step matches the Physical Address. If it matches, this is the Local Area Connection for the Public Network, and it if doesn't match it's the LAC for the iSCSI network.
Area Connection and rename it to Public Network or iSCSI Network, then rename the other one. It's essential to get these straight; otherwise you'll put the wrong IP addresses in them.
d. For both the Public and iSCSI networks, right-click and choose Properties, then double-click on "Internet Protocol Version 4 (TCP/IPv4)", click on "Use the following IP address:" and enter the values from the table in Appendix A or whatever network values you need.
9. Change the computer name, add the server to the domain running in your Hyper-V host, and set the time zone.
10. From the host's Hyper-V Manager, change the Hyper-V settings for WSS08 to assign the DVD drive to the iSCSI CD (en_windows_storage_server_2008_iscsi_cd_x64_x86_x15-49563). You can do this whether WSS08 is on or off.
11.
Back inside WSS, open the DVD/CD to the x64 folder and launch iscsitarget.msi. Remember, the '.msi' is hidden unless you've change the default folder views. This adds "Microsoft iSCSI Software Target" to the Administrative Tools folder and a service with automatic startup that's also listed as "Microsoft iSCSI Software Target". The iscsitarget is the version for hosting an iSCSI session on WSS; the iscsitargetclient is for the cluster system to use the iscsitargetclient, but you won't need the client software since it's build into Windows Server 2008 (and Windows 7).12.
From the Server Manager (or from the Initial Configuration Tools window if you have that open), click on Features, then Add Features, and add "Internet Storage Name Server". This is also referred to as iSNS, and it adds "iSNS Server" to Administrative Tools. Though I got a previous hyper-cluster to communicate fine without it (no surprise since everything is on the same LAN segment), it's good practice to include it since it will probably be used in most production environments.13. Format the virtual shared disks so they can be seen by the iSCSI targeting service: a. Open Administrative Tools --> Computer Management, and click on Disk
Management (or whatever your favorite way to get there is).
b. Wait for the Initialize Disk window to open and click Okay to initialize the virtual SCSI disks you assigned to WSS08 from the Hyper-V Manager.
c. For each virtual SCSI disk:
i. Right-click on the graphical representation of each of these unformatted drives and chose New Simple Drive...
iii. Assign a drive letter and click Next. My only suggestion is to use Q for the quorum disk.
iv. Choose NTFS, the default allocation unit size (or align them if you know your physical drive's allocation unit size), assign a label to match the name you gave the underlying .vhd in the Hyper-V Manager (e.g. Quorum, SqlData, SqlLog), choose the Perform A Quck Format option, and click Next. 14. Configure the firewall.
a. Launch Server Manager, expand the Configuration node, then the Windows Firewall node, or go to Administrative Tools --> Windows Firewall.
b. Go to Inbound Rules, scroll down to "iSCSI Service (TCP-In)", right-click it and choose "Enable Rule."
c. Go to Outbound Rules and enable "iSCSI Service (TCP-Out)."
d. Now right-click Inbound Rules and choose New Rule..., Port, then Next.
e.
Leave TCP selected and type in "135, 3260" for local ports. f. Leave Allow The Connection selected and click Next.g. Leave all profiles checked. (WSS sees the iSCSI virtual network as a Public network.) h. Give it a descriptive name like "iSCSI TCP Ports" and click Finish.
It seems to me like it should be enough to just enable the built-in iSCSI service rules (both inbound and outbound), but that didn't work for me even though it's a pretty loose rule. UDP port 138 must also be open to your iSCSI clients, but that was open in a default rule on my installation. Of course, you could just turn off the firewall, but it's good training to learn how to get it right.
Now install Windows on the cluster nodes. That way, when we come back to WSS and configure the iSCSI targets, it'll be able to see the clients so we can allow our cluster node clients and only our cluster node clients to connect to our iSCSI targets. Limiting access to iSCSI targets is obviously necessary to keep the data secure.
Install Node1
15. From the host Hyper-V Manager, create a new virtual machine.
Name: Node1
Location: E:\ProgramData\Microsoft\Windows\Hyper-V\
Memory: 2048MB
Connection: Public Network
Virtual Hard Disk: E:\ProgramData\Microsoft\Windows\Hyper-V\Node1.vhd, 50GB
Install OS from ISO:
en_windows_server_2008_r2_standard_enterprise_datacenter_web_vl_build_x 64_dvd_x15-49574
16. From the host Hyper-V Manager, open the Settings on Node1 and add the iSCSI and heartbeat networks. Though only one network is required for a cluster, this represents a real-world scenario where one private network is used exclusively for iSCSI traffic, and another is used exclusively for a heartbeat.
a. Click on Add Hardware, Network Adapter, then the Add button.
b.
Choose the iSCSI Virtual Network and click OK. c. Repeat to add the Heartbeat Virtual Network.17. From the host Hyper-V Manager, turn on Node1 and connect.
18.
Choose Windows Server 2008 R2 Enterprise (Full Installation) x64 (Only Enterprise and DataCenter editions support clustering. Hyper-V integration services are included in the OS and don't need to be installed from the Hyper-V manager.)19. Change the Public Network IP address, gateway, and DNS server in the network settings. 20. Add an IP Address and mask for the iSCSI and Heartbeat networks. These networks don't
need a gateway or DNS server.
21. Change the server name and join the node to the domain. 22. Optional: Activate Windows and install all updates. 23. Run Administrative Tools --> iSCSI Initiator.
a. Click yes to start the service and set it to autostart. This also registers Node1 with iSNS.
b. Click the Discovery tab, then the Add Server... button. c. Type in WSS08 as the iSNS server name and click Okay. d. Click yes to unblock the iSCSI service in the firewall.
e. Click the cancel button to close the iSCSI Initiator Properties window or just leave it open for later.
Now we want to get Node 2 to the same state.
Install Node2
Installation is identical to Node1 except for the virtual machine name and server name.
Configure iSCSI Targets
24.
Back on WSS08, launch Administrative Tools --> Microsoft iSCSI Software Target to open the iSCSITarget window.25.
In the iSCSITarget window, right-click on Microsoft iSCSI Software Target and choose Properties.26. On the Network tab, uncheck everything except the IP address for the iSCSI Network (192.168.2.20 if you're following Appendix A).
27.
Click on the iSNS tab, click the Add button, and enter "WSS08" as the DNS name of the iSNS server and hit Enter, then OK to close the properties window.28.
Right-click iSCSI Targets and choose Create iSCSI Target.29. On the iSCSI Target Identification screen, type in a target name, which will be used by the iSCSI clients. I used "HyperClusterSharedDisks."
30.
On the iSCSI Initiators Identifiers screen, click the Advanced button, and for each node you're going to have, click the Add button, then the Browse button, and choose one of the iSCSI initiators listed. This tells the iSCSI target service which computers are allowed to connect to this target. You can also edit these values later. Once you've finished this step, you only have an empty target. Now you have to add virtual hard drives to the target. 31. For each virtual shared disk you want to create via iSCSI:a. Right-click on the target you just created and choose "Create Virtual Disk for iSCSI Target"
existing one. If you choose an existing one, it will warn you that its data will be overwritten. And yes, this will create a virtual disk inside a virtual disk file. To help me keep track of inner vs. outer virtual disks, I name these inner disks with a
"Level2" suffix. E.g., For my quorum disk, I enter "Q:\QuorumDiskLevel2.vhd". The Q drive itself is a .vhd at the Hyper-V level, and this new QuorumDiskLevel2.vhd file is on the Q drive, so it's inside the quorum.vhd file. Cool, huh? This is a necessary characteristic of a Hyper-Cluster, but there's no need to virtualize any additional layers.
c. On the Size screen, note that you have to enter values in Megabytes, even though it's telling you the space available in Gigabytes. Annoying, isn't it?
You should now be done with WSS, and its iSCSI targets should be available to your cluster nodes' iSCSI initiators.
Configure Node1
32. If you didn't leave the iSCSI Initiator open, launch it from the Administrative Tools folder or use Start -- Run... iscsicpl.
33. If everything is configured correctly, you should have 2 entries under Discovered targets on the Targets tab. The iSCSI Qualified Name (IQN) of hyperclustershareddisks should be highlighted but have an inactive status. Click the Connect button to connect it.
34. Leave the checkmark to add this target to your Favorite Targets, which will enable the iSCSI initiator on this node to automatically reconnect to this target after reboots.
35. Click the Advanced button.
a. Switch the Local Adapter to "Microsoft iSCSI Initiator."
b. Set the Initiator IP to "192.168.2.30 if you're using Appendix A.
c. Set the Target Portal IP to "192.168.2.20 / 3260" if you using Appendix A. d. Click the OK button.
If you get a connection failure, you'll need to double check your network configurations and firewall rules. You can temporarily disable your firewalls just to determine if they’re the problem. 36. Click on the Volumes and Devices tab, click the Auto Configure button, and click OK.
37. Open Disk Manager on this node and for each new disk:
a. Find the gray box on the left side of the graphical representation of the disk where it says "Offline" or "Unknown."
b. If it says "Offline," then right-click in and click "Online." It should now say "Unknown."
c. When it says "Unknown," right-click the gray box and choose "Initialize Disk."
d. Now right-click the white rectangle that represents the disk and choose "New Simple Volume..." Use the maximum size and assign the same drive letter and label
you used in WSS08 for that disk and use the quick format option. 38. Open Server Manager, click on the Features node, and click on Add Features. 39. Click Failover Clustering, Next, and Install.
Configure Node2
Configure Node2 identically to Node1 with these exceptions:
• The Initiator IP is 192.168.2.31 and the Target Portal IP is 192.168.2.31/3260.
• You can skip the Disk Manager for Node2 since these are shared disks. However, if you want to use them on Node2 before you create a cluster, you'll need to open the disk manager, right-click, and click to bring the disks online. If you do bring them online on Node2, you won't have to initialize them, format them, or label them. Also, if you bring them online this way, the drive letters probably won't match what you set them to in Node1.
Install and Configure Cluster
After installing Failover Clustering on both nodes:40. In Server Manager on either one of the nodes, under Features, click on Failover Cluster Manager (or Administrative Tools --> Failover Cluster Manager).
41. Under Management, click on "Validate a Configuration."
42. Enter "node1" and click Add, then enter "node2" and click Add, then click Next.
43. Choose Run All Tests, click Next, and Next one more time to start the validation process. Watch the status to see all the cool tests the validator does that you don't have to! 44. If the validation report shows any problems, fix them before going on.
45.
When the validation report is clear, I recommend that you make Hyper-V snapshops of your virtual machines (WSS08 and your cluster nodes) so you can roll back to this clean pre-cluster configuration. This is especially useful if you want to practice creating a pre-cluster several times without having to recreate your Hyper-Cluster environment.At this point, you're through configuring your Hyper-Cluster, and setting up a cluster and clustered services should be exactly the same as if it were on physical systems.
If you're really anxious to create your cluster and don't have that documentation handy, just find the "Create a Cluster..." link inside the Failover Cluster Manager on any of the cluster nodes.
One final tip: Once you're got your cluster created, consider making another set of Hyper-V snapshots of WSS08 and each cluster node. Then if you make a change that breaks something, you can roll it back.
Appendix A: Sample Network Chart
Network Setting Host WSS08 Node1 Node2
Public IP Address 192.168.1.10 192.168.1.20 192.168.1.30 192.168.1.31 Public Subnet Mask 255.255.255.0 255.255.255.0 255.255.255.0 255.255.255.0 Public Gateway 192.168.1.1 192.168.1.1 192.168.1.1 192.168.1.1 Public DNS Server 192.168.1.10 192.168.1.10 192.168.1.10 192.168.1.10
iSCSI IP Address N/A 192.168.2.20 192.168.2.30 192.168.2.31 iSCSI Subnet Mask N/A 255.255.255.0 255.255.255.0 255.255.255.0
iSCSI Gateway N/A N/A N/A N/A
iSCSI DNS Server N/A N/A N/A N/A
Heartbeat IP Address N/A N/A 192.168.3.30 192.168.3.31
Heartbeat Subnet Mask N/A N/A 255.255.255.0 255.255.255.0
Heartbeat Gateway N/A N/A N/A N/A
Heartbeat DNS Server N/A N/A N/A N/A
The default gateway and DNS server are needed on the Public network for Internet access for Windows activation and updates; the Hyper-V host doesn't need the iSCSI or Heartbeat networks; and only the cluster nodes need the Heartbeat network.
