This section describes how to grow a DR cluster. Growing a DR enabled cluster requires that both sites grow by the same number of nodes. In general the installation of each active/backup server couple needs not to be done at the same time, but it’s highly recommended. The overcautious technician may also decide to install the backup node prior to the active node.
The following sections will only explain the differences from the basic DR installation in the sections before.
8.6 Mixing eX5/X6 Server in a DR Cluster
Please read chapter 9.2: Mixed eX5/X6 DR Clusters on page 97. Information given there takes precedence over the instructions below.
8.6.1 Hardware Setup
Please refer to 8.3: Hardware Setup on page 71 and follow the instructions there. Ping the new machine on the GPFS network from all machines to test if the network configuration is correct. Ping the new machine on the HANA network from all servers, it is supposed to be reachable only from nodes on the same site.
8.6.2 GPFS Part 1
1. First step is to add/etc/hosts entries on every machine. Let’s assume that the new nodes are the 9th and 10th nodes with node09 going to the active site and 10 into the backup site. Distribute any new nodes evenly into the existing failure groups (topology), so that a failure group has at most one more node than the other, put the backup server into the corresponding FG on the backup site.
In the example above, the 9th node will go into failure group 1 (1,0,x) getting the topology vector 1,0,3 and the 10th node will go into failure group 3 (1,1,x) with topology vector 1,1,3.
On all existing nodes, add host entries for the the GPFS network, .e.g.:
1 192.168.1.109 gpfsnode09
2 192.168.1.110 gpfsnode10
On the new nodes add entries for all other nodes. Copying the entries from one of the existing nodes is the easiest way.
First add host keys for the new nodes to the existing machines. Run on any existing node
1 # for srcnode in gpfsnode0{1..8} ; do echo node $srcnode ; ssh $srcnode 'for ←-,→target in gpfsnode0{9,10} ; do echo -n $target ; ssh-keygen -R $target ; ←-,→ssh-keyscan -t rsa target >> /root/.ssh/known_hosts ; done '; done
The value gpfsnode01..8 will generate a list from gpfsnode01 to gpfsnode08, if the host names differ or are not consecutive, replace this with a space separated list of host names. The same applies to gpfsnode09,10 which are the new nodes in this example.
Then copy the root SSH key to the new news. Issue these command on one of the existing cluster nodes:
1 # scp /root/.ssh/authorized_keys /root/.ssh/id_rsa /root/.ssh/id_rsa.pub ←-,→root@gpfsnode09:/root/.ssh/
2 # scp /root/.ssh/authorized_keys /root/.ssh/id_rsa /root/.ssh/id_rsa.pub ←-,→root@gpfsnode10:/root/.ssh/
On all new cluster nodes run this command
1 # for node in gpfsnode{01..10} ; do echo -n $node ; ssh-keygen -R $node ; ssh-←-,→keyscan -t rsa $node >> /root/.ssh/known_hosts ; done
Test the SSH key exchange by runnign this command on any node
1 # for srcnode in gpfsnode{01..10} ; do echo from node $srcgpfsnode ; ssh ←-,→$srcnode 'for target in gpsfnode{01..10} ; do echo To node $target ; ssh ←-,→$target hostname ; done '; done
The command should run without interaction and errors.
2. Install GPFS (base package):
1 # cd /var/tmp/install/gpfs-<GPFS-RELEASE>
2 # rpm -ivh gpfs.base-<GPFS-RELEASE>-0.x86_64.rpm 3. Update to the latest GPFS Maintenance Release
Warning
It is highly recommended to upgrade to GPFS 3.5.0-17 or higher.
Install the following three packages for the latest (X) maintenance release:
1 # rpm -ivh gpfs.docs-<GPFS-RELEASE>-X.noarch.rpm
2 # rpm -ivh gpfs.gpl-<GPFS-RELEASE>-X.noarch.rpm
3 # rpm -ivh gpfs.msg.en_US-<GPFS-RELEASE>-X.noarch.rpm 4. Verify your GPFS installation:
1 # rpm -qa | grep gpfs
The installed packaged from above should be listed here.
5. Build the GPFS Portability Layer
Follow the instructions in/usr/lpp/mmfs/src/README:
1 # cd /usr/lpp/mmfs/src
2 # make Autoconfig
3 # make World
4 # make InstallImages
6. To add the new nodes to the cluster run on any running node
1 # mmaddnode -N gpfsnode09,gpfsnode10 7. Mark the servers as licensed:
1 # mmchlicense fpo --accept -N gpfsnode09,gpfsnode10
Please use the correct licensed for the nodes. Server and FPO are just examples.
8. Start the new nodes
1 # mmstartup -N gpfsnode09,gpfsnode10
9. Create the disk descriptor files. Before adding the disks to the shared file system, you must create the disk descriptor or stanza files. You can create them on any node on the cluster, but it is preferably done on the node where the files for the initial cluster creation are located. Please see chapter 8.4.3: GPFS Disk configuration on page 77 for a description of the stanza files. You only need to create entries for the drives on the new nodes and you can omit the pool configuration entries. Let us assume the new file is/var/mmfs/config/disk.list.data.gpfsnode0910.
10. Create NSDs
1 # mmcrnsd -F /var/mmfs/config/disk.list.data.gpfsnode0910
8.6.3 HANA Backup Node Installation
Skip this for a node on the active site. For the HANA installation on the backup site, we need a temporary filesystem which must satisfy some requirements. RAM based filesystems are not sufficient, so we use the fresh created NSDs for a temporary filesystem, install the backup instance, and destroy the temporary filesystem afterwards before continuing with the installation.
1. Create a temporary filesystem
1 /usr/lpp/mmfs/bin/mmcrfs sapmnttmp -F
/var/mmfs/config/disk.list.data.←-,→gpfsnode0910 -A no -B 1M -N 3000000 -v no -m 1 -M 3 -r 1 -R 3 j hcluster ←-,→--write-affinity-depth 1 -s failureGroupRoundRobin --block-group-factor 1←-,→ -Q yes
Before continuing with the installation make sure that the GPFS file system sapmntdata is not mounted at/sapmnt on the new nodes.
Mount this filesystem on all new backup nodes
1 mmmount sapmnttmp /sapmnt -N <new backup nodes>
2. Install HANA on backup site
In order to prepare the backup site, it is necessary to do a standard HANA installation and then delete the installed content on the shared filesystem. A tool to automate this procedure is currently in development by SAP.
Install SAP HANA on the backup site as described in the official SAP documentation available here: http://help.sap.com/hana_appliance. The location of the SAP HANA installation files is /var/tmp/saphana. Do a single node installation on each node. Make sure to use exact the same SAP SID, SAP instance number, user names, user IDs, group names and group IDs, paths as in the original DR-HANA installation. You can use the command id to query user and group information.
3. Stop HANA and SAP Host agent on backup site
Log in as <SID>adm on one node and stop SAP HANA:
1 $ HDB stop
Then log in as root and stop SAP Host agent and other services:
1 # /etc/init.d/sapinit stop
Afterwards disable the autostart of the sapinit service
1 # chkconfig sapinit off
Do the last two steps on all backup nodes.
4. Delete SAP HANA shared content
5. Disable mmfsup script on backup site nodes An installation with the Recovery Image will install a mmfsup script which will automatically start SAP HANA after the file system comes up. This must be deactivated as it may start SAP HANA on both sites (using the same hostnames.) The script resides in/var/mmfs/etc. Remove it on all cluster nodes.
1 # rm /var/mmfs/etc/mmfsup
6. Delete temporary filesystem After installing all new backup nodes, unmount temporary Filesystem on all nodes
1 mmmumount sapmnttmp -a and delete it
1 mmdelfs sapmnttmp
This will delete all shared HANA content and will leave the node specific HANA parts installed.
8.6.4 GPFS Part 2
1. Add disks to sapmntdata filesystem
1 # mmadddisk sapmntdata -F /var/mmfs/config/disk.list.data.gpfsnode0910 2. Verify NSD status
Verify that all NSDs are up and running
1 # mmlsdisk sapmntdata 3. Mount GPFS on active
On the new active nodes and only on these, mount the GPFS file system
1 # mmmount sapmntdata -N gpfsnode09,gpfsnode10 GPFS setup is now complete.
8.6.5 HANA
8.6.5.1 Install HANA on active site
1. Please make sure that you have mounted the shared file system on the new nodes.
1 # mmlsmount sapmntdata -L
2. If not already installed, install the SAP host agent
1 # cd /var/tmp/install/saphana/DATA_UNITS/SAP_HOST_AGENT_LINUX_X64
2 # rpm -ihv saphostagent.rpm
As recommended by the RPM installation, a password for sapadm may be set.
3. Deactivate automatic startup through sapinit at startup.
Running SAP’s startup script during system boot must be deactivated as it will will be executed by a GPFS startup script after cluster start. Execute:
1 # chkconfig sapinit off
4. Install SAP HANA worker and standby nodes as described in the guide "SAP HANA Administration Guide".
Warning
SAP HANA in this DR solution must be installed using the hostname of the HANA-internal network (usually on bond1, hostname hananodeXX). The host based routing used in the HA solution is not applicable for the DR solution.