Shared Storage Setup with System Automation

IBM Tivoli System Automation for Multiplatforms

Shared Storage Setup with System Automation

Setup a shared disk on both nodes

To transparently failover an application from one cluster node to another, all relevant program and runtime data must be located on a shared disk. A running application instance transparently accesses this data underneath a common mount point. Any node that potentially hosts the application must be able to mount the file system on the shared disk.

Shared disk requirements

Proper disk sharing by the nodes that can run the application consuming data on shared disk

requires some precautions to prevent data corruption that might be caused by multiple nodes accessing the same disk.

Control tasks

System Automation for Multiplatforms starts, stops, and monitors the shared disk and makes its file system accessible to the application consuming the data on shared disk. These tasks are the mount, unmount, and test operations including the associated LVM operations in case the file systems are stored on logical volumes. Only System Automation is allowed to operate the shared disk and mount and unmount its file system. Make sure that no automount features of the operating system mount the file system on the disk during system boot or runtime. The file system of the shared disk must not be checked

automatically after system reboot, for example on Linux, the /etc/fstab file must have a 0 (zero) set as the sixth field for the filesystem entry.

Common mount point

The same mount point must be used for the file system on the shared disk on all cluster nodes.

No multiple mounts

The file system on the shared disk may be mounted only to one node at a time. System Automation for Multiplatforms will enforce this rule to ensure file system integrity.

The same applies to logical volume, volume groups and software raid block devices that must not be activated, resp. varied on on more than one system at a time.

Data mirroring

Several techniques can provide disk mirroring functions. Hardware disk storage systems often provide raw device data mirroring to a second hardware disk storage system usually from the same vendor. When hardware mirroring features are available, they are usually the preferred choice.

Another choice is the use of software mirroring techniques like RAID1 mirroring provided by the MD device support on Linux. Comprehensive documentation and examples for this feature can be found at

https://raid.wiki.kernel.org/index.php/RAID_setup.

When the shared file system used for the application to be made highly-available is hosted on a Linux MD device, System Automation for Multiplatforms controls the MD devices as well while the file system is made available to the node where the consumer application is started. You have to ensure that the MD configuration file /etc/mdadm.conf describes the MD device on each node. If only one storage system is used to hold the shared disk, ensure that some RAID redundancy technique is used, preferably provided by the storage system itself, to store the data on the storage system.

LVM setups

If the shared file system resides on a LVM logical volume. The volume group containing the logical volume is implicitly automated by System Automation for Multiplatforms as well. Make sure that the shared disk holding the volume group contains only application specific data. Then other applications do not need to access the volume group.

The associations between the shared file system, its logical volume, and volume group containers is harvested by System Automation for Multiplatforms automatically. When the shared disk is implemented as an MD device on Linux, this additional association is handled automatically too. System Automation starts and stops the underlying devices of the shared file system automatically in order to make the file system available or unavailable.

IBM.AgFileSystem naming of the shared file system

When System Automation for Multiplatforms harvests the disk storage on each cluster node, it assigns names to each file system found on the nodes. On Linux, these names are taken from the volume label, otherwise long serial number are assigned as the IBM.AgFileSystem name.

You may want to set the filesystem volume name of the shared file system to something like app-fs to make life simpler for operators. On Linux, the volume name can be set by using the -L parameter of the tune2fs command. Example: To set and list the volume label on the logical volume vg0-lvolsco, enter

the following commands:

tune2fs -L app-fs /dev/mapper/vg0-lvolapp tune2fs -l /dev/mapper/vg0-lvolapp | grep name

Examples of shared disk or file system setups

Example 1: Shared disk setup in a VMWare ESX environment

Configure VMWare ESX to map shared disks to the guest operating systems. Make sure that the guest operating system is stopped before you change the configuration.

Shared disk configuration options:

Compatibility Mode: virtual - independent/persistent

Disks in persistent mode behave like conventional disks on a physical computer. All data written to a disk in persistent mode are written permanently to the disk.

SCSI Bus sharing: physical

Virtual disks can be shared between virtual machines on any server.

Steps to setup shared disks in VMWare ESX:

1. Create at least one shared disk that is accessible from both systems in the SAN.

2. Attach the shared disk to the first guest by using the VMware configuration menu: Add Hardware -> Create

Figure 4. Attaching the shared disk to the first guest

3. Attach all shared disks to the second guest by using the VMWare ESX configuration menu: Add Hardware -->

Figure 5. Attaching all shared disks to the second guest

Example 2: Shared disk setup on a Linux MD device

Creating a data mirror using a Linux MD RAID1 device

The following figure shows two servers, for example rack, blade, or stand-alone systems and two disk storage systems.

/dev/sdb, /dev/sdc /dev/sdb, /dev/sdc

Cluster node 1

Cluster node 2

disk storage system 1 disk storage system 2

Figure 7. Sample setup with two servers and two disk storage systems

Both servers can access both disk storage systems at the same time by using cross cabling. Two raw disks /dev/sdb and /dev/sdc are used without disk partitions, for example /dev/sdb instead of /dev/sdb1.

Steps to setup MD device /dev/md0:

On both nodes:

Inspect the file /etc/mdadm.conf to contain no entries "ARRAY /dev/md0 ". If such an entry already exists, check whether the entry is a leftover or a device /dev/md0 already exists in the system. If so, choose another device name, for example /dev/md1 for a new array and use this name in the following steps.

#node1:

Create a mirrored MD devices /dev/md0

mdadm --create --verbose /dev/md0 --level=mirror --raid-devices=2 /dev/sdb /dev/sdc

Check that the MD device has been created

Add the entry for /dev/md0 to /etc/mdadm.conf

mdadm --detail --scan | grep /dev/md0 >> /etc/mdadm.conf

Stop the MD device before proceeding on node2

mdadm --stop /dev/md0

#node2:

Assemble the array previously created on node1

mdadm --assemble /dev/md0 /dev/sdb /dev/sdc

Check that the MD device has been assembled

mdadm --detail /dev/md0

Add the entry for /dev/md0 to /etc/mdadm.conf

mdadm --detail --scan | grep /dev/md0 >> /etc/mdadm.conf

Stop the MD device before further proceeding on node1

mdadm --stop /dev/md0

node1:

Make the array available on node1 again:

mdadm --assemble /dev/md0 /dev/sdb /dev/sdc

Creating a Logical Volume Manager (LVM2) setup

One physical volume, one volume group, and one logical volume is recommended for the high availability setup. More physical devices can be added later and the logical volume can be extended dynamically. The creation of physical volume, volume group, and logical volume will only be done on one node.

CLVM (Cluster Aware LVM) is not supported with System Automation for Multiplatforms.

LVM2 setup is recommended even without data mirroring. If no data mirroring is used, then replace /dev/md0 with the /dev/sd* device used. For more information refer to Related documentation.

Steps to setup the volume group and logical volume:

# node1:

Make the MD device /dev/md0 a physical volume

pvcreate /dev/md0

pvscan pvdisplay

Create the volume group vg0

vgcreate --clustered n vg0 /dev/md0

Check if the volume group is created

vgscan vgdisplay vgs

Create the logical volume lvolapp. For application using more than one filesystem, create follow the steps below multiple times and use the --size parameter instead of –extents. In this example we use the complete available size of the volume group

lvcreate --name lvolsco --extents 100%VG vg0

Check that the logical volume is created

lvscan lvdisplay

#node2

No special steps are needed at this time. The volume group and its contents will be imported later.

Creating the file system

A filesystem must be created on the new logical volume. Recommended file system types are ext3, ext4, or xfs. The file system type GFS2 is not supported with System Automation for Multiplatforms. You have to format the disk on one node only. When entering the file system into /etc/fstab on Linux, the flag noauto must be specified, so that file system is not mounted automatically. System Automation for Multiplatforms handles all required mounts. The file system entry in the file /etc/fstab must have the same content on both nodes.

Steps to setup the file system:

# node1:

Create an ext3 filesystem on the logical volume and set the file system volume label to app-fs

mkfs -t ext4 -L app-fs /dev/vg0/lvolapp Create the mount point for the file system

mkdir /var/app

Save a backup of /etc/fstab

cp /etc/fstab /etc/fstab_orig

Add a new entry into /etc/fstab

Mount the file system

mount /var/app

Check if the file system is successfully mounted

mount

# node2:

Create the mount point for the file system

mkdir /var/app

Save a backup of /etc/fstab

cp /etc/fstab /etc/fstab_orig

Add a new entry into /etc/fstab

echo "/dev/vg0/lvolsco /var/sco ext3 rw,noauto 0 0" >> /etc/fstab

Refreshing the cluster after the shared disk setup

The following paragraph applies to Linux only.

For System Automation for Multiplatforms to harvest the newly created disk setup and file system, use

refrsrc IBM.Disk

first on node1 where application filesystem is mounted (!). List the filesystem resources with

lsrsrc IBM.AgFileSystem

to find out whether the harvesting process has finished and application filesystem and its underlying resources have already been found on node1.

Then again run

refrsrc IBM.Disk

on node2.

Again list the filesystem resources until there is a floating filesystem resource for the application. For that purpose use the command

Troubleshooting hints

StorageRM's default setting are in some cases not supported by SANs, so in general they should be changed

Disk resources are by default reserved (SCSI-2) – only DS4k disks support this right now, so turn it off for shared disks

chrsrc -s “Name like '%' & ResourceType=1” IBM.Disk DeviceLockMode=0

Then stop periodic harvesting

chrsrc -c IBM.Disk HarvestInterval=0

Then make sure StorageRM harvesting has found the shared filesystem(s) with the following command:

lsrsrc -s “Name like '%' & ResourceType=1” IBM.AgFileSystem Name MountPoint SysMountPoint

If the filesystem is not there, delete the current configuration (!) to force the StorageRM to repopulate its resource model with a harvest

rmrsrc -s “Name like '%'” IBM.Disk

and trigger a reharvest on all nodes starting with the node where the filesystem is currently mounted

refrsrc IBM.Disk

Then the filesystem should be there and can be added to a SA MP resource group.

Important !

• Make sure you have package rsct.opt.storagerm-3.1.4.3-13192.ppc.rpm or later installed on every node.

• Make sure that shared filesystems are reflected in /etc/fstab of every node

• Make sure that shared filesystems have suitable mount options in /etc/fstab and are not marked as read-only