Caché High Availability Guide

(1)

Caché High Availability Guide

Version 2012.2

(2)

Caché High Availability Guide

This book was assembled and formatted in Adobe Page Description Format (PDF) using tools and information from the following sources: Sun Microsystems, RenderX, Inc., Adobe Systems, and the World Wide Web Consortium at www.w3c.org.The primary document development tools were special-purpose XML-processing applications built by InterSystems using Caché and Java.

, ,

Caché WEBLINK, Distributed Cache Protocol, M/SQL, M/NET, and M/PACT are registered trademarks of InterSystems Corporation.

, , , ,

InterSystems Jalapeño Technology, Enterprise Cache Protocol, ECP, and InterSystems Zen are trademarks of InterSystems Corporation. All other brand or product names used herein are trademarks or registered trademarks of their respective companies or organizations. This document contains trade secret and confidential information which is the property of InterSystems Corporation, One Memorial Drive, Cambridge, MA 02142, or its affiliates, and is furnished for the sole purpose of the operation and maintenance of the products of InterSystems Corporation. No part of this publication is to be used for any other purpose, and this publication is not to be reproduced, copied, disclosed, transmitted, stored in a retrieval system or translated into any human or computer language, in any form, by any means, in whole or in part, without the express prior written consent of InterSystems Corporation.

The copying, use and disposition of this document and the software programs described herein is prohibited except to the limited extent set forth in the standard software license agreement(s) of InterSystems Corporation covering such programs and related documentation. InterSystems Corporation makes no representations and warranties concerning such software programs other than those set forth in such standard software license agreement(s). In addition, the liability of InterSystems Corporation for any losses or damages relating to or arising out of the use of such software programs is limited in the manner set forth in such standard software license agreement(s).

THE FOREGOING IS A GENERAL SUMMARY OF THE RESTRICTIONS AND LIMITATIONS IMPOSED BY INTERSYSTEMS CORPORATION ON THE USE OF, AND LIABILITY ARISING FROM, ITS COMPUTER SOFTWARE. FOR COMPLETE INFORMATION REFERENCE SHOULD BE MADE TO THE STANDARD SOFTWARE LICENSE AGREEMENT(S) OF INTERSYSTEMS CORPORATION, COPIES OF WHICH WILL BE MADE AVAILABLE UPON REQUEST.

InterSystems Corporation disclaims responsibility for errors which may appear in this document, and it reserves the right, in its sole discretion and without notice, to make substitutions and modifications in the products and practices described in this document.

For Support questions about any InterSystems products, contact:

InterSystems Worldwide Customer Support

+1 617 621-0700 Tel:

+1 617 374-9391 Fax:

[email protected] Email:

(3)

List of Figures

Figure 1–1: Failover Cluster Configuration... 5

Figure 1–2: Concurrent Cluster Configuration... 6

Figure 1–3: ECP Cluster Configuration... 7

Figure 3–1: Single Failover Cluster... 14

Figure 3–2: Failover Cluster with Node Failure... 15

Figure 3–3: Multiple Failover Cluster... 15

Figure 3–4: Multiple Failover Cluster with Node Failure... 16

Figure 3–5: Physical Disk Dependency Properties... 20

Figure 3–6: Cluster Resource General Properties... 21

Figure 3–7: Cluster Resource Dependencies Properties... 22

Figure 3–8: Cluster Resource Policies Properties... 22

Figure 3–9: Cluster Resource Advanced Policies Properties... 23

Figure 3–10: Cluster Resource Parameters Properties... 23

Figure 4–1: Mirror ... 54

Figure 4–2: Async Member Connected to Multiple Mirrors... 55

Figure 4–3: Multiple Async Members Connected to Single Mirror... 56

Figure 4–4: Mirror Communication Channels... 57

Figure 4–5: Status of Systems... 61

Figure 4–6: Status of Systems... 62

Figure 4–7: Sample Configuration: Direct Connect... 64

(6)

List of Tables

Table 4–1: Mirror System Tunable Options... 43

Table 4–2: Mirror Configuration Details Form (Part 1) — Mirror and First Failover Member (Configuration Items)... 51

Table 4–3: Mirror Configuration Details Form (Part 2) — Second Failover Member (Configuration Items)

... 53

Table 4–4: Mirroring Processes on Primary Failover Member ... 58

Table 4–5: Mirroring Processes on Backup Failover/Async Member ... 58

(7)

About This Book

As organizations rely more and more on computer applications, it is vital to safeguard the contents of databases. This guide explains the many mechanisms that Caché provides to maintain a highly available and reliable system. It describes strategies for recovering quickly from system failures while maintaining the integrity of your data.

There are mechanisms available to maintain high availability including shadow journaling and various recommended failover strategies involving Caché ECP (Enterprise Cache Protocol) and clustering. The networking capabilities of Caché can be customized to allow cluster failover.

The following topics are addressed:

• System Failover Strategies

• ECP Failover

• Caché and Windows Clusters

• Mirroring

This guide also contains the following platform-specific appendixes:

• Using Red Hat Enterprise Linux Clusters with Caché

• Using IBM PowerHA SystemMirror with Caché

• Using HP ServiceGuard with Caché

For detailed information, see the Table of Contents.

(8)

(9)

1

System Failover Strategies

Caché fits into all common high-availability configurations supplied by operating system providers including Microsoft, IBM, HP, and EMC. Caché provides easy-to-use, often automatic, mechanisms that integrate easily with the operating system to provide high availability.

There are four general approaches to system failover. In order of increasing availability they are:

• No Failover Strategy

• Failover Cluster

• Concurrent Cluster

• ECP Cluster

Each strategy has varying recovery time, expense, and user impact, as outlined in the following table.

User Impact Expense

Recovery Time Approach

High No cost to low cost

Unpredictable No Failover Strategy

Moderate Moderate

Minutes Failover Cluster

Low Moderate to high

Seconds Concurrent Cluster

None Moderate to high

Immediate ECP Cluster

There are variations on these strategies; for example, many large enterprise clients have implemented ECP cluster and also use failover cluster for disaster recovery.

It is important to differentiate between failover and disaster recovery. Failover is a methodology to resume system avail-ability in an acceptable period of time, while disaster recovery is a methodology to resume system availavail-ability when all failover strategies have failed.

If you require further information to help you develop a failover and backup strategy tailored for your environment, or to review your current practices, please contact the InterSystems Worldwide Response Center (WRC).

(10)

is maintained through global journaling and transaction processing. While global journaling can be disabled and transaction processing is optional, InterSystems highly recommends using them.

If a production system failure occurs, such as a hardware failure, the database and application are generally unaffected. Disk degradation, of course, is an exception. Disk redundancy and good backup procedures are vital to mitigate problems arising from disk failure.

With no failover strategy in place, system failures can result in significant downtime, depending on the cause and your ability to isolate and resolve it. If a CPU has failed, you replace it and restart, while application users wait for the system to become available. For many applications that are not business-critical this risk may be acceptable. Customers that adopt this approach share the following common traits:

• Clear and detailed operational recovery procedures

• Well-trained, responsive staff

• Ability to replace hardware quickly

• Disk redundancy (RAID and/or disk mirroring)

• Enabled global journaling

• 24x7 maintenance contracts with all vendors

• Expectations from application users who tolerate moderate downtime

• Management acceptance of risk of an extended outage

Some clients cannot afford to purchase adequate redundancy to achieve higher availability. With these clients in mind, InterSystems strives to make Caché 100% reliable.

1.2 Failover Cluster

A common and often inexpensive approach to recovery after failure is to maintain a standby system to assume the production workload in the event of a production system failure. A typical configuration has two identical computers with shared access to a disk subsystem.

After a failure, the standby system takes over the applications formerly running on the failed system. Microsoft Windows Clusters, HP MC/ServiceGuard, OpenVMS Clusters, and IBM HACMP provide a common approach for implementing failover cluster. In these technologies, the standby system senses a heartbeat from the production system on a frequent and regular basis. If the heartbeat consistently stops for a period of time, the standby system automatically assumes the IP address and the disk formerly associated with the failed system. The standby can then run any applications (Caché, for example) that were on the failed system. In this scenario, when the standby system takes over the application, it executes a pre-configured start script to bring the databases online. Users can then reconnect to the databases that are now running on the standby server. Again, WIJ, global journaling, and transaction processing are used to maintain structural and data integrity.

Customers generally configure the failover server to mirror the main server with an identical CPU and memory capacity to sustain production workloads for an extended period of time. The following diagram depicts a common configuration:

4 Caché High Availability Guide System Failover Strategies

(11)

Figure 1–1: Failover Cluster Configuration

IP address of STDBY IP address of PROD

State of PROD

191.10.25.50 191.10.25.1

FUNCTIONAL

191.10.25.1 N/A

OUT OF SERVICE

Note: Shadow journaling, where the production journal file is continuously applied to a standby database, includes inherent latency and is therefore not recommended as an approach to high availability. Any use of a shadow system for availability or disaster recovery needs should take these latency issues into consideration.

1.3 Concurrent Cluster

The concurrent cluster approach exploits a standby system that is immediately available to accept user connections after a production system failure. This type of failover requires the concurrent access to disk files provided, for example, by OpenVMS clusters.

In this type of failover two or more servers, each running an instance of Caché and each with access to all disks, concurrently provide access to all data. If one machine fails, users can immediately reconnect to the cluster of servers.

A simple example is a group of OpenVMS servers with cluster-mounted disks. Each server has an instance of Caché running. If one server fails, the users can reconnect to another server and begin working again.

(12)

Figure 1–2: Concurrent Cluster Configuration

C B

A State

300 users 300 users

300 users Normal

300 users 0 users

300 users B fails

450 users 0 users

450 users B users log on again

The 600 users on A and C are unaware of B's failure, but the 300 users that were on the failed server are affected.

1.4 ECP Cluster

The ECP cluster approach can be complicated and expensive, but comes closest to ensuring 100% uptime. It requires the same degree of failover as for a failover cluster or concurrent cluster, but also requires that the state of a running user process be preserved to allow the process to resume on a failover server. One approach, for example, uses a three-tier configuration of clients and servers.

6 Caché High Availability Guide System Failover Strategies

(13)

Figure 1–3: ECP Cluster Configuration

Users connect directly or via Web servers to a bank of ECP application servers.

In the case of application server failure, either new user traffic is routed to surviving application servers or failover cluster techniques automatically start a standby server which then is able to accept the traffic. In turn, the ECP application servers are connected to ECP data servers clustered in a failover cluster or concurrent cluster.

If a data server fails, any application server waiting for a response will have its request answered by a surviving member of the cluster based on the guarantees of a failover cluster or concurrent cluster.

(14)

(15)

2

ECP Failover

One of the most powerful and unique features of Caché is the ability to efficiently distribute data and application logic among a number of server systems. The underlying technology behind this feature is the Enterprise Cache Protocol (ECP), a distributed data caching architecture that manages the distribution of data and locks among a heterogeneous network of server systems. ECP is an important part of an application failover strategy for high-availability systems.

This chapter describes how the architecture works to maintain high availability:

• ECP Recovery

• ECP and Caché Clusters

• ECP Clusters

For more detailed information about ECP, see the Caché Distributed Data Management Guide.

2.1 ECP Recovery

The simplest case of ECP recovery is a temporary network interruption that is long enough to be noticed, but short enough that the underlying TCP connection stays active during the outage. During the outage, the application server (or client) notices that the connection is nonresponsive and blocks new network requests for that connection. Once the connection resumes, processes that were blocked are able to send their pending requests. If the underlying TCP connection is reset, the data server waits for a reconnection for a timeout (configurable, default is set to one minute). If the client does not succeed in reconnecting during that interval, all the work done by the previous connection is rolled back and the connection request is converted into a request for a brand new connection.

A more complex case is where the network outage is severe enough to reset the underlying TCP connection, but both client and data server stay up throughout the outage, and the client reconnects within the data server’s reconnection window. On reconnection, the main action that must be performed is to flush (or, eventually, re-validate) the client’s cache of downloaded blocks and the client’s cache of downloaded routines. In addition, the client keeps a queue of locks to remove and transactions to roll back once the connection is reestablished. By keeping this queue, there is never a problem with allowing a process to halt right away whenever it wants to, whether or not the servers it has pending transactions and locks

(16)

• The data server’s view of the current active transactions from each application server has been restored from the data server’s journal file.

• The data server’s view of the current active Lock operations from each application server has been restored, by having the application server upload those locks to the data server.

• The application server and the data server both agree on exactly which requests from the application server can be ignored (because it is certain they completed before the crash) and which ones should be replayed. Hence, the last recovery step is to simply let the pending network requests complete, but only those network requests that are safe to replay.

• Finally, the application server delivers to the data server any pending unlock or rollback indications that it saved from jobs that halted while the data server was restarting. All guarantees are maintained, even in the face of sudden and unanticipated data server crashes, as long as the integrity of the databases (including the WIJ file and the journal files) are maintained.

During the recovery of an ECP-configured system, Caché guarantees a number of recoverable semantics which are described in detail in the ECP Recovery Guarantees section of the “ ECP Recovery Guarantees and Limitations ” appendix of the

Caché Distributed Data Management Guide.

There are limitations to these guarantees which are described in detail in the ECP Recovery Limitations section of the “ ECP Recovery Guarantees and Limitations ” appendix of the Caché Distributed Data Management Guide.

2.2 ECP and Caché Clusters

Adding cluster nodes which utilize ECP is simple and straightforward. On each cluster node, configure the system as an ECP data server, by enabling the ECP service from the [System] > [Security Management] > [Services] page. Click %Ser-vice_ECP and select the Service enabled check box. This is the only configuration setting required to use this node as an ECP data server.

If you add a new member to the cluster, Caché does not need to change network configuration on every running member. Only the lock and increment requests are delivered in the internal cluster ECP connection. There is no data block to be sent from the master to the other cluster members as would be sent in ECP without clusters. When cluster failover happens, the cluster member asks the new master to do ECP recovery.

A cluster member creates ECP connections to the other cluster members so it can access the privately-mounted databases in the other members. If you configure a node to be an ECP application server, it is not used as the cluster connection; it works as a regular ECP connection. Though this connection can also be used for accessing clustered mounted databases, the cluster failover does not recover the connection.

If this is the first member to join the cluster, it is the master. Each node that joins the cluster does the following:

• Retrieves connection information (IP address and port) for each cluster member from the PIJ file.

• Validates the connection to each existing member to ensure cluster failover success.

• Allocates a system # (index to netnode array) and sets up a null system name for the system #.

• Initializes the netnode structure with ECP connection using the IP address and port from the master entry in the PIJ

file and puts the connections in Not Connected state.

Caché declares any ECP connection from a failing cluster member Disabled and releases all resources for that connection, including the locks in the lock table owned by the failed system.

The following sections outline what happens under the described conditions on a clustered system using the ECP architecture:

• Application Server Fails

10 Caché High Availability Guide ECP Failover

(17)

• Data Server Fails

• Network Is Interrupted

• Cluster as an ECP Database Server

2.2.1 Application Server Fails

If the data server becomes aware that the application server has halted, crashed, disconnected, or otherwise declared the connection dead, the data server declares the connection dead.

If a data server declares a connection is dead, it rolls back any open transactions for that application server and releases any locks held for that application server. It is then available for a new connection from that application server.

During application server recovery the application server retransmits any requests it had previously sent for which it had not yet received a response, and, in the case of a data server crash, it also transmits any locks it owned on the data server. Following this phase the users on the application server resume operations without any noticeable effect other than the pause—no data is lost or rolled back, and no application server user processes get errors. However, a few processes that were waiting for a $Increment or $Bit function that sets or clears a bit and returns the former value, may receive errors and have their open transactions rolled back.

2.2.2 Data Server Fails

If a data server crashes while application server connections are open, the data server does the following at restart, prior to allowing general system usage:

• Attempts to reestablish a connection with the application servers that were active.

• Allows them to reestablish locks they had on the data server.

• Reprocesses earlier requests for which the answers were never received by the application servers.

• Declares the connection dead and rolls back open transactions for any application server not heard from during the startup phase.

Following a data server crash, an application server waits for 20 minutes (a configurable time limit) before declaring the connection dead. If during that time the data server restarts, the application server goes through a recovery phase and resumes operation.

2.2.3 Network Is Interrupted

If either the application or the data server detects a network outage, the data server waits up to one minute (a configurable time limit) for the network to start working before declaring the connection dead. When the application server is not receiving responses to requests, and cannot determine if there is a network outage or a problem with the data server, it waits for up to 20 minutes (configurable) which gives the system manager time to discover that the data server has crashed and restart it.

Once either the application or the data server has declared a connection dead, there is no longer any ability to recover from a failure. In that case the data server rolls back any open transactions for that application server and releases its application locks. The application server is expected to issue <NETWORK> errors to any application processes that are still waiting for

(18)

2.2.4 Cluster as an ECP Database Server

In the Caché-supported genuine cluster environments, namely OpenVMS, using your Caché cluster as an ECP database server has the following limitations:

• You must run ECP to particular members of a cluster, not to the cluster as a whole. Do not use the IP address of the cluster. If a cluster master fails, you must reconnect to the new master IP address.

• In this type of cluster, ECP is used as a lock transport mechanism, not to transfer data.

2.2.4.1 Master Fails

Caché does the following on a cluster member after its cluster master fails:

• Sets the state of any ECP connection from the failed master to Disabled and releases all resources for that connection including all the locks in the lock table owned by the failed system.

• Sets the state of the cluster ECP connection to Trouble.

• Locates the IP address and port of the new master through the PIJ file.

• Creates a connection to the new master and performs the recovery procedure.

2.2.4.2 New Cluster Master

Caché does the following on a new cluster master after the previous cluster master fails:

• Sets the state of any ECP connections from the failed master to Disabled and releases all resources for that connection, including all locks in the lock table owned by the failed system.

• Stops all ECP daemons for the cluster ECP connection.

• Converts the granted remote locks to the old master into local locks.

• Removes all the pending cancel lock entries to the old master in the lock table, Caché does not remove the pending unlock entries because the process that issues the unlock request removes it eventually.

• Waits for all cluster members to upload their locks.

• Reissues the pending lock entries in the lock table by waking up the requesting processes.

• Sets the state of the cluster ECP connection to Disabled.

2.3 ECP Clusters

It is also possible to use ECP on third-party clustering platforms. Caché ECP Clusters is a high availability feature that enables failover from one ECP data server to another, using operating system level clustering to detect a failed server. The following appendix describes the specifics on the indicated platform:

• Using Red Hat Enterprise Linux Clusters with Caché

12 Caché High Availability Guide ECP Failover

(19)

3

Caché and Windows Clusters

Microsoft Windows operating systems do not support shared-all clusters:

• They do not offer a shared resource cluster model.

• They do not allow simultaneous access to shared drives: you cannot lock, read, or write to a cluster.

• If a drive fails, the operating system does not swap in a backup drive.

However, Microsoft Windows Server 2003 and Windows Server 2008 platforms allow you to cluster computers that share the same storage. You must have a RAID or SCSI disk drive system to do so.

This chapter contains the following subsections:

• Setting up Failover Clusters

• Example Procedures

3.1 Setting up Failover Clusters

This section provides an overview of the steps required to set up a cluster. For suggestions on other ways to run a large enterprise system, contact the InterSystems Worldwide Response Center (WRC).

To set up a failover cluster on a Windows Server platform, you must perform the steps as follows:

1. Configure the Microsoft cluster group on either cluster node and verify that it works. Depending on Windows Server platform you are using, refer to the following Microsoft Support documentation for more information:

• “Available Features in Windows Server 2003 Clusters”

• “Introducing Windows Server 2008 Failover Clustering” 2. Install Caché on the shared disks on both cluster nodes, as follows:

(20)

Note: If you define two or more Caché instances in a single cluster resource group, they will failover simulta-neously with the group.

b. Configure Caché.

Important: You must ensure that the default automatic startup setting is disabled; for information, see

Memory and Startup Settings in the “ Configuring Caché ” chapter of the Caché System

Adminis-tration Guide.

c. Stop Caché.

Important: Do not start and stop Caché from the Caché Launcher. Instead, using Failover Cluster Management, take the Caché cluster resource offline to stop Caché, and bring the Caché cluster resource online to start Caché.

d. Move the cluster group to the second cluster node. e. Install Caché on the second cluster node.

Note: Caché instances (members of the same Caché failover cluster representing the same cluster group resource) must be installed in the same directories and use the same install options. Caché instances from the same group, but representing different cluster group resources, must use the shared disks and IP addresses of the same cluster group.

3. On either node, configure Caché clustering via the Failover Cluster Management option (in the Microsoft Windows

Administrative Tools menu).

For descriptions of supported cluster configurations are described in the following subsections:

• Single Failover Cluster

• Multiple Failover Clusters

3.1.1 Single Failover Cluster

The following illustration shows a single failover cluster:

Figure 3–1: Single Failover Cluster

14 Caché High Availability Guide Caché and Windows Clusters

(21)

CLUNODE-1 and CLUNODE-2 are clustered together, and Caché is running on one node. During normal operations, the following conditions are true:

• Disk S is online on CLUNODE-1, and CLUNODE-2 has no database disks online.

• The instance CacheA runs on CLUNODE-1; CLUNODE-2 is idle. In this setup, if CLUNODE-1 fails, your system looks like this:

Figure 3–2: Failover Cluster with Node Failure

• Disk S is online on CLUNODE-2; CLUNODE-1 has no database disks online.

• The instance CacheA runs on CLUNODE-2; CLUNODE-1 is down. See the Example Procedures section of this chapter for a detailed example.

3.1.2 Multiple Failover Cluster

You may also set up a cluster with multiple failover nodes using the same procedures described in the previous sections for the single failover cluster. The following shows a failover cluster on multiple nodes:

Figure 3–3: Multiple Failover Cluster

(22)

CLUNODE-1 and CLUNODE-2 are clustered together, and Caché is running on both nodes. During normal operations, the following conditions are true:

• Disk S is online on CLUNODE-1, and Disk T is online on CLUNODE-2.

• The CacheA instance runs on CLUNODE-1; the CacheB instance runs on CLUNODE-2.

• Instances CacheA and CacheB cannot directly access each other’s cache.dat files; they can directly access only their own mounted cache.dat files.

With this type of setup, if CLUNODE-2 fails, your system looks like this:

Figure 3–4: Multiple Failover Cluster with Node Failure

Both CacheA and CacheB run on CLUNODE-1. Once you repair or replace CLUNODE-2, you can move your CacheB instance back to CLUNODE-2. If CLUNODE-1 were to fail, both CacheA and CacheB would run on CLUNODE-2.

See the Example Procedures section of this chapter for a detailed example.

3.1.3 CSP Gateway Considerations

For high availability solutions running over CSP, InterSystems recommends that you use a hardware load balancer for load balancing and failover. InterSystems requires that you enable sticky session support in the load balancer (see your load balancer documentation for directions on how to enable sticky session support); this guarantees that — once a session has been established between a given instance of the gateway and a given application server — all subsequent requests from that user run on the same pair. This configuration assures that the session ID and server-side session context are always in sync; otherwise, it is possible that a session is created on one server but the next request from that user runs on a different system where the session is not present, which results in runtime errors (especially with hyperevents, which require the session key to decrypt the request).

Note: It is possible to configure a system to work without sticky sessions but this requires that the CSP session global be mapped across all systems in the enterprise and can result in significant lock contention so it is not recommended.

Caché protects server passwords in the CSP Gateway configuration file (CSP.ini) using Windows DPAPI encryption. The encryption functions work with either the machine store or user store. The web server hosting the CSP Gateway operates within a protected environment where there is no available user profile on which to base the encryption; therefore, it must use the machine store. Consequently, it is not possible to decrypt a CSP Gateway password that was encrypted on another computer.

(23)

This creates a situation for clustered environments where the CSP.ini file is on a shared drive and shared among multiple participating computers. Only the computer that actually performs the password encryption can decrypt it. It is not possible to move a CSP.ini file containing encrypted passwords to another computer; the password must be reentered and re-encrypted on the new machine.

Here are some possible approaches to this issue:

• Use a machine outside of the cluster as the web server.

• Each time you fail over, reset the same password in the CSP Gateway.

• Configure each computer participating in the cluster so that it has its own copy of the CSP Gateway configuration file (CSP.ini) on a disk that does not belong to the cluster. Caché maintains the file in the directory hosting the CSP Gateway DLLs. Save and encrypt the password on each individual computer before introducing the node to the cluster. For example, where Disk C from each machine does not belong to the cluster and Caché is installed on Disk S, you may have the following:

CLUNODE-1: C:\INSTANCEDIR\CSP\bin\CSP.ini with password XXX encrypted by CLUNODE-1 CLUNODE-2: C:\INSTANCEDIR\CSP\bin\CSP.ini with password XXX encrypted by CLUNODE-2

• Disable password encryption by manually adding the following directive to the CSP.ini file before starting the CSP Gateway and adding the passwords:

[SYSTEM] DPAPI=Disabled

See the CSP Gateway Configuration Guide for more information.

3.2 Example Procedures

This section describes common procedures in the cluster building process. They apply to the single failover example, but you can adapt them to the additional steps in the multiple failover setup by replacing the CacheA names with the appropriate

CacheB names.

3.2.1 Create a Cluster Service

To create a cluster service, do the following:

1. From Failover Cluster Management, right-click Services and Applications, then click Create Empty Service or Application. 2. In the New service or application Properties dialog box, on the General tab, enter the group name (in this example,

CacheA Group) in the Name box.

(24)

3. Verify the preferred owners, for example, CLUNODE-1 and CLUNODE-2, as shown in the preceding graphic for CacheA Group, and click OK.

3.2.2 Create a Client Access Point

To create a client access point, do the following:

1. From Failover Cluster Management, right-click the group name (CacheA Group), click Add a resource, then add the IP address.

2. Right-click Resource Name, select Properties. then enter CacheA_IP as the name.

Note: If the initial (default) State is Offline, select Bring the resource online.

3. Assign the alias IP address used to connect to the instance (CacheA) by your users. This is not the cluster or node IP, but a new and unique IP specific for the instance (CacheA).

For this example, the value of CacheA_IP is 192.9.205.253. Once finished, the CacheA_IP resource has the following properties:

(25)

CacheA_IP has no dependencies.

3.2.3 Create a Physical Disk Resource

To create a physical disk resource for the shared disk containing CacheA, do the following:

1. From Failover Cluster Management, right-click the group name (CacheA Group), and click Add Storage.

2. From the list of available disks, select the Windows cluster node on which Caché will be installed, then click OK:

(26)

Figure 3–5: Physical Disk Dependency Properties

3.2.4 Install Caché

For information about installing Caché on the Windows cluster node, see Setting up Failover Clusters in this chapter and the “Installing Caché on Microsoft Windows” chapter of the Caché Installation Guide.

Each time you install an instance on a new node that is part of a Windows cluster, you must change the default automatic startup setting. Navigate to the [System] > [Configuration] > [Memory and Startup] page of the Management Portal and clear the Start Caché on System Boot check box to prevent automatic startup; this allows the cluster manager to start Caché. For more information, see Memory and Startup Settings in the “ Configuring Caché ” chapter of the Caché System Administration

Guide.

Following the installation you can remove the shortcuts from the Windows Startup folder (C:\Documents and Settings\All Users\Start Menu\Programs\Startup) that start the Caché Launcher on Windows login. The shortcut has the name you give the instance when you install (CACHE, for example).

The recommended best practice is to manage the cluster remotely from the launcher on a workstation connecting to the cluster IP address. If you choose to use the launcher locally from the desktop of one of the cluster nodes, be aware that certain configuration changes require a Caché restart and if you restart Caché outside the context of the cluster administrator, the cluster will declare the group failed and attempt failover.

3.2.5 Create a Caché Cluster Resource

On Windows Server 2003 and later, Caché automatically adds a new resource type, ISCCres2003, to Failover Cluster Man-agement when you install on an active Windows cluster node.

To add a Caché cluster resource of this type, perform the following steps:

1. From Failover Cluster Management, right click the group name, CacheA Group, point to Add resource of type ISC-Cres2003, and the select Properties.

2. On the General tab, enter the resource name, (CacheA_controller in this example), which is the name that is dis-played by Failover Cluster Management.

(27)

3. On the Dependencies tab, update, as necessary, the following settings:

• Click Insert to enter a dependency.

• Enter Disk S: in the Name and Physical Disk in the Resource Type, then click OK. 4. On the Policies tab, update, as necessary, the following settings:

• Clear the If restart is unsuccessful, fail over all resources in this service or application check box.

• Adjust Pending timeout to allow for any extra time required for user shutdown procedures. You should consider increasing the default value by the amount of the ShutdownTimeout configured for the Caché instance.

5. On the Advanced Policies tab, verify and update, as necessary, the following settings for the controller:

• From the Possible Owners list box, select cluster members on which Caché should be permitted to run.

• In both Basic resource health check interval and Thorough resource health check interval sections, select Use standard time period for the resource type.

6. On the Parameters tab, verify and update, as necessary, the following settings:

• In the Instance text box, enter the name of the Caché instance controlled by the cluster resource (specified in step 3).

• Enter a description of the Caché instance in the optional Comments field.

When you are finished, the CacheA_controller cluster resource has the following properties:

Figure 3–6: Cluster Resource General Properties

(28)

Figure 3–7: Cluster Resource Dependencies Properties

Figure 3–8: Cluster Resource Policies Properties

(29)

Figure 3–9: Cluster Resource Advanced Policies Properties

Figure 3–10: Cluster Resource Parameters Properties

(30)

(31)

4

Mirroring

Traditional availability and replication solutions often require substantial capital investments in infrastructure, deployment, configuration, software licensing, and planning. Caché Database Mirroring (Mirroring) is designed to provide an econom-ical solution for rapid, reliable, robust, automatic failover between two Caché systems, making mirroring the ideal automatic failover high-availability solution for the enterprise.

In addition to providing an availability solution for unplanned downtime, mirroring offers the flexibility to incorporate planned downtimes (for example, Caché configuration changes, hardware or operating system upgrades, etc.) on a particular Caché system without impacting the overall Service Level Agreements (SLA’s) for the organization. Combining InterSystems Enterprise Cache Protocol (ECP) application servers with mirroring provides an additional level of availability; application servers treat a failover as an ECP data server restart and allow processing to seamlessly continue on the new system once the failover is complete, thus greatly minimizing workflow and user disruption. Configuring the two failover mirror members in separate data centers offers additional redundancy and protection from catastrophic events.

Traditional availability solutions that rely on shared resources (such as shared disk) are often susceptible to a single point of failure with respect to that shared resource. Mirroring reduces that risk by maintaining independent components on the primary and backup mirror systems. Further, by utilizing logical data replication, mirroring reduces the potential risks associated with physical replication, such as out-of-order updates and carry-forward corruption, which are possible with other replication technologies such as SAN-based replication.

Finally, mirroring allows for special async members, which can be configured to receive updates from multiple mirrors across the enterprise. This allows a single system to act as a comprehensive Enterprise Data Warehouse, allowing enterprise-wide data mining and Business Intelligence using InterSystems DeepSee™. The async member can also be deployed in a Disaster Recovery model in which a single mirror can update up to six geographically-dispersed async members; this model provides a robust framework for distributed data replication, thus ensuring business continuity benefits to the organization; for more information, see Disaster Recovery in this chapter.

This chapter is divided into the following topics:

• Configuring Mirroring

• Caché Mirroring Concepts

• Mirroring Special Considerations

(32)

4.1 Configuring Mirroring

One of the primary goals of mirroring is to provide a robust, economic replication solution. In keeping with this goal, mir-roring has been designed to be adaptable to various system configurations and architectures. However, to ensure an optimal experience (that is, to maximize the likelihood of automated failover in the event of a failure of the primary, and minimize the amount of time taken for the failover to complete and users to be brought back on the new primary), you should adhere to the following general configuration guidelines:

• Caché Versions — When creating a mirror or adding members to a mirror, all Caché instances involved must be of the same version. However, instances that belong to an existing mirror can be upgraded separately and at different times.

• ICMP — Do not disable Internet Control Message Protocol (ICMP) on any system that is configured as a mirror

member because mirroring relies on ICMP to detect whether or not members are reachable.

• Network — InterSystems recommends that you use a high-bandwidth, low-latency, reliable network between the two

failover members. If possible, it is desirable to create a private subnet for the two failover members such that the data-and control-channel traffic can be routed exclusively on this private network. A slow network could impact the perfor-mance of both the primary and the backup failover members, and could directly impact the ability of the backup failover member to take over as primary in the event of a failover.

• Disk Subsystem — In order for the backup failover member to keep up with the primary system, the disk subsystems

on both failover members should be comparable; for example, if configuring a storage array on the first failover member, it is recommended that you configure a similar storage array on the second failover member. In addition, if network-attached storage (NAS) is used on one or both systems, it is highly recommended that separate network links be configured for the disk I/O and the network load from the mirror data to minimize the chances of overwhelming the network.

• Journal Throughput — As journaling is the core of mirror synchronization, it is essential to monitor and optimize

the performance of journaling on the failover members. For more information, see Journaling Best Practices in the “ Journaling ” chapter of the Caché Data Integrity Guide.

• Virtualization — While it is possible to configure one or all of the mirror components in a virtualized environment,

it is highly recommended that you configure these systems with appropriate redundancy; for example, the two failover members should not reside on the same physical host.

• User-defined Startup Routines — If you are migrating to mirroring and have ^%ZSTART or ^ZSTU routines, you

should instead use the ^ZMIRROR routine (see ^ZMIRROR User-defined Routine in this chapter). In addition, if you want to run a task on a specific failover member, use the task manager (see Using the Task Manager in the

“ Managing Caché ” chapter of the Caché System Administration guide) to specify the mirror member on which the task should run.

Important: After you set up a mirror, data in mirrored CACHE.DAT files is kept in sync automatically; rather than mirroring the CACHE.DAT file itself (as when copying a CACHE.DAT file from one system to another), only the data contained in the CACHE.DAT file on the primary mirror member is mirrored on other members. Other entities — including (but not limited to) users, roles, namespaces, non-mirrored databases, mappings (especially global mappings and package mappings) — are not kept in sync by the mirror on both failover members. Therefore, when you configure/update (add/modify/delete) these entities on one failover member, you must manually configure/update them consistently on the other failover member to keep them in sync. In addition, you must also perform the same manual operations on any async members on which you want these entities to be available.

For more information, see the following topics:

• Starting/Stopping ISCAgent

26 Caché High Availability Guide Mirroring

(33)

• Creating a Mirror

• Editing Mirror Configurations

• Adding Async Members to a Mirror

• Adding Databases to a Mirror

• Removing Mirror Configurations

• Disconnecting/Connecting Mirror Members

• Configuring an ECP Application Server to Connect to a Mirror

• Configuring a Mirror Virtual IP (VIP)

• Customizing the ISCAgent Port

• Mirror Tunable Parameters

• ^ZMIRROR User-defined Routine

• Forms

4.1.1 Starting/Stopping ISCAgent

The ISCAgent process must be installed and started on every system hosting a failover or async member before you can create or use a mirror. The component is installed when you install or upgrade Caché (see Mirror Upgrade Tasks in the

“ Upgrading Caché ” chapter of the Caché Installation Guide). Depending on the platform on which you are running, you can start or stop the ISCAgent process as follows:

• On Windows, start the ISCAgent process as follows:

1. In the Microsoft Windows Control Panel, select Services from the Administrative Tools drop-down list, and double-click ISCAgent to display the ISCAgent Properties window.

2. On the Extended tab, click Start to start, or Stop to stop ISCAgent.

3. On the Extended tab, select Automatic from the Startup type drop-down list.

• On non-Windows platforms, run the ISCAgent start/stop script, which is installed in the following locations, depending on the operating system:

– AIX®: /etc/rc.d/init.d/ISCAgent

– OpenVMS: the scripts, RunAgent and StopAgent, are located in the instance [.BIN] subdirectory. To start the ISCAgent process, run the following command as root from the [.BIN] subdirectory:

@RUNAGENT

– HP-UX: /sbin/init.d/ISCAgent – Linux: /etc/init.d/ISCAgent

– Mac OS X: /Library/StartupItems/ISCAgent/ISCAgent – Solaris: /etc/init.d/ISCAgent

(34)

Note: On UNIX®/Linux platforms, mirroring uses the general-purpose logging facility, syslog, to log messages: informational messages are logged as priority LOG_INFO; error messages are logged as priority LOG_ERR

(for information about configuring the syslog facility, see the documentation for your platform). In addition, mirroring requires that the bash shell is loaded, and the env executable script is installed in the /usr/bin

directory.

4.1.2 Creating a Mirror

Creating a mirror involves configuring two failover members and, optionally, one or more async members. After the mirror is created, you can add databases to be mirrored.

In addition, you can configure members to use SSL/TLS and to encrypt data; for detailed information, see Creating and Editing SSL/TLS Configurations for a Mirror in the “ Using SSL/TLS with Caché ” chapter, and the “Managed Key Encryption” chapter, respectively, of the Caché Security Administration Guide.

Important: Before you can create a mirror, you must ensure that the ISCAgent process has been started as described in the Starting/Stopping ISCAgent section in this chapter.

When you are creating a new mirror, configure the mirror members in the following order: 1. Create Mirror and Configure First Failover Member

2. Configure Second Failover Member

3. Add Second Failover Member to Mirror

4. Add Async Members to Mirror

5. Adding Databases to a Mirror

To simplify the configuration task, you can use the form provided at the end of this chapter to record essential system information; see Mirror Configuration Details Form.

4.1.2.1 Create Mirror and Configure First Failover Member

The following procedure describes how to create a mirror and configure the first failover member.

1. Navigate to the [System] > [Configuration] > [Create Mirror] page of the Management Portal on the first failover member (for example, SystemA), and click Create a Mirror; if the link is not active, click Enable Mirror Service and select the

Service Enabled check box.

2. On the [System] > [Configuration] > [Create Mirror] page, enter the following information in the Mirror Information section: a. Mirror Name — Enter a name for the mirror.

Note: Valid names must be 1–15 uppercase alphanumeric characters.

b. Use SSL/TLS — Specify whether or not you want to use SSL/TLS security by selecting Yes or No from the drop-down list. If you select Yes, click Set up SSL/TLS and follow the instructions in the Creating and Editing SSL/TLS Configurations for a Mirror section of the “ Using SSL/TLS with Caché ” chapter of the Caché Security

c. Use Virtual IP — Specify whether or not you want to use a Virtual IP address by selecting Yes or No from the drop-down list. If you select Yes, you are prompted for the following information:

• IP Address — Enter an IP address in the text box (for more information, see the Configuring a Mirror Virtual IP (VIP) section of this chapter).

• Mask (CIDR format) — Enter a Classless Inter-Domain Routing (CIDR) mask in the text box.

(35)

• Network Interface — Select a network interface from the drop-down list (for more information, see the Network Interface Considerations section of this chapter).

3. All other fields on this page are pre-populated with information specific to this node:

• Click Advanced Settings to display and edit additional pre-populated information about the mirror and the first failover member. In the Mirror Settings section (for more information, see the Mirror Tunable Parameters section in this chapter):

– Quality of Service Timeout (msec) — The maximum time, in milliseconds, that processes on this failover member wait for data to be acknowledged by the other member; the default is 2000 msec.

– Acknowledgment Mode — Select the acknowledgement mode that the primary failover member uses during a “ Mirror Synchronization ” process from the drop-down list; the default is Received.

– Agent Contact Required for Failover — Select whether or not the active backup failover member should take over as the primary failover member if it not able to communicate with the primary system; the default is Yes.

Note: If you select No, you must provide the ^ZMIRROR user-defined routine (see ^ZMIRROR User-defined Routine in this chapter).

• In the Mirror Failover Member Information section:

– Mirror Member Name — The name of the failover member you are configuring on this node (for example, SystemA); it defaults to a unique system name.

– Superserver Address — The IP address or host name that external systems can use to communicate with this failover member.

– Mirror Agent Address — The IP address or host name of the ISCAgent on this failover member.

– Mirror Agent Port — The port number of the ISCAgent on this failover member. For information, see the

ISCAgent section in this chapter.

• In the This Failover Member section:

– Mirror Private Address — The IP address or host name that the failover members use to communicate with each other for mirroring information and data.

4. Click Save.

4.1.2.2 Configure Second Failover Member

This procedure describes how to configure the second failover member in the mirror.

1. Navigate to the [System] > [Configuration] > [Join Mirror as Failover] page of the Management Portal on the second failover member, and click Join as Failover; if the link is not active, click Enable Mirror Service and select the Service Enabled check box.

2. On the [System] > [Configuration] > [Join Mirror as Failover] page, enter the following information in the Mirror Information

section:

(36)

a. Agent Address on Other System — Enter the superserver IP address or host name you specified when you created the mirror on the first failover member.

b. Mirror Agent Port — Enter the port of the ISCAgent you specified when you created the mirror on the first failover member.

c. Caché Instance Name — Enter the Caché instance name of the first failover member. 4. Click Connect to retrieve and display information about the mirror and the first failover member. 5. Enter or edit, if necessary, the following information in the Mirror Failover Member Information section:

a. In the This System column:

• Mirror Member Name — The name for the failover member you are configuring on the current node (for example, SystemB).

• Superserver Address — The IP address or host name that other systems can use to communicate with this failover member.

• Mirror Agent Port — The port number of the ISCAgent on this failover member. For information, see the

ISCAgent section in this chapter.

• Network Interface for Virtual IP — If the mirror is configured to use virtual IP, select a network interface from the drop-down list (for more information, see the Network Interface Considerationssection of this chapter).

• SSL/TLS Config — The link lets you add or edit the SSL/TLS security; for information, see the Creating and Editing SSL/TLS Configurations for a Mirror section of the “ Using SSL/TLS with Caché ” chapter of the

Caché Security Administration Guide.

Important: If you configured the mirror to use SSL/TLS, this failover member must also be configured to use SSL/TLS to be able to join the mirror.

• Mirror Private Address — The IP address or host name that the failover members use to communicate with each other for mirroring information and data.

b. The Mirror Information for <mirror name> section displays information about the mirror that this member is joining:

• Use SSL/TLS — Whether or not the mirror is using SSL security.

• Quality of Service Timeout (msec) — The maximum time, in milliseconds, that processes on this failover member wait for data to be acknowledged by the other member; the default is 2000 msec.

• Acknowledgment Mode — The acknowledgement mode that the primary failover member uses during a “ Mirror Synchronization ” process from the drop-down list.

• Agent Contact Required for Failover — Whether or not the active backup failover member should take over as the primary failover member if it not able to communicate with the primary system; the default is “ Yes. ”

• Mirror Virtual IP — The virtual IP address if the mirror uses a virtual IP address.

6. Click Save.

To complete the process of joining the mirror, follow the instructions in the Add Second Failover Member to Mirror section of this chapter.

(37)

4.1.2.3 Add Second Failover Member to Mirror

After you have created the mirror and configured the first failover member, and configured the second failover member, on the system on which you created the mirror and configured the first failover member (SystemA), you must add the second failover member to the mirror, as follows:

1. Navigate to the [System] > [Configuration] > [Edit Mirror] page of the Management Portal, and click Edit Mirror. 2. On the [System] > [Configuration] > [Edit Mirror] page of the Management Portal, click the Add Failover Member link

in the Mirror Failover Member Information section.

3. Enter the following in the Mirror Information section in the Add Failover Member to Mirror window:

a. Mirror Private Address — The IP address or host name that the other failover members (for example, SystemB) use to communicate with each other for mirroring information and data.

b. Mirror Agent Port — Enter the port for the ISCAgent on the second failover member (SystemB). c. Caché Instance Name — Enter the Caché instance name of the other failover member.

4. Click Connect to display information in the Failover Member Information section about both failover members in this mirror:

• Mirror Member Name — The name of each failover member.

• Superserver Address — The IP address or host name that external systems can use to communicate with this failover member.

• Mirror Agent Port — The port for the ISCAgent for each failover member.

• SSL/TLS Config — The link lets you add or edit the SSL/TLS security; for information, see the Creating and Editing SSL/TLS Configurations for a Mirror section of the “ Using SSL/TLS with Caché ” chapter of the Caché

Security Administration Guide.

Important: If you configured the mirror to use SSL/TLS, this failover member must also be configured to use SSL/TLS to be able to join the mirror.

• Mirror Private Address — The IP address or host name that the failover members use to communicate with each other.

5. Click Save to save the displayed information and open the Edit Mirror page.

4.1.3 Editing Mirror Configurations

Note: If the Add Failover Member link is displayed in the Mirror Failover Member Information section, go to Add Second Failover Member to First Failover Member in this section.

The following procedure describes how to use this page to edit information about the mirror, as follows:

1. Navigate to the [System] > [Configuration] > [Edit Mirror] page of the Management Portal, and click Edit Mirror. 2. On the [System] > [Configuration] > [Edit Mirror] page of the Management Portal, the following information about the

(38)

• Mirror Private Address — The IP address or host name that the failover members use to communicate with each other.

3. The Mirror Information section displays the following information:

• Mirror Name — The name of the mirror; you cannot edit this field.

• Use SSL/TLS — If you initially configured the mirror to use SSL/TLS security (that is, the drop-down box displays

Yes), you can click Set up SSL/TLS and follow the instructions in the Creating and Editing SSL/TLS Configurations for a Mirror section of the “ Using SSL/TLS with Caché ” chapter of the Caché Security Administration Guide to modify the certificate information..

• Use Virtual IP — You can change whether or not you want to use a Virtual IP address by selecting Yes or No from the drop-down list. If you select Yes, you can add or modify the following information:

– IP Address — Enter an IP address in the text box (for more information, see the Configuring a Mirror Virtual IP (VIP) section of this chapter).

– Mask (CIDR format) — Enter a Classless Inter-Domain Routing (CIDR) mask in the text box.

– Network Interface — Select a network interface from the drop-down list (for more information, see the Network Interface Considerations section of this chapter).

4. Click Advanced Settings to display the Mirror Settings and This Failover Member subsections, and edit pre-populated information about the mirror:

• In the Mirror Settings subsection (for more information, see the Mirror Tunable Parameters section in this chapter):

– Quality of Service Timeout (msec) — The maximum time, in milliseconds, that processes on this failover member wait for data to be acknowledged by the other member; the default is 2000 msec.

– Acknowledgment Mode — Select the acknowledgement mode that the primary failover member uses during a “ Mirror Synchronization ” process from the drop-down list; the default is Received.

– Agent Contact Required for Failover — Select whether or not the active backup failover member should take over as the primary failover member if it not able to communicate with the primary system; the default is Yes.

Note: If you select No, you must provide the ^ZMIRROR user-defined routine (see ^ZMIRROR User-defined Routine in this chapter).

• In the This Failover Member subsection, the following values, which were specified when the mirror was created:

– Mirror Agent Address — The IP address or host name of the ISCAgent on this failover member.

– Mirror Agent Port — The port for the ISCAgent for this failover member.

– Mirror Private Address — The IP address/host name used by external systems, such as ECP application servers, to communicate with the mirror.

5. Click Save.

4.1.4 Adding Async Members to a Mirror

The following procedure describes how to configure async members:

1. Navigate to the [System] > [Configuration] > [Join Mirror as Async] page of the Management Portal and click Join as Async; if the link is not active, click Enable Mirror Service and select the Service Enabled check box.

(39)

2. On the [System] > [Configuration] > [Join Mirror as Async] page, enter the following information in the Mirror Information

section:

• Mirror Name — Enter the same mirror name you specified when you created the mirror (see Creating a Mirror in this chapter).

3. Enter the following information in the Other Mirror Failover Member’s Info section:

a. Agent Address on Other System — Enter the superserver IP address or host name of either failover member. b. Mirror Agent Port — Enter the port of the ISCAgent you specified when you created the mirror on the specified

failover member.

c. Caché Instance Name — Enter the Caché instance name of the other failover member whose IP address/host name you specified in Agent Address on Other System.

4. Click Connect to retrieve and display information about the async member. 5. Enter the following information in the Async Member System Info section:

a. Async Member Name — Enter a name for the async member you are configuring (for example, ASYNC1). b. Async Member System Type — Select one of the following types from the drop-down list:

• Disaster Recovery — This option is for systems configured as disaster recovery systems, where all mirrored databases are read-only.

Important: A disaster recovery async member must have direct TCP/IP connectivity to the failover members and be on the same VLAN for proper virtual IP address reassignment. When the async member is not in the same data center as the failover members, a VLAN subnet can be extended across the two data centers to continue supporting the same virtual IP address. This requires proper Layer 2 connectivity between the two sites, the use of appropriate pro-tocols such as IEEE 802.1Q for efficient operations, and support for Inter-Switch Links (ISL) for high availability. Another possible solution involves the use of hardware-based site selectors (for example, Cisco Global Site Selectors or f5 BIG-IP Global Traffic Manager), which can direct incoming traffic to a single fully qualified domain name to separate IP addresses within multiple data centers. Consult your organization’s network administrators for information about available technical options.

• Read-Only Reporting — This option is for systems where new mirrored databases are set to read-only by default.

• Read-Write Reporting — This option is for systems where new mirrored databases are set to read-write by default. It is for systems configured as reporting/business intelligence systems, where it is possible to modify the data during analysis.

c. Use SSL/TLS — Specify whether or not you want to use SSL/TLS security by selecting Yes or No from the drop-down list; if you select Yes, click SSL/TLS Config and follow the instructions in the Creating and Editing SSL/TLS Configurations for a Mirror section of the “ Using SSL/TLS with Caché ” chapter of the Caché Security

d. SSL/TLS Config — The link lets you add or edit the SSL/TLS security; for information, see the Creating and