ase120 choosing the right high availability solution

(1)

ASE120: Choosing the Right High

Availability Solution

Chris N. Brown

Principal Systems Consultant [email protected]

(2)

Agenda

 Intro: Cutting through all the hype  _{Why clustering is not enough}

 Ways to achieve HA

 _{Physical Copy}  _{Logical Copy}

 Client-Side High Availability

 _{OpenClient 12.x}  _OpenSwitch  _{DNS Update}

 DBA administration in a 24x7 environment  Summary

(3)

HA – Do you need it?

 HA (High Availability) has evolved almost into an industry buzzword

 _{Everyone talks about it}

 _{Everyone says that they can do it}

 _{Everyone wants to sell you a “solution”}

 _{... but in the end, what are they really offering you?}

 HA isn't something new, it's been around in one form or another for

years

 _{Now seen as something critical because of heavy reliance on computing}

systems for business critical processes

(4)

And That's Why You're Here!

 We are going to answer that question in the next 90 minutes  _{Examine what's out there}

 Analyze what is being offered (both Sybase and non-Sybase)  Discuss how they work

 _{... when they are appropriate}  _{... when they aren't appropriate}

 _{Talk about how to make them all work together}  And address the <gasp!> users out there as well  Let's cut through the Bull and make it simple

 _{So standby for some business level discussion... but it sets the}

(5)

So do you need it?

 Before you embark down the HA path....

 There are some questions that you should ask...

 _{And know the answers to!}

 Many times, people THINK they need HA when they really don't.

 _{As a result, thousands (and even millions) are spent that don't need to be}

 _{What about the man hour cost?}

 _{What about the increased administration?}

(6)

Think about it....

 What you are buying is an “insurance policy”.

 _{A protection from incurring a loss.}

 This is very similar to an auto insurance policy.

 _{You can have high deductibles or low ones}

 _{You can have liability only or full coverage}

 _{You can choose the level of protection}

 _{You might be covered if you hit a Yugo, but will you be covered if you hit a}

Mercedes or Jaguar or Bentley?

 _{How much of an out-of-pocket loss are you willing to take?}

 The same principle is true with an HA architecture.

 _{Decide how much loss (downtime and cost) is acceptable.}

 _{Architect around THAT}

(7)

How critical is the system in question?

 The first thing to ask is... how critical is the system?

 How much does it cost if it goes down?

 _{How long could your company operate without it?}

 _{How much would your company lose if it went down?}

 _{... and for how long?}

 _{Sometimes the costs are intangible}

 SLA's (Service Level Agreements) are usually put into place and

should take this question into account.

 _{Blanket SLA's sometimes used, not always prudent}

 _{Some systems (billing, customer service) are more important than others}

(email, instant messaging, LAN-based fileserver).

(8)

How much can you spend?

 Highly Available Systems can cost from $ to $$$$$$$

 _{That incremental .09% or .009% can hike the cost exponentially.}

 _{Big derailer of HA implementations.}

 _{It's extremely important to understand the business requirements as}

discussed earlier.

 _{I.E., does the system REALLY have to be 24x7 or can it be 25x5 or 9x5?}

 _{This will dramatically change the artchitecture chosen and of course, the cost of} implementation.

 _{Raw system cost should not (completely) drive the architecture}

(9)

Hardware Redundancy Hardware Redundancy RAID/Mirroring/HW Cluster RAID/Mirroring/HW Cluster Cold Standby Cold Standby Backup/Restore Backup/Restore Warm Standby Warm Standby Database Replication Database Replication Automatic Failover Automatic Failover DBMS HA DBMS HA Continuous Operations Continuous Operations Online Maintenance Online Maintenance

High Availability

Continuous

Availability

High Availability Levels

(10)

Agenda

 Intro: Cutting through all the hype  _{Why clustering is not enough}

 Ways to achieve HA

 _{Physical Copy}  _{Logical Copy}

 Client-Side High Availability

 _{OpenClient 12.x}  _OpenSwitch  _{DNS Update}

 DBA administration in a 24x7 environment  Summary

(11)

Clustering : The traditional solution

 For many years, the traditional

HA solution was hardware level clustering.

 Generally what most IT

professionals think of when you say “HA”.

 _{Mainly addresses hardware failures.}  _{Evolved now to watch for process}

failure and will re-start them.  _{When failure detected, tries to}

restart services on a redundant host “as fast as possible”

(12)

Why isn't clustering enough?

 In today's computing environments,

hardware level redundancy isn't always enough.

 _{It only provides the foundation.}

 _{Sometimes the amount of time}

required to restart services is unacceptable

 _{Can take minutes when seconds}

are required ($$$$$).

 What if the problem is with a

shared resource?

(13)

Agenda

 Intro: Cutting through all the hype  _{Why clustering is not enough}

 _{OpenClient 12.x}  _OpenSwitch  _{DNS Update}  ₃rd_party

 _{DBA administration in a 24x7 environment}  Summary

(14)

How can I achieve HA (50,000 ft view)?

 There are 2 main ways to achive HA from a database level:

 _{Physical Database re-creation}  _{Logical Database re-creation.}

 _{They range from the simple to the down and dirty to the complex.}

 _{Which one you use depends on your requirements}  _{... and your budget}

 _{... and your level of risk}

 _{In the 'ideal world', a combination of these strategies provides the}

(15)

Physical Database Recreation: Dump and Load

 This is the easiest way to get a

backup copy and VERY basic HA

 Dump the primary DB, and then load

it to an ASE running on another host.  _{Keep in sync via incremental transaction}

log dumps

 _{Inexpensive to implement}

 Issues:

 _{Size of dump and getting it to backup site.}  _{What if a dump is corrupt?}

 _{What if tranlog loads get out of sync?}  _{Usually manual; can be automated but}

requires some 'babysitting' and has quite a few moving parts.

 _{Good for non-critical systems or}

those whose data does not change much (think Log Size).

(16)

Physical Database Recreation: Quiesce Database

 If you have a SAN, another easy way to achieve basic HA is via the

'quiesce database' functionality.  _{New feature of ASE 12.x}

 _{Works best with ASE 12.5 and higher}

 _{Similar in principle to dump and load, but much faster.}

 _{Quiesce DB suspends writes to databases, so that underlying}

devices can quickly be copied  _{Read still allowed}

 _{Initally targeted for:}

 _{Quick refreshes of production to development}  _{Quick creation of DSS environment}

 _{Quick troubleshooting of production with a 'snapshot'.}

 However, customers wanted to use it for more of an HA solution

(17)

Primary Secondary

2:00 AM

quiesce database hold; <copy database using external command>; quiesce database release

7:00 AM

dump tran with standby_access

9:00 AM

10:00 AM

Repeat each hour

until activity tapers off; then lengthen intervals accordingly 2:10 AM dataserver-q .. 7:05 AM load tran;

online database for standby_access

9:07 AM

load tran;

10:10 AM

load tran;

(18)

Things To Think About With Quiesce Database

 Quiesce Datbase is a solution to a specific problem.

 _{Can be very very fast}

 _{True physical copy so WYSIWYG. But that may not be what you want}

 _{HA-ish solution with tranlog loads works best with ASE 12.5.x}

 _{You can do maintenance on replicate copy (dbcc, etc)}

 _However...

 _{It's a physical copy}

 _{Dependent on tranlog loads}

 _{Can't really use replicate since users must be kicked off for tranlog load to}

occur

(19)

Physical Database Recreation: Block Replication

 This is something that is offered by

SAN vendors.

 Very attractive:

 Copies data from one area in

SAN to another

 Copies data from one SAN to

another.

 Often times pitched as an HA or DR

solution, WHICH IT CAN BE.

 Operates in 2 modes: synchronous

and asynchronous

 Because it is a block-level copy, what

exists on the primary will exist on the replicate.

(20)

Block Copy – How It Works (sync)

 Methodology:

 _{The Host OS will write its I/O to the}

primary SAN cache

 _{The Primary cache copies its I/O to the}

secondary SAN cache

 The secondary SAN sends an ACK to

the primary SAN that it received the I/O

 Both the primary and the secondary then

write their I/O to disk

 _{In this case, every disk I/O is copied.}

 This is similar to RAID-1 or a variation of a

2-phase commit.

 The standby server (ASE in this case) can

restart at the same spot that the primary ended.

 Used over shorter distances.

(21)

Block Copy – How It Works (async)

 _Methodology:

 _{The primary OS writes its I/O to the}

primary SAN cache

 _{The primary SAN sends an ACK to the}

primary OS that it received the I/O.

 The primary SAN copies that I/O to the

secondary SAN cache

– Here is where it gets tricky

– Not every I/O is copied

» The block could have changed many times

» Changed blocks are 'scored' and the latest change is what is sent over.

 _{Both SAN's write to disk}

 _{Think of the replicate as a point-in-time}

(22)

This sounds like A Great Thing! (TM)

 Since the SAN is copying data at the bit level, it makes sense as a

DR / HA mechanism  _{No data loss}

 _{Server can be restarted where the other once crashed}  _{Copy from primary to secondary is usually very fast}

 _{There are some issues to be aware of though.}

 _{Sometimes, what you see on disk isn't what you want at the replicate} (corruption)

 _{Be aware of how ASE writes data to disk, and how the OS writes data} – We write 2k (4k, 8k, 16k) pages, they write 512k Blocks

– We log first then write the data for consistency, so what happens if data pages are written in the SAN before the log pages are, and you go down? (eeeeek)

 Overall, this is a good stragegy that many people use, but it cannot

(23)

ASE HA Option: Riding The Clustering Wave

 ASE 12.0 introduced a new feature we call the HA option.

 _{It brings clustering technology to the database server.}

 _{No logical IP needed for the ASE to 'listen on'.}

 _{Failover designed to be very very fast.}

 You can utilize both nodes in a 2 way cluster (prev. one was usally idle)

 _{This results in better leverage of your hardware investment and can make}

multiple systems highly available with less cost.

(24)

Disk

HA System

S2

Replicate Users/Logins

S1

_CompanionEstablish

Node 1

Node 2

Shared Disk Storage

Disk

(25)

Disk

HA System

S2

Node 2

Disk

Fail Over

(26)

Disk

HA System

S2

Node 2

Disk

Fail Back

Prepare

(27)

Disk

HA System

S2

Replicate Users/Logins

S1

_CompanionEstablish

Node 1

Node 2

Disk

Fail Back

(28)

Some notes on the HA option

 We rely on the HA “Heartbeat” to notify us when one ASE fails.

 _{Brings up several administration aspects}

 Both ASE's must be at the same version

 Currently we only support 2-node failover

 _{One of the 2 ASE's must be a fresh install}

 It's possible to access data from one server on another

 _{Via proxy tables, this is done via CIS}

 _{Performance issues to consider}

 _{Might be a feasible load-balancing option}

 We failover fast (since that's unplanned) but failing back is unplanned

and manual (and slower).

 _{Significant improvements in this area since the 12.0 release.}

(29)

Logical Database Recreation

 So far, we have only discussed ways of re-creating the database

server “physically”

 _{Meaning, copying the data (disk, devices, dumps) from point A to point B}

 All of these work well and in some cases work very fast

 _{They provide near zero or zero data loss}

 However, they all suffer from the same common drawbacks

 _{What you see is What you get (WYSIWYG)}

 _{Corruption is almost always copied over, making backup copy useless.}  _{You cannot change the data as it is being moved over}

 _{In most cases, the replicate is down or not useable.}

 _{The only way to get around these problems today is to use a logical}

database recreation scheme.

(30)

Quickies on Queues and 3

rd

_parties

 We will quickly discuss message queueing and 3rd_parties.

 _{Message queueing takes “events” and publishes them out on a bus}

 _{The event could be a data event or an application level event}  _{A listener subscribes to certain events}

 _{Data can be manipulated based on rules.}

 _{There are 3}rd_{party products out there that can also do this}

 _DataMirror  _UPSuite

 _{They may not use log-based replication though ... some use}

(31)

Relication Server Architecture

Replication Agent

Primary Data Server

1

2

3

4

Replicate Data ServerReplicate Data Server

Replication Server

Client Applications

1) The client application updates data on the primary.

2) The primary data server manages its local data.

3) Replication Agent notifies Replication Server of primary server data updates.

4) Replication Server coordinates data replication of those updates with other

Replication Servers.

(32)

How Replication Server Works

LTL

Replication Agent

•Monitors Transaction LogMonitors Transaction Log •Truncation PointTruncation Point

•Marked TablesMarked Tables •Creates LTLCreates LTL

RSSD

•Rep-Defs Rep-Defs •PublicationsPublications •SubscriptionsSubscriptions •RoutesRoutes

Stable Device / Stable Queues

•Inbound QueueInbound Queue •Outbound QueueOutbound Queue •Materialization QueueMaterialization Queue Primary Data Server

Primary Data Server

Replicate Data Server

Replicate db

LAN/WAN – DSI (Data Srv Int)

Primary db

(33)

Physical ASE “A”

Logical ASE “XYZ”

Physical ASE “B”

(IP Address 192.233.56.20)

(IP Address 192.233.56.20) (IP Address 192.233.56.21)(IP Address 192.233.56.21)

(34)

Some notes on Replication Server

 Warm Standby is a variant of “traditional replication”

 _{You can replicate DDL changes if you replcate at a database level}  It can be tuned to near zero latency

 Better to have the RepServer on its own host or on the replicate

host.

 _{Beware of failure points and how they might affect your application.}  The primary and the secondary must be controlled by the same

RepServer

 Currently limited to 1 primary, 1 warm standby (will change in

(35)

Agenda

 Intro: Cutting through all the hype  _{Why clustering is not enough}

 _{OpenClient 12.x}  _OpenSwitch  _{DNS Update}

(36)

What about the client?

 Often times, HA solutions only include the back-end.

 _{Archtectures consider only how quick we can recover the downed}

system, but what about the end user?

 Some questions to ponder:

 _{How is uptime and availability measured?}

 _{If the system was down for 5 minutes but the user couldn't connect for 30,}

how long was the outage?

 _{What if the system were down, but the user didn't really know or}

notice?

 _{It's possible today!}

(37)

Method #1: OpenClient 12.x

 OpenClient 12.x integrates with

the HA option of ASE.

 Provides client-side failover

from the failed ASE server to the surviving ASE server.

 _{ONLY useful if you are using}

the HA option.

 ONLY can be used if you can

recompile your applications against OpenClient 12.x

Primary Server

Companion Server Primary

Server

Companion Server

(38)

OpenClient 12.x and HA: How Does It Work?

 To support this feature, 2 things need to be done

 _{The First thing is change the interfaces file.}

 _{Typical entry would contain master/query syntax and connectivity info}  _{A new entry is added in the interfaces file at the end}

 _{It indicates what server is the failover (companion) server for a primary node}

 For example:

ASTRO

master tcp ether stewie 5000 query tcp ether stewie 5000 hafailover ELROY

ELROY

master tcp ether felix 5000 query tcp ether felix 5000

 If using LDAP, would add an entry to the LDAP server containing the

same information

(39)

OpenClient 12.x and HA: How Does It Work (Cont'd)?

 The second thing that needs to be done is re-compile against

OpenClient 12.x

 There is a new property that need to be addressed to utilize the HA

functionality

 _{CS_HAFAILOVER}

 _{CS_RET_HAFAILOVER}

 These are set using the ct_config and ct_con_props syntax at the connection or context level

 _{This is only with ctlib (dblib}_{DOES NOT}_{support this functionality)}

 Client will receive an error 1205

 _{Client failed over to server listed as hafailover server in the}

interfaces file

(40)

Method #2: Sybase OpenSwitch

 Much more flexible than OpenClient 12.x

 Does not require recompile of

applications

 _{Not tied to HA option of ASE}

 Can be used against existing and 3rd

party applications

 Allows for increased flexibility and user

management.

 User logs directly into OpenSwitch, not

into ASE.

 OpenSwitch manages the user

connection and migrates them when it detects a 'failure'.

 _{Integrates with Business Logic.}

(41)



For each incoming connection OpenSwitch

• _{decides where it should go and opens up a new connection}



Manual switch capability

Transparent Connection Management

OpenSwitch

ISQL

PowerBuilder

Any Open Client Application and Platform

EAServer

Administrator (ISQL) RPC Switch

Request

(42)

 Coordination Module provides an API to coordinate with third party HA

solutions

 This is the “brains” of OpenSwitch

 _{OpenSwitch defers switching decision to CM if present}

HA Coordination

OpenSwitch

ASE Server A

Application

What do I do? Response

Action

C

M

C

(43)

Typical OpenSwitch Usage Scenario

New York New York OpenSwitch OpenSwitch Denver Denver

CM

Connection Lost! What do I do?

Check if transactions are pending in Rep Server

queue

OK, Failover

Application

Check if it is a real failure or a network hiccup

Check if warm-standby is really up and functional

CM = Coordination Module

(44)

The “Pie In The Sky”

Via CM

 This covers all possible areas: physical, logical, and users (and

(45)

Agenda

(46)

24x7 DBA Administration

 Any HA scenario MUST allow for DBA maintenance activities

 _DBCC

 _{Dump and Load}  Update Statistics  _{Reorg Rebuilds}  Upgrades

 If it doesn't, then by definition it's not hightly available

 _{Simply because to do any of the above actions, you have to take the server} down

 ... or you might impact performance so much that the system beomes 'pseudo-down'.

(47)

Well....

 Update stats: always the “achilles heel”

 _{Attend Eric Miner's ASE 126 class on speeding up Update Stats}  _{Thursday 3:30pm, 90 minutes, Sun “A”}

 _DBCC

 _{Use a phyusical database recreation scheme (bit level rep, quiesce, etc)}  _{Then run DBCC on the copy}

 _{Since it's a physical recreation, then errors in the copy will be in the primary,} then you can take action accordingly to fix it.

 Reorg

 _{Starting with ASE 12, you can specify parameters around it's use}  _{Done at an extent level, doesn't lock the entire table down.}

 _{Dump and Load}

(48)

Well... (cont'd)

 Upgrades

 _{Best done with a logical database recovery scheme (like RepServer, etc)}  _{This will allow you to keep both the old and vew version in-sync and you can}

(49)

Agenda

(50)

HA Questions To Ask....

 How does HA solution cover …

 _{… host machine failures}  … operating system faults

 … database failures/corruptions  _{… datacenter loss}

 … online maintenance

 _{... DBMS, OS, Database Schema, User Admin}

 How well does it handle …

 … latency between synchronization  … outage during failover

 _{… client connections}

(51)

SDN Presents CodeXchange

 New SDN feature enables community collaboration

 _{Download tools created by Sybase}

 _{Leverage contributions of others to help administer and monitor your servers}  _{Contribute your own code or start your own collaborative project with input}

from other ASE experts

 Any SDN member can participate

 _{Log in using your MySybase account via SDN}

 Join the collaboration already underway

 _{http://ase.codexchange.sybase.com or via SDN at} www.sybase.com/developer