High Availability for Microsoft Servers

(1)

1

High Availability for Microsoft

Servers

Brendan Murphy

IFIP, New York

June 2000

(2)

2

Agenda

¥ Windows 2000 dependability.

¥ Tomorrows problems.

(3)

3

Dependability Goal Setting.

From the customers perspective.

¥ Win 2k dependability > NT4.

Irrespective of additional new feature.

OpenVMS Metrics (running on VAX Systems) Post Installation Behaviour

0 10 20 30 40 50 60 70 80 90 100 V5.5 V5.5-1 V5.5-2 V6.0 V6.1 V6.2 V7.0 V7.1

Operating System Version

Rate Of System Outages

Upper Confidence Bound Average

Lower Confidence Bound

VAX Servers: ©FTSC 1999 Madison, Murphy, Davies, Compaq. VAX Servers: ©FTSC 1999 Madison, Murphy, Davies, Compaq.

Note reliability improvements

between each Ôservice packÕ

release.

Note drop in reliability

between Ôservice packÕ and

major release.

Exception

(4)

4

Making NT Measurable

Logging System Events

¥ Capturing time and type of outage.

Ð 6006 = clean shutdown.

Ð 6008 = blue screen.

Ð 6005 = Time of reboot.

¥ Capturing OS version and upgrade.

Ð 6009 = OS version on reboot.

Ð 4353 = installation of service pack.

¥ Capturing cause of system crash

Ð 1001 = brief description of crash details.

(5)

5

NT4 Failure Classes

Source: Sample from PSS Incidents: NT Server 4.0 8/5/96-4/7/98

Device

drivers

16%

Core NT

43%

Other

third-party drivers

party drivers

16%

Anti-virus

12%

Hardware

Failure

13%

(6)

6

Windows 2000 Bluescreen Reduction

Anti-virus

n

Anti-virus dev labs

n

Driver verifier

n

Better DDK

Core NT

n

Kernel verifier

n

Driver verifier

n

Full time source code reviewers

n

Better

longhaul

testing

n

Better component stress

n

PREfix

source scanning

Hardware failures

n

Hardware

compatibility list

Device drivers

n

Broader device

test matrix

n

Driver verifier

n

Better DDK

Other third-party drivers

n

File system dev labs

n

Driver verifier

n

(7)

7

NT4 Server Reboot Causes

OS install

27%

Hardware

install/

config

3%

Preventative

reboots

20%

OS

Configuration

7%

Application

install/

config

8%

Application

Failure

21%

System failure

14%

35% of outages

are unplanned

65% of outages

are planned

Source: One site, 1,180 servers, 9/1/98-5/7/99, SP4

(8)

8

Windows 2000 Reboot Reduction

Preventative reboots

n

Published best practices

n

Resource Partitioning

n

IIS restart

System Failure

n

Bluescreen reduction

Unplanned reboot reduction

Planned reboot reduction

App install/

config

n

Windows file protection

n

Windows installer

OS configuration

n

Eliminated dozens

of configuration

reboots

OS install

n

Service pack

slipstreaming

Hardware

install/

config

n

Plug-n-Play

App Failure

n

Resource Partitioning

n

Task mgr Òkill proc treeÓ

n

IIS restart

n

(9)

9

Windows 2000 Verification

Problem

Windows 2000

Verification

Features

Bug Fixes

_{+ features}

Test

Development

Field Tests

(Beta Testing)

38 packs/working day

288 variants/working day

7000+

>5000

Customer

Servers

5000+ Supported

Computers

Application

Test & Development

(10)

10

Testing Process.

Incremental Improvement

Daily Builds

Weekly Builds

RC/Beta Builds

RTM

(11)

11

Windows 2000

Failure Analysis.

Drivers for HCL HW 7%

Drivers for NonHCL HW 20% HW Failure 22% Anti-Virus 4% System Config 34% Other 3rd Party Kernel code 11% MSInternalCode 2% Other IFSDrivers 0%

Device

drivers

16%

Core NT

43%

Other

third-party drivers

party drivers

16%

Anti-virus

12%

Hardware

Failure

13%

Source: Sample from PSS Incidents:

(12)

12

Future/current dependability

issues

¥ Scalability

Ð Vertical

¥ Complex Faults.

¥ Change control.

¥ Application configuration management.

Ð Horizontal

¥ System Management.

¥ Shared data.

(13)

13

Future/current dependability

issues contÉ

¥ Measurement and characterization.

Ð System not the metal box.

¥ Measure MSN behaviour?

Ð Relate to reality.

¥ Not 99.999% availability claims.

Ð Drive correct behaviour.

¥ For both developers and customers.

(14)

14

Future / current dependability

issues É. cont

¥ Size.

Ð Geographic solutions.

Ð Reassess failure rates.

Failure rates per TB of memory.

¥ Real time change management.

¥ Cost of ownership.

Ð Decrease skill and number of system managers.

(15)

15

Future / current dependability

issues É. cont

¥ Lack of hierarchy.

Ð Storage, computer, network, applications.

¥ The system is not the computer.

Ð MSN gets a rack of computers every 2 weeks.

¥ Mobiles.

Ð Security/trust.

Ð Synchronization.

(16)

16

Microsoft Research

Development tools

¥ Code Analysis.

Ð Generic analysis applied to all new code.

¥ Decrease false positives.

Ð Automating verification of best practices(e.g. driver

development).

¥ Driver verifiers.

Ð Fault injection(ish).

Ð Software development environments.

Ð Using OS as test environment.

Ð Objective is to protect the operating system.

(17)

17

Testing

$162 million problem and rising

¥ Testing.

Ð Virtual labs.

Ð Specific labs

Ð Test development.

Ð Attribute testing.

Ð Measurements.

Ð Release criteria.

Ð Motivation is $$$$$$$$

(18)

18

Failure predictions

¥ Hardware failure prediction methodologies

applicable to software failures?

¥ Analysis of SQL and exchange using NT

logs.

¥ Apply to other applications.

(19)

19

System Fault Management.

¥ Assume no hierarchy.

¥ Assume limited knowledge between

different elements (i.e. abstraction).

¥ System is not a metal box.

¥ Need to work under failure conditions.

¥ Pattern recognition time based.

Ð Pattern variations based on system speed.

(20)

20

Conclusions/Issues

¥ Dependability is improving.

Ð In spite of using ÔwrongÕ methods?

¥ Industry has problems and research have solutions.

Ð But what is the relationship?

¥ Industry addressing current problems through

innovation and brute force.

Ð Opportunities to address future issues (risky predicting

the future).