Algorithms and Methods for Distributed Storage Networks 2. Hard Disks Christian Schindelhauer

(1)

Albert-Ludwigs-Universität Freiburg

Institut für Informatik

Rechnernetze und Telematik

Wintersemester 2007/08

Algorithms and Methods for

Distributed Storage

Networks

2. Hard Disks

(2)

Hard Disks

‣

History

• Capacity and Access Speed

• Prices

• Form factors

‣

Construction and Operation

• Mechanics

• Storage technology

‣

Low-Level Data Structures

• Encoding, Decoding

• Tracks , Cylinders

• LBA

‣

Interfaces

• ATA, SATA

• SCSI, SAS

• Fibre-Channel

• eSATA

‣

Lifetime and Disk Failures

• Error Management and Recovery

• Types

• S.M.A.R.T.

• Counter methods

‣

Special Issues

• Sound avoidance

• Data security

(3)

Distributed Storage Networks Winter 2008/09

Rechnernetze und Telematik Albert-Ludwigs-Universität Freiburg Christian Schindelhauer

History

Hard Disks

(4)

(5)

History

5 ‣

1956 IBM invents 305 RAMAC

(Random Access Method of

Accounting and Control)

• 5 MBytes, 24 in

‣

1961 IBM invents air bearing heads

‣

1970 IBM invents 8 in floppy disk

drives

‣

1973 IBM ships 3340 Winchester

sealed hard drives

• 30 MBytes

‣

1980 Seagate introduces 5.25 in hard

disk drive

• 5 MBytes

‣

1981 Sony ships first 3.25 in floppy

drive

‣

1983 Rodime produces 3.25 in disk

drive

‣

1986 Conner introduces first 3.25 in

voice coil actuators

‣

1997 Seagate introduces 7,200 RPM

Ultra hard disk

‣

1996 Fujitsu introduse aero dynamic

design for lower flighing heads

‣

1999 IBM develops the smallest hard

disk of the World 1in (340 MB)

‣

2007 Hitachi introduces 1 TB hard

(6)

age had significantly undercutfilm and paper costs by the late 1990s. It is obvious that this is a principal advantage to the growth and popularity of on-line (i.e., nonarchival) magnetic storage.

Concurrently, there has been an accompanying in-crease in the quantity ofDRAMfound in most

stor-age subsystems. Whenfirst introduced in the mid-1980s, a system cache of 8 megabytes was considered large. Today, a system cache of 8 gigabytes is con-sidered relatively small. The corresponding decrease

in price-per-storage capacity of DRAM, although

prices started at a higher level than magnetic hard disk drives, has enabled larger caches while still main-taining a price reduction per capacity at the system level, as Figure 6 indicates.

Form factor miniaturization

Figure 8 displays form-factor miniaturization for disk drives for the time period 1956–2002. Large

form-Figure 5 Storage system volumetric density trend

V O L U M ET R IC D EN S IT Y , G B IT S /C U B IC IN C H 100 10–1 10–2 10–3 10–4 10–5 101 1980 1985 1990 1995 2000 2005 2010 PRODUCTION YEAR RANGE OF POSSIBLE VALUES SYSTEMS W/SMALL FF DRIVES SYSTEMS W/LARGE FF DRIVES

Figure 6 Cost of storage at the disk drive and system level

P R IC E /M B Y T E , D O LL A R S 100 10 1 0.1 0.01 0.001 0.0001 1000 1980 1985 1990 1995 2000 2005 2010 PRODUCTION YEAR DESKTOP HDD HIGH PERFOR-MANCE, SERVER HDD INDUSTRY PROJECTION FROM MULTIPLE SOURCES SYSTEMS W/LARGE

FF DRIVES SYSTEMS W/SMALLFF DRIVES

SMALL FF HDD STORAGE SYSTEM

Figure 7 Cost of storage for disk drive, paper, film, and semiconductor memory P R IC E (D O LL A R S P E R M E G A B Y TE ) 100 10 1 0.1 0.01 0.001 1000

PAPER AND FILM

3.5 INCH 2.5 INCH SEMICONDUCTOR MEMORY (DRAM AND FLASH) DISK

DRIVES Distributed Storage Networks

Winter 2008/09

History of Hard Disk Prices

6 Technological impact of magnetic

hard disk drives on storage systems,

Grochowski, R. D. Halem

(7)

Construction and

Operation

Hard Disks

7

(8)

Construction of a Hard Disk

(9)

Construction of a Hard Disk

9

(10)

Physical Components

‣

Platters

• round flat disks with special material

to store magnetic patterns

• stacked onto a spindle

• rotate at high speed

‣

Read/Write Devices

• usually two per platter

• Actuator

- old: stepper motor

✴

mechanic adjusts to discrete

positions

✴

low track density

✴

still used in floppy disks

- now: voice coil actuator

✴

servo system dynamicall

positions the heads directly

over the data tracks

- Head arms

✴

are moved by the actuator to

choose the tracks

- Head sliders

✴

are responsible to keep the

heads in a small defined

distance above the platter

✴

heads „fly“ over the platter on

an air cushion

- Read/write heads mounted on top

of arms

(11)

Slider

11

Figure 6. Illustration of suspension and slider. Left: schematic. Right: photograph. (Source: Tom

Albrecht, IBM)

coil

/

inductive head

MR element

Figure 7. Schematic of readwrite transducer. (Source: Tom Albrecht, IBM)

41 6

Authorized licensed use limited to: Christian Schindelhauer. Downloaded on October 19, 2008 at 06:24 from IEEE Xplore. Restrictions apply.

Proceedings of the American Control Conference ,Arlington, VA June 25-27, 2001

A Tutorial on Controls for Disk Drives William Messner , Rick Ehrlich

(12)

Magnetization Techniques

‣

Longitudinal recording

• magnetic moments in the direction of

rotation

• problem: super-paramagnetic effect

• 100-200 Gigabit per square inch

‣

Perpendicular

• magnetic moments are orthogonal to the

rotation direction

• increases the data density

• 1 Terabit per square inch

‣

HAMR (Heat Assisted Magnetic Recording)

• upcoming technology

• Laser heats up area to keep the necessary

magnetic field as small as possible

(13)

Electronic Components

‣

Magnetized Surface on platter

‣

Read/Write-Head

‣

Embedded controller

‣

Disk buffer (disk cache)

• store bits going to and from the platter

• read-ahead/read-behind

• speed matching

• write acceleration

• command queueing

‣

Interface

13

(14)

Low Level Data

Structure

(15)

Tracks and Cylinders

‣

Tracks

• is a circle with data on a platter

‣

Cylinder

• is the set of tracks on all platters

that are simultaneously

accessed by the heads

‣

Sector

• basic unit of data storage

• angular section of a circle

15

(16)

Addressing

‣

CHS (cylinder, head, sector)

• each logical unit is addressed by the cylinder

- set of corresponding tracks on both sides of the

platters

• head

• sector (angular section)

• old system

‣

LBA (Logical Block Addressing)

• simpler system all logical blocks are number

• the translation to CHS is

(17)

Adapting Sectors

‣

Zoned bit recording

• adapt the sector size to the bit sensity

• different number of sectors depending

from the distance from the center

‣

Sector interleaving

• for cylinder switch

• when the arm moves then the disk

continues spinning

• to avoid waiting times the numbering

of the sectors has an offset

17

http://www.storagereview.com/guide2000/ref/hdd/geom/ tracksZBR.htm

l

(18)

Sector Format

‣

A sector is the atomic data unit of an hard disk

‣

No absolute position

• must be identified from its contents

‣

• ID Information (number and location)

• Synchronization fields

• Data

• ECC: Error correcting codes

• Gaps

(19)

Formatting

‣

Low-level formatting

• creates the physical structures (tracks, sectors, control

information)

- starts from empty platter

- map out bad sectors

‣

Partitioning

• divides the disk into logical pieces (i.e. hard disk volumes)

‣

High-level formatting

• logical structures for the operating-system level

components

(20)

Encoding

‣

Problem

• Only the difference of orientation can be measured

• Because of the para-magnetic effect orientation

changes need a minimum distance

• Long sequences of same orientation lead to errors

‣

Encoding

(21)

MFM

‣

R: Flux reversal

‣

N: no flux reversal

‣

FM (Frequency Modulation)

• 0 -> RN

• 1 -> RR

‣

MFM (Modified Frequency Modulation)

• 0 (preceded by 0) -> RN

• 0 (preceded by 1) -> NN

• 1 -> NR

(22)

Run Length Limited (RLL)

(23)

Partial Response, Maximum

Likelihood (PRML)

‣

Peak detection by analog to digital conversion

• use multiple data samples to determine the peak

• increase areal density by 30-40% to standard peak detection

‣

Extended PRML

• further improvement

of PRML

23

(24)

Interfaces

(25)

ATA (AT Attachment)

‣

Parallel connection standard, a.k.a.

P-ATA

‣

evolves since 1994

• ATA-1: 1994-99, up to 8.3 MB/s

• ATA-2: 1996-01

- PCMCIA connector

• ATA-3: 1997-02

- introduces S.M.A.R.T.

• ATA-4: 1998-,

- supports CD-ROM, tape, etc.

- features for solid state drives,

- hidden protected area

✴

hidden against OS

• ATA-5: 2000-, up to 66 MB/s

• ATA-6: 2002-, up to 100 MB/s,

automatic acoustic management

• ATA-7: 2005-, SATA 1.0, up to 150

MB/s

• ATA-8, in progress

(26)

SATA – Serial Advanced

Technology Attachmnet

‣

Serial ATA

‣

Computer bus designed for transfer of data between

motherboard and mass storage device

• faster transfers than P-ATA, designed as successor

• allows removing and adding devices while operating

(hot swapping)

‣

Evolution:

• SATA 1.5 Gbit/s

• SATA 3.0 Gbit/s

• SATA 6.0 Gbit/s

(27)

SCSI

‣

Small Computer System Interface

• offers higher data rates than SATA

• hides the complexity of physical

format

• peripheral interface: 8/16 devices

can be attached to a single bus

• buffered interface

‣

Evolution of Parallel SCSI

• SCSI: 1986, 5 MB/s

• Fast SCSI: 1994, 10 MB/s

• ...

• Ultra SCSI: 1999, 160 MB/s

• Ultra-320 SCSI: 2002, 320 MB/s

• Ultra-640 SCSI: 2003, 640 MB/s

‣

Evolution of Serial SCSI

• SSA: 1990 40 MB/s

• SAS: Serial Attached SCSI, 300 MB/

s

‣

SCSI-Fibre Channel interface

• FC-AL 1Gb: 1993 Fibre Channel 100

MB/s

• FC-AL 2Gb: Fibre Channel 200 MB/s

• FC-AL 4Gb: Fibre Channel 400 MB/s

• length 500m / 3km

(28)

Other Interfaces

‣

eSATA (since 2004)

• variant of SATA for consumer market

• maximum cable length of 2m

‣

USB (Universal Serial Bus)

• allows hot swapping

• 12 or 480 MBit/s

‣

Firewire (IEEE 1394 interface)

• serial bus interface standard

• Firewire 400: ~100/200/400 MBit/s half-duplex

• Firewire 800: 786 MBit/s full-duplex

(29)

Lifetime and Disk

Failures

Hard Disks

(30)

Disk Failure Rates

‣

Failure Trends in a Large Disk Drive Population, Pinheiro,

Weber, Barroso, Google Inc. FAST 2007

30 raw numbers, are likely to be good indicators of

some-thing really bad with the drive. Filtering for spurious

values reduced the sample set size by less than 0.1%.

3 Results

We now analyze the failure behavior of our fleet of disk

drives using detailed monitoring data collected over a

nine-month observation window. During this time we

recorded failure events as well as all the available

en-vironmental and activity data and most of the SMART

parameters from the drives themselves. Failure

informa-tion spanning a much longer interval (approximately five

years) was also mined from an older repairs database.

All the results presented here were tested for their

statis-tical significance using the appropriate tests.

3.1 Baseline Failure Rates

Figure 2 presents the average Annualized Failure Rates

(AFR) for all drives in our study, aged zero to 5 years,

and is derived from our older repairs database. The data

are broken down by the age a drive was when it failed.

Note that this implies some overlap between the sample

sets for the 3-month, 6-month, and 1-year ages, because

a drive can reach its 3-month, 6-month and 1-year age

all within the observation period. Beyond 1-year there is

no more overlap.

While it may be tempting to read this graph as strictly

failure rate with drive age, drive model factors are

strongly mixed into these data as well. We tend to source

a particular drive model only for a limited time (as new,

more cost-effective models are constantly being

intro-duced), so it is often the case that when we look at sets

of drives of different ages we are also looking at a very

different mix of models. Consequently, these data are

not directly useful in understanding the effects of disk

age on failure rates (the exception being the first three

data points, which are dominated by a relatively stable

mix of disk drive models). The graph is nevertheless a

good way to provide a baseline characterization of

fail-ures across our population. It is also useful for later

studies in the paper, where we can judge how consistent

the impact of a given parameter is across these diverse

drive model groups. A consistent and noticeable impact

across all groups indicates strongly that the signal being

measured has a fundamentally powerful correlation with

failures, given that it is observed across widely varying

Figure 2:

Annualized failure rates broken down by age groups

ulation. The higher baseline AFR for 3 and 4 year old

drives is more strongly influenced by the underlying

re-liability of the particular models in that vintage than by

disk drive aging effects. It is interesting to note that our

3-month, 6-months and 1-year data points do seem to

indicate a noticeable influence of infant mortality

phe-nomena, with 1-year AFR dropping significantly from

the AFR observed in the first three months.

3.2 Manufacturers, Models, and Vintages

Failure rates are known to be highly correlated with drive

models, manufacturers and vintages [18]. Our results do

not contradict this fact. For example, Figure 2 changes

significantly when we normalize failure rates per each

drive model. Most age-related results are impacted by

drive vintages. However, in this paper, we do not show a

breakdown of drives per manufacturer, model, or vintage

due to the proprietary nature of these data.

Interestingly, this does not change our conclusions. In

contrast to age-related results, we note that all results

shown in the rest of the paper are

not

affected

signifi-cantly by the population mix. None of our SMART data

results change significantly when normalized by drive

model. The only exception is seek error rate, which is

dependent on one specific drive manufacturer, as we

dis-cuss in section 3.5.5.

3.3 Utilization

The literature generally refers to utilization metrics by

employing the term duty cycle which unfortunately has

no consistent and precise definition, but can be roughly

(31)

Reasons for Failures

‣

From: www.datarecorvery.org

‣

Physical reasons

• scratched platter

• broken arm/slider

• hard drive motor failed

• humidity, smoke in the drive

• manufacturer defect

• firmware corruption

• bad sectors

• overheated hard drive

• head crash

• power surge

• water or fire damage

‣

Logical Reasons

• failed boot sector

• master boot record failure

• drive not recognized by BIOS

• operating system malfunction

• accidentally deleted data

• software crash

• corrupt file system

• employee sabotage

• improper shutdown

• disk repair utilities

• computer viruses

• ...

(32)

Reasons for Failure

‣

Failure Trends in a Large Disk Drive Population, Pinheiro,

Weber, Barroso, Google Inc. FAST 2007

32 Figure 4:

Distribution of average temperatures and failures rates.

Figure 5:

AFR for average drive temperature.

ported temperature effects only for the high end of our

temperature range and especially for older drives. In the

lower and middle temperature ranges, higher

tempera-tures are not associated with higher failure rates. This is

a fairly surprising result, which could indicate that

data-center or server designers have more freedom than

pre-viously thought when setting operating temperatures for

equipment that contains disk drives. We can conclude

that at moderate temperature ranges it is likely that there

are other effects which affect failure rates much more

strongly than temperatures do.

3.5 SMART Data Analysis

We now look at the various self-monitoring signals that

are available from virtually all of our disk drives through

ones. At the end of this section we discuss our results

and reason about the usefulness of SMART parameters

in obtaining predictive models for individual disk drive

failures.

We present results in three forms. First we compare

the AFR of drives with zero and non-zero counts for a

given parameter, broken down by the same age groups

as in figures 2 and 3. We also find it useful to plot the

probability of survival of drives over the nine-month

ob-servation window for different ranges of parameter

val-ues. Finally, in addition to the graphs, we devise a

sin-gle metric that could relay how relevant the values of

a given SMART parameter are in predicting imminent

failures. To that end, for each SMART parameter we

look for thresholds that increased the probability of

fail-ure in the next 60 days by at least a factor of 10 with

respect to drives that have zero counts for that

parame-ter. We report such

Critical Thresholds

whenever we are

able to find them with high confidence (

>

95%).

3.5.1 Scan Errors

Drives typically scan the disk surface in the background

and report errors as they discover them. Large scan error

counts can be indicative of surface defects, and therefore

are believed to be indicative of lower reliability. In our

population, fewer than 2% of the drives show scan errors

and they are nearly uniformly spread across various disk

models.

Figure 6 shows the AFR values of two groups of

drives, those without scan errors and those with one or

more. We plot bars across all age groups in which we

have statistically significant data. We find that the group

of drives with scan errors are ten times more likely to fail

than the group with no errors. This effect is also noticed

when we further break down the groups by disk model.

From Figure 8 we see a drastic and quick decrease in

survival probability after the first scan error (left graph).

A little over 70% of the drives survive the first 8 months

after their first scan error. The dashed lines represent the

95% confidence interval. The middle plot in Figure 8

separates the population in four age groups (in months),

and shows an effect that is not visible in the AFR plots. It

appears that scan errors affect the survival probability of

young drives more dramatically very soon after the first

scan error occurs, but after the first month the curve

flat-tens out. Older drives, however, continue to see a steady

decline in survival probability throughout the 8-month

period. This behavior could be another manifestation of

(33)

S.M.A.R.T.

‣

Self-Monitoring, Analysis and

Reporting Technolgoy

‣

Relevant Parameters

• Seek error rate

- track was not hit

• Raw read error rate

- problems in the magnetic medium

• hardware ECC recovered

- recovered bits by error correction

(not really alarming)

• Scan error rate

- at periodic check non repairable

error occurs (problems in the

magnetic medium)

• Throughout performance

- spinning rate problem

• Spin up time

- startup time

• Reallocated sector count

- number of used reserve sectors

• Drive temperature

‣

Informative parameters

• Start/stop count

• Power on hours count

• Load/unload cycle count

• Ultra DMA CRC Error Count

(34)

Special Issues

(35)

Landing Zones

‣

Problem

• When a hard disk is switched off the head can damage

the platter

• Power loss

• Sudden movements

- can lead to permanent damage

‣

Solution

• Extra landing zones

- in the middle of the disk

- outside of the disk at extra park situation

(36)

Sound Control

‣

In desktop computers the working sound can irritate

and disturb

‣

Solution

• Extra mode with

- slower actuator arm movement

- slower rotation time

‣

Problem

(37)

Data Safety

‣

Problem

• Resold or disposed hard disk still carry sensible data

• Deleting data does not overwrite data

• Overwriting does not completely erase the information

‣

Solution

• Extra hardware (strong magnets, physical destruction)

• Cryptographic algorithms for storing

• Sophisticated overwriting algorithms

(38)

Albert-Ludwigs-Universität Freiburg

Institut für Informatik

Rechnernetze und Telematik

Algorithms and Methods for

Distributed Storage

Networks

2. Hard Disks