PCI-X SCSI RAID
Controller
Reference Guide for Linux
ESCALA POWER6
REFERENCE
86 A1 60EV 00
ESCALA POWER6
PCI-X SCSI RAID
Controller
Reference Guide for Linux
November 2007 BULL CEDOC 357 AVENUE PATTON B.P.20845 49008 ANGERS CEDEX 01 FRANCE
REFERENCE
86 A1 60EV 00
Hardware
The following copyright notice protects this book under Copyright laws which prohibit such actions as, but not limited to, copying, distributing, modifying, and making derivative works.
Copyright Bull SAS 1992, 2007 Printed in France
Suggestions and criticisms concerning the form, content, and presentation of this book are invited. A form is provided at the end of this book for this purpose. To order additional copies of this book or other Bull Technical Publications, you are invited to use the Ordering Form also provided at the end of this book.
Trademarks and Acknowledgements
We acknowledge the right of proprietors of trademarks mentioned in this book.
AIX is a registered trademark of International Business Machines Corporation, and is being used under licence. UNIX is a registered trademark in the United States of America and other countries licensed exclusively through the Open Group.
Contents
SafetyInformation . . . ix
HandlingStaticSensitiveDevices. . . xi
AboutThisBook . . . xiii
ISO9000 . . . xiii
Highlighting . . . xiii
Referencesto LinuxOperatingSystem . . . xiii
RelatedPublications . . . xiii
Trademarks . . . xiv
Part
1.
Linux
2.6
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 1
Chapter1. PCI-XSCSIRAIDControllerOverview . . . 3
DiskArrays . . . 3
SupportedRAIDLevels. . . 4
RAIDLevel0 . . . 5
RAIDLevel5 . . . 6
RAIDLevel6 . . . 7
RAIDLevel10 . . . 7
DiskArrayCapacities . . . 9
RAIDLevelSummary . . . 9
Stripe-UnitSize. . . 9
DiskArrayOverview . . . 10
DiskArraysStates . . . 11
PhysicalDisk States . . . 11
I/OAdapterStates . . . 11
AuxiliaryCache . . . 12
Chapter2. PCI-XSCSIRAIDControllerSoftware . . . 13
VerifyingtheInstallationof thePCI-XSCSIRAIDControllerSoftware . . . 13
Chapter3. CommonPCI-XSCSIRAIDControllerTasks . . . 15
Usingiprconfig . . . 15
ViewingtheStatusof DisksandDiskArrays . . . 15
ViewingDiskArrayStatus . . . 17
FormattingDisksforUseinPCI-XSCSIRAIDDiskArrays . . . 18
FormattingtoAdvancedFunction. . . 18
Formattingto JBOD . . . 18
DeletingaPCI-XSCSIRAIDDiskArray . . . 18
AddingDiskstoanExistingDiskArray . . . 19
UsingHotSpareDisks . . . 19
CreatingHot SpareDisks . . . 19
DeletingHotSpareDisks . . . 20
ViewingandChangingPCI-XSCSIRAIDControllerBusSettings. . . 20
SettingBusSpeedat Boot . . . 21
Creatinga PCI-XSCSIRAIDDiskArray . . . 22
Chapter4. PCI-XSCSIRAIDControllerMaintenance . . . 23
Do’sandDon’ts . . . 23
UpdatingthePCI-XSCSIRAIDControllerMicrocode . . . 24
SeparatingaRemovableCacheCard FromtheBase CardonType 2780and571ECards . . . 24
ReplacingtheCacheDirectoryCard . . . 35
RechargeableBatteryMaintenance . . . 38
DisplayingRechargeableBatteryInformation . . . 39
ForcingaRechargeableBattery Error . . . 39
ReplacingtheRechargeableCacheBattery Pack. . . 40
Separatingthe571F/575BCard SetandMovingtheCacheDirectoryCard . . . 43
ReplacingPhysicalDisks . . . 49
RemovingaFailedDisk . . . 50
InstallingaNewDisk . . . 50
RecoveryProceduresforDiskFailures . . . 51
RAIDLevel0 . . . 51
RAIDLevel5 . . . 51
RAIDLevel6 . . . 52
RAIDLevel10 . . . 52
ReclaimingIOACacheStorage . . . 53
Chapter5.ProblemDeterminationandRecovery. . . 55
ErrorLogAnalysis . . . 55
BasicviCommands . . . 56
SampleError Logs . . . 57
AdapterDump . . . 61
IdentifyingtheDiskArrayProblem . . . 62
UnitReferenceCode(URC)Tables. . . 62
MaintenanceAnalysisProcedures(MAPs) . . . 66
MAP3300 . . . 66 MAP3310 . . . 66 MAP3311 . . . 67 MAP3312 . . . 68 MAP3313 . . . 68 MAP3320 . . . 69 MAP3321 . . . 71 MAP3330 . . . 71 MAP3331 . . . 72 MAP3332 . . . 75 MAP3333 . . . 76 MAP3334 . . . 76 MAP3335 . . . 78 MAP3337 . . . 78 MAP3340 . . . 79 MAP3350 . . . 80 MAP3351 . . . 83 MAP3390 . . . 84
Part
2.
RedHat
Enterprise
Linux
3.0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 85
Chapter6.PCI-XSCSIRAIDControllerOverview. . . 87
DiskArrays. . . 87
SupportedRAIDLevels . . . 88
RAIDLevel0 . . . 89
RAIDLevel5 . . . 90
RAIDLevel10 . . . 91
DiskArrayCapacities . . . 92
RAIDLevelSummary . . . 92
PhysicalDisk Models . . . 94
DiskArraysStates . . . 94
PhysicalDisk States . . . 94
I/OAdapterStates . . . 95
Chapter7. PCI-XSCSIRAIDControllerSoftware . . . 97
VerifyingtheInstallationof thePCI-XSCSIRAIDControllerSoftware . . . 97
Chapter8. CommonPCI-XSCSIRAIDControllerTasks . . . 99
Usingiprconfig . . . 99
ViewingtheStatusof DisksandDiskArrays . . . 99
ViewingParityStatus . . . 102
DiskArrayParityStatus. . . 102
DiskParityStatus . . . 103
FormattingDisksforUseinPCI-XSCSIRAIDDiskArrays. . . 103
FormattingtoAdvancedFunction . . . 103
Formattingto JBOD . . . 104
Creatinga PCI-XSCSIRAIDDiskArray . . . 104
DeletingaPCI-XSCSIRAIDDiskArray. . . 105
AddingDiskstoanExistingDiskArray . . . 105
UsingHotSpareDisks . . . 106
CreatingHot SpareDisks . . . 106
DeletingHotSpareDisks . . . 106
ViewingandChangingPCI-XSCSIRAIDControllerBusSettings . . . 106
Chapter9. PCI-XSCSIRAIDControllerMaintenance . . . 109
Do’sandDon’ts . . . 109
UpdatingthePCI-XSCSIRAIDControllerMicrocode . . . 110
ReplacingtheCacheDirectoryCard . . . 110
RechargeableBatteryMaintenance . . . 112
DisplayingRechargeableBatteryInformation . . . 112
ForcingaRechargeableBatteryError . . . 113
ReplacingtheRechargeableCacheBattery Pack . . . 113
ReplacingPhysicalDisks . . . 115
RemovingaFailedDisk. . . 116
InstallingaNewDisk. . . 116
RecoveryProceduresfor DiskFailures . . . 116
RAIDLevel0 . . . 116
RAIDLevel5 . . . 117
RAIDLevel10 . . . 117
ReclaimingIOACacheStorage . . . 118
Chapter10.ProblemDeterminationandRecovery. . . 119
ErrorLogAnalysis. . . 119
BasicviCommands . . . 120
SampleError Logs . . . 121
AdapterDump . . . 125
IdentifyingtheDiskArrayProblem. . . 126
UnitReferenceCode(URC)Tables . . . 126
MaintenanceAnalysisProcedures(MAPs) . . . 130
MAP3300 . . . 130 MAP3310 . . . 131 MAP3311. . . 132 MAP3312 . . . 132 MAP3313 . . . 133 MAP3320 . . . 134
MAP3321 . . . 135 MAP3330 . . . 135 MAP3331 . . . 136 MAP3332 . . . 138 MAP3333 . . . 139 MAP3334 . . . 139 MAP3335 . . . 140 MAP3337 . . . 141 MAP3350 . . . 142 MAP3351 . . . 145 MAP3390 . . . 146
Part
3.
SuSE
Linux
Enterprise
Server
8
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 147
Chapter11.PCI-XSCSIRAIDControllerOverview. . . 149
DiskArrays . . . 149
SupportedRAIDLevels. . . 150
RAIDLevel0 . . . 151
RAIDLevel5 . . . 152
RAIDLevel10 . . . 153
DiskArrayCapacities . . . 154
RAIDLevelSummary . . . 154
Stripe-UnitSize. . . 154
DiskArrayOverview . . . 154
DiskArrayModels. . . 155
PhysicalDisk Models . . . 156
DiskArraysStates. . . 156
PhysicalDisk States . . . 156
I/OAdapterStates. . . 157
Chapter12.PCI-XSCSIRAIDControllerSoftware . . . 159
VerifyingtheInstallationofthePCI-XSCSIRAIDControllerSoftware. . . 159
Chapter13.CommonPCI-XSCSIRAIDControllerTasks . . . 161
Usingsisconfig . . . 161
ViewingtheStatusof DisksandDiskArrays . . . 161
ViewingParityStatus . . . 164
DiskArrayParityStatus. . . 164
DiskParityStatus . . . 165
FormattingDisksforUseinPCI-XSCSIRAIDDiskArrays. . . 165
FormattingtoAdvancedFunction . . . 165
Formattingto JBOD . . . 166
Creatinga PCI-XSCSIRAIDDiskArray . . . 166
DeletingaPCI-XSCSIRAIDDiskArray. . . 167
AddingDiskstoanExistingDiskArray . . . 167
UsingHotSpareDisks . . . 168
CreatingHot SpareDisks . . . 168
DeletingHotSpareDisks . . . 168
ViewingandChangingPCI-XSCSIRAIDControllerBusSettings . . . 168
Chapter14.PCI-XSCSIRAIDControllerMaintenance . . . 171
Do’sandDon’ts . . . 171
UpdatingthePCI-XSCSIRAIDControllerMicrocode. . . 172
DisplayingRechargeableBatteryInformation . . . 179
ForcingaRechargeableBatteryError . . . 180
ReplacingtheRechargeableCacheBattery Pack . . . 180
ReplacingPhysicalDisks . . . 182
RemovingaFailedDisk . . . 183
InstallingaNewDisk. . . 183
RecoveryProceduresfor DiskFailures . . . 183
RAIDLevel0 . . . 183
RAIDLevel5 . . . 184
RAIDLevel10 . . . 184
ReclaimingIOACacheStorage . . . 185
Chapter15.ProblemDeterminationandRecovery. . . 187
ErrorLogAnalysis. . . 187
BasicviCommands . . . 188
SampleError Logs . . . 189
AdapterDump . . . 193
IdentifyingtheDiskArrayProblem. . . 194
UnitReferenceCode(URC)Tables . . . 194
MaintenanceAnalysisProcedures(MAPs) . . . 198
MAP3300 . . . 198 MAP3310 . . . 199 MAP3311. . . 200 MAP3312 . . . 200 MAP3313 . . . 201 MAP3320 . . . 202 MAP3321 . . . 203 MAP3330 . . . 203 MAP3331 . . . 204 MAP3332 . . . 206 MAP3333 . . . 207 MAP3334 . . . 207 MAP3335 . . . 209 MAP3337 . . . 209 MAP3350 . . . 210 MAP3351 . . . 213 MAP3390 . . . 214 Appendixes. . . 215 CommunicationsStatements . . . 215
FederalCommunications Commission(FCC)Statement. . . 215
EuropeanUnion(EU)Statement . . . 215
InternationalElectrotechnicalCommission(IEC)Statement. . . 215
UnitedKingdomTelecommunicationsSafety Requirements. . . 215
AvisdeconformitéauxnormesduministèredesCommunicationsduCanada . . . 216
CanadianDepartmentofCommunications ComplianceStatement . . . 216
VCCIStatement . . . 216
ElectromagneticInterference(EMI)Statement -Taiwan . . . 216
RadioProtectionfor Germany . . . 216
Notices. . . 217
Safety
Information
DANGER
Anelectricaloutletthatis notcorrectlywiredcouldplace hazardousvoltage onmetalpartsof the systemorthe devicesthatattachtothe system.Itisthe responsibilityofthecustomer to ensure thattheoutletis correctlywiredandgroundedto preventanelectricalshock.
Before installingorremovingsignalcables,ensurethatthe powercablesforthe systemunit andallattacheddevicesare unplugged.
Whenaddingorremoving anyadditionaldevicestoorfromthe system,ensurethat thepower cablesforthosedevicesareunplugged beforethe signalcablesareconnected.Ifpossible, disconnectallpowercablesfromtheexistingsystembeforeyouadda device.
Use onehand,when possible,toconnectordisconnectsignalcablestopreventapossible shock fromtouchingtwosurfaceswithdifferentelectricalpotentials.
During anelectricalstorm,donotconnect cablesfordisplaystations,printers,telephones,or station protectorsforcommunicationlines.
Handling
Static
Sensitive
Devices
Attention:Static electricitycandamagethis deviceandyoursystemunit.Toavoiddamage,keepthis deviceinitsanti-staticprotectivebaguntil youarereadytoinstall it.Toreducethepossibilityof electrostaticdischarge,followtheprecautionslistedbelow:
v Limityourmovement.Movementcancausestaticelectricityto builduparoundyou.
v Handle thedevice carefully,holding itbyitsedgesoritsframe.
v Donottouchsolderjoints, pins,orexposedprintedcircuitry.
v Donotleavethedevicewhereotherscanhandleandpossiblydamagethedevice.
v While thedevice isstillinitsanti-staticpackage,touchitto anunpaintedmetalpartofthesystemunit
for atleasttwoseconds.(Thisdrainsstaticelectricityfromthepackageandfromyourbody.)
v Removethedevicefromitspackageandinstall itdirectlyinto yoursystemunitwithout settingitdown.If
it isnecessary tosetthedevicedown,placeitonitsstatic-protectivepackage.(If yourdeviceisan adapter,placeit component-sideup.)Donotplacethedeviceonyoursystem unitcoveror onametal table.
v Takeadditionalcarewhenhandlingdevicesduringcoldweather,asheatingreduces indoorhumidity
About
This
Book
Thisbookprovidesusageandmaintenanceinformationregarding thePCI-XSCSIRAIDControllerfor variousversionsoftheLinuxkernel.Chapters1 through4 ofthisbook containgeneralinformationthatis intendedfor allusersof thisproduct.Chapter5containsservice informationintendedfora service representativespecificallytrainedonthesystemunit andsubsystembeingserviced.Usethisbook in conjunctionwithyourspecificsystemunit andoperating systemdocumentation.
ISO
9000
ISO9000registeredquality systemswere usedinthedevelopmentandmanufacturingof thisproduct.
Highlighting
Thefollowinghighlightingconventionsareusedinthisbook:
Bold Identifiescommands,subroutines,keywords,files,structures,directories,andotheritems whosenamesarepredefinedbythesystem.Alsoidentifiesgraphicalobjectssuchasbuttons, labels,andiconsthattheuserselects.
Italics Identifiesparameterswhoseactualnamesorvaluesaretobesuppliedbytheuser.
Monospace Identifiesexamplesofspecificdatavalues,examplesoftextsimilartowhatyoumightsee displayed,examplesofportionsofprogramcodesimilartowhatyoumightwriteasa programmer,messagesfromthesystem,orinformationyoushouldactuallytype.
References
to
Linux
Operating
System
ThisdocumentmaycontainreferencestotheLinuxoperatingsystem,andisintendedtobeused with systemsrunningtheLinuxkernel.Threedifferentversionsof Linuxaredescribedinthisdocument:Linux 2.6,SuSELinuxEnterpriseServer8,andRedHatEnterpriseLinux3.0Makesure youareconsulting the appropriatesection ofthis documentfor theoperatingsystemyouareusing.
Thisdocumentmaydescribehardwarefeaturesandfunctions.Whilethehardware supportsthem,the realizationofthesefeaturesandfunctionsdependsupon supportfromtheoperating system.Linux
providesthis support.Ifyouareusinganotheroperating system,consulttheappropriatedocumentationfor thatoperatingsystemregarding supportforthosefeaturesandfunctions.
Related
Publications
Thefollowingpublicationscontainrelatedinformation: v
Systemunitdocumentationfor informationspecifictoyourhardware configuration
v IPRLinuxDeviceDriverWebsite,availableontheinternetathttp://sourceforge.net/projects/iprdd/
v RS/6000Eserver pSeriesAdapters,Devices,andCableInformationfor MultipleBusSystems,order
numberSA38-0516.Alsoavailableontheinternetathttps://techsupport.services.ibm.com/server/library/ v LinuxDocumentationProjectWebsite,availableontheinternetat http://www.tldp.org/
v LinuxforIBMEserverpSeriesWebsite, availableontheinternetat http://www-1.ibm.com/servers/
eserver/pseries/linux/
v RS/6000Eserver pSeriesDiagnosticInformationforMultipleBusSystems,ordernumberSA38-0509.
Alsoavailableontheinternetathttps://techsupport.services.ibm.com/server/library
v TheRAIDbook:AHandbookof StorageSystems Technology,Edition6,Editor:PaulMassiglia
v
PenguinppcWebsite,dedicatedto LinuxonPowerPC,availableontheinternetat
Trademarks
ThefollowingtermsaretrademarksofInternationalBusinessMachinesCorporationintheUnitedStates, othercountries,orboth:
v Eserver
v PowerPC
v pSeries
v RS/6000
Chapter
1.
PCI-X
SCSI
RAID
Controller
Overview
ThePCI-XSCSIRAIDControllerisa64-bitPCI-Xadapter.Thisadapterhasthefollowingfeatures: v
PCI-Xsysteminterface
v DualorQuadChannel,Ultra320(320MB/s)SCSIcontrollers
v Embedded PowerPCRISCProcessor,HardwareXORDMAEngine,andHardwareFiniteFieldMultiplier
(FFM)DMAEngine(forRAIDlevel6) v Non-volatilewritecache
v Support forRAID(RedundantArrayofIndependent Disks)levels0, 5,6, and10diskarrays
v Diskarraysupportedasabootabledevice
v Supportsattachmentof otherdevicessuchasnon-RAIDdisks, tape,andopticaldevices
v AdvancedRAIDfeatures:
– Hotsparesfor RAIDlevel5, 6,and10diskarrays
– AbilitytoincreasethecapacityofanexistingRAIDlevel5or6 diskarraybyaddingdisks
– Backgroundparitychecking
– Backgrounddatascrubbing
– Disksformattedto522bytes/sector,providingLongitudinalRedundancyChecking(LRC)and
LogicallyBadBlockchecking CCIN (CustomCard Identification Number) PCI Form Factor Channels RAID Levels Supported WriteCache Size ReadCache Size Multi Initiatorand High Availability Support Auxillary Cache Support
AdapterLED/FFC Cache BatteryPack Technology Cache BatteryPack Concurrent Maintenance Cache Battery PackFFC 2780 Yes 4 0,5,and 10 Upto757MB (compressed) Upto1024 MB (compressed) No No 2527 LiIon Yes 2D01
5702,1974 Yes 2 None 0MB 0MB Yes No 2522 None N/A N/A
5703,1975 Yes 2 0,5,and 10 40MB 0MB No No 2523 NiMH No 2526 5709,1976 No1 2 0,5,and 10 16MB 0MB No No 2523 NiMH No 2526
571A,1912 Yes 2 None 0MB 0MB Yes No 252D None N/A N/A
571B,1913 Yes 2 0,5,6,and 10
90MB 0MB No No 2514 LiIon No 2D03
571E6 Yes 4 0,5,6,and 10 Upto1.5GB (compressed) Upto1.6GB (compressed) No No 2512 Lilon Yes 2D01
572E No2 2 0and10 0MB 0MB No No 252B N/A N/A N/A
573D,1908 No 2 0,5,and 10 40MB 0MB No No 2529 NiMH No 2526 571F/575B3,6 Yes 44 0,5,6and 10 Upto1.5GB (compressed) Upto1.6GB (compressed)
No Yes 2513/252E LiIon Yes5 2D06
1 SomePCI-XSCSIRAIDControllersdonothavetheformofastandardPCIadapter.TheformofthesecontrollersisacombinationofSCSIandPCI-Xbusinterfacelogic integratedontothesystemboardandaRAIDEnablementCard.ThecardplugsintothesystemboardinordertoconnecttotheSCSIandPCI-Xbusinterfacelogic. 2ThisPCI-XSCSIRAIDControllerisintegratedontothesystemboardandattachestoSerialAttachedSCSI(SAS)disksvia3Gbconnections.
3 Double-wideadapterwhichtakesuptwoPCI-Xslots.ItiscomposedofbothastorageI/Oadapter(571F)andanAuxiliaryCacheadapter(575B).
4Threechannelsavailableforexternaldeviceattachment.OneinternalchanneldedicatedtoAuxiliaryCacheattachment.SeetheAuxiliaryCachesectionformoreinformation. 5 TheCacheBatteryPackforbothadaptersiscontainedinasinglebatteryFRUwhichisphysicallylocatedonthe575BAuxiliaryCachecard.
6
ThisadapterisonlysupportedinaLinuxpartitiononanIBMSystemiserver.
Disk
Arrays
RAIDtechnologyisusedto storedataacrossagroup ofdisksknownasadiskarray.Depending onthe RAIDlevelselected,thisstoragetechniqueprovidesthedataredundancyrequiredtokeep datasecure andthesystem operational.Ifa diskfailureoccurs,thediskcanusuallybereplacedwithout interrupting normalsystem operation.Diskarraysalsohave thepotentialto providehigherdatatransferandinput and output(I/O)rates thanthoseprovided bysinglelarge disks.
Eachdiskarray canbeused byLinuxinthesamewayasitwoulda singleSCSIdisk.For example,after creatingadiskarray,youcanuseLinuxcommandsto makethediskarrayavailabletothesystem by partitioningandcreatingafile systemonit.
ThePCI-XSCSIRAIDControllerismanagedbytheiprconfigutility.Theiprconfigutilityistheinterface totheRAIDconfiguration,monitoring,andrecoveryfeaturesofthecontroller.
Ifadiskarrayistobeusedasthebootdevice,it mayberequiredtopreparethedisksbybootinginto Rescuemodeandcreatingthediskarray beforeinstallingLinux.Youmightwantto performthis procedure whentheoriginal bootdriveistobeusedaspartofa diskarray.
Thefollowingfigureillustratesa possiblediskarrayconfiguration.
Supported
RAID
Levels
TheRAIDlevelof adiskarray determineshowdataisstoredonthediskarray andthelevelofprotection thatisprovided.Whenapartof theRAIDsystem fails,differentRAIDlevelshelpto recoverlostdatain differentways.WiththeexceptionofRAIDlevel0,ifa singledrivefailswithinanarray,thearraycontroller canreconstructthedataforthefaileddiskbyusingthedatastoredonotherdiskswithinthearray.This datareconstructionhaslittleornoimpacttocurrentsystemprogramsandusers.ThePCI-XSCSIRAID
ControllersupportsRAIDlevels0,5, 6,and10.EachRAIDlevelsupportedbythePCI-XSCSIRAID Controllerhasitsownattributesandusesa differentmethodof writingdata.Thefollowinginformation detailseachsupportedRAIDlevel.
RAID
Level
0
RAIDlevel0 stripesdataacrossthedisksinthearrayfor optimalperformance.Fora RAIDlevel0 array ofthreedisks, datawouldbewritteninthefollowingpattern.
RAIDlevel0 offersahighpotentialI/Orate,butitisanonredundantconfiguration.As aresult,thereisno dataredundancyavailablefor thepurposeof reconstructingdataintheeventofa diskfailure.Thereisno errorrecovery beyondwhatisnormallyprovidedonasingledisk.Unlikeother RAIDlevels, thearray controllernever marksa RAIDlevel0 arrayasDegradedastheresultof adiskfailure.Ifaphysicaldisk failsinaRAIDlevel0diskarray,thediskarrayismarkedasFailed.Alldata inthearraymustbebacked upregularly toprotectagainstdataloss.
RAID
Level
5
RAIDlevel5 stripesdataacrossalldisksinthearray.RAIDlevel5also writesarrayparitydata.Theparity dataisspreadacross allthedisks.For aRAIDlevel5arrayof threedisks,arraydataandparity
informationarewritteninthefollowingpattern:
Ifadiskfailsina RAIDlevel5array,you cancontinueto usethearraynormally.ARAIDlevel5array operatingwithasinglefaileddiskissaid tobeoperatinginDegradedmode.Wheneverdataisreadfrom anDegradeddiskarray,thearraycontrollerrecalculatesthedataonthefaileddiskbyusingdataand parityblocks ontheoperationaldisks.If aseconddiskfails,thearray willbeplacedintheFailedstate and willnotbeaccessible.
RAID
Level
6
RAIDlevel6 stripesdataacrossalldisksinthearray.RAIDlevel6also writesarray“P”and“Q”parity data.ThePandQparitydata, whichisbasedonReedSolomonalgorithms,isspreadacrossallthe disks.Fora RAIDlevel6 arrayoffourdisks,arraydataandparityinformationarewritten inthefollowing pattern:
Ifoneortwo disksfailina RAIDlevel6 array,youcancontinueto usethearraynormally.ARAIDlevel6 arrayoperatingwithaoneortwo Faileddisksissaid tobeoperatinginDegradedmode.Wheneverdata isreadfroma Degradeddiskarray,thearraycontrollerrecalculatesthedataontheFaileddisk(s)byusing dataandparityblocks ontheoperationaldisks.ARAIDlevel6 arraywitha singleFaileddiskhassimilar protectionto thatofa RAIDlevel5array withnodiskfailures. Ifathirddiskfails,thearraywillbeplaced intheFailedstateandwillnotbeaccessible.
RAID
Level
10
RAIDlevel10usesmirroredpairs toredundantlystore data.Thearraymust containanevennumberof disks.Twoistheminimumnumberof disksneededtocreateaRAID10array.Thedataisstripedacross themirroredpairs.Forexample,aRAIDlevel10arrayof fourdiskswouldhavedatawrittento itinthe followingpattern:
Disk 1 Disk 2 Disk 3 Disk 4
RAIDlevel10cantoleratemultiplediskfailures.If onediskineachmirroredpairfails,thearraywillstillbe functional,operatinginDegradedmode.Youcancontinuetouse thearraynormallybecausefor each faileddisk,thedataisstoredredundantlyonitsmirroredpair.However, ifbothmembers ofamirroredpair fail,thearraywillbeplacedintheFailedstateandwillnotbeaccessible.
WhenaRAIDlevel10diskarrayiscreated,thecontrollerwillautomaticallyattemptto selectthedisks for eachmirroredpairfromadifferentSCSIbus.For example,if fourdisks selectedfor thediskarrayreside ononeof thecontroller’sSCSIbusesandanotherfourdisksselectedresideonanotherof thecontroller’s SCSIbuses,thecontrollerwillautomaticallyattemptto createeach mirroredpairfromonediskoneach SCSIbus.Intheeventofa SCSIbusfailure,eachmirroredpairwillcontinuetooperateinDegraded mode.
Disk
Array
Capacities
Thecapacityof adiskarraydependsonthecapacityofthedisksusedandtheRAIDLevelofthearray. Tocalculatethecapacityof adiskarray,dothefollowing:
RAIDLevel 0
Multiplythenumberofdisksbythediskcapacity.
RAIDLevel 5
Multiplyonefewerthanthenumberofdisksbythediskcapacity.
RAIDLevel 6
Multiplytwo fewerthanthenumberofdisksbythediskcapacity.
RAIDLevel 10
Multiplythenumberofdisksbythediskcapacityanddivideby2.
Note: If disksofdifferentcapacitiesareused inthesamearray,alldisksaretreatedasiftheyhave the
capacityof thesmallestdisk.
RAID
Level
Summary
Thefollowinginformationprovidesdataredundancy, usablediskcapacity,readperformance,andwrite performanceforeach RAIDlevel.
RAIDLevel DataRedundancy UsableDisk Capacity
ReadPerformance WritePerformance
RAID0 None 100% VeryGood Excellent
RAID5 VeryGood 67%to94% VeryGood Good
RAID6 Excellent 50%to89% VeryGood FairtoGood
RAID10 Excellent 50% Excellent VeryGood
RAIDLevel 0
Doesnotsupportdataredundancy, butprovidesapotentially higherI/Orate.
RAIDLevel 5
Createsarrayparityinformationsothatthedatacanbereconstructedifa diskinthearrayfails. ProvidesbettercapacitythanRAIDlevel10butpossiblylowerperformance.
RAIDLevel 6
Createsarray“P”and“Q”parityinformation,usingReedSolomonalgorithms,sothatthedatacan bereconstructedifoneor twodisksinthearrayfail. Providesbetterdataredundancy thanRAID level5 butwithslightlylowercapacityandpossiblylowerperformance.Provides bettercapacity thanRAIDlevel10butpossiblylowerperformance.
RAIDLevel 10
Storesdataredundantlyonmirroredpairstoprovidemaximumprotectionagainstdiskfailures. ProvidesgenerallybetterperformancethanRAIDlevel5 or6,buthaslowercapacity.
Stripe-Unit
Size
WithRAIDtechnology, dataisstripedacrossanarrayof physicaldisks.Thisdatadistribution scheme complementsthewaytheoperatingsystem requestsdata.Thegranularityatwhichdata isstoredonone diskof thearraybeforesubsequentdata isstoredonthenext diskof thearray iscalledthestripe-unit size.Thecollectionof stripeunits,fromthefirstdiskofthearraytothelastdiskof thearray,iscalleda stripe.
Youcansetthestripe-unitsizeof aPCI-XSCSIRAIDDiskArrayto 16KB,64KB,or 256KB.You may beable tomaximizetheperformanceof yourPCI-XSCSIRAIDDiskArraybysettingthestripe-unitsizeto
avaluethatisslightlylarger thanthesizeof theaveragesystem I/Orequest.Forlargesystem I/O requests,usea stripe-unitsizeof 256KB.Therecommendedstripesizeformost applicationsis64KB.
Disk
Array
Overview
Diskarraysaregroupsofdisksthatworktogetherwitha specializedarraycontrollerto potentiallyachieve higherdatatransferandinput andoutput(I/O)ratesthanthoseprovidedbysinglelarge disks.Thearray controllerkeepstrack ofhow thedataisdistributedacrossthedisks.RAIDlevel5,6, and10diskarrays alsoprovidedataredundancy, sothatnodataislostif asinglediskinthearrayfails.
Note: This guideandtheiprconfigutilityusecommon terminologyfordiskformats:
JBOD AJBODdiskisadiskthatisformatted to512bytes/sector.JBODstandsfor″JustaBunch OfDisks″.
AdvancedFunction
An AdvancedFunctiondiskisadiskthatisformatted to522bytes/sector.Thisformat allowsdiskstobeusedindiskarrays.
PCI-XSCSIRAIDDiskArraysareaccessed inLinuxasstandardSCSIdiskdevices.Thesedevicesare automaticallycreatedwhenadiskarrayiscreated,anddeletedwhenevera diskarrayisdeleted.The individualphysicaldisksthatcomprisediskarraysorarecandidatesto beused indiskarrays,whichare formattedforAdvancedFunction, arehiddenfromLinuxandareaccessibleonly throughtheiprconfig
utility.LinuxseesallJBODdisks.ThesedisksmustbeformattedforAdvancedFunctionbefore theycan beused indiskarrays.ForinformationonformattingJBODdisksto makethem availableforuseinPCI-X SCSIRAIDDiskArrays,see“FormattingDisksfor UseinPCI-XSCSIRAIDDiskArrays”onpage18.The
DisplayHardwareStatusoptionintheiprconfigutilitycanbeusedto displaythesedisksandthe resourcenamesassociatedwiththem.For detailsregardinghow toviewthediskinformation,see
“ViewingtheStatusof DisksandDiskArrays” onpage15.Thefollowingsampleoutputisdisplayedwhen theDisplayHardwareStatusoptionisinvoked.
Display Hardware Status
Type option, press Enter.
1=Display hardware resource information details
OPT Name PCI/SCSI Location Description Status
--- --- --- ---
0000:02:04.0/2: PCI-X SCSI RAID Adapter Operational 0000:02:04.0/2:1:14:0
Advanced Function Disk Active 0000:02:04.0/2:1:6:0 Advanced Function Disk Active 0000:02:04.0/2:1:8:0 Advanced Function Disk Active 0000:02:04.0/2:1:9:0 Advanced Function Disk Active sdt 0000:02:04.0/2:255:0:0 RAID 5 Disk Array 62% Rebuilt 0000:02:04.0/2:1:10:0 RAID 5 Array Member Active 0000:02:04.0/2:1:11:0
RAID 5 Array Member Active 0000:02:04.0/2:1:12:0 RAID 5 Array Member Active 0000:02:04.0/2:1:13:0 RAID 5 Array Member Active sdb 0000:02:04.0/2:255:0:2 RAID 0 Disk Array Active 0000:02:04.0/2:1:4:0 RAID 0 Array Member Active sdc 0000:02:04.0/2:255:0:3 RAID 0 Disk Array Active 0000:02:04.0/2:1:3:0
RAID 0 Array Member Active sdd 0000:02:04.0/2:255:0:4 RAID 0 Disk Array Active
More...
Disk
Arrays
States
Thefifthcolumnintheprecedingoutputdisplaysthestate ofthediskarray.Thesevenvalidstatesfor PCI-XSCSIRAIDDiskArraysare:Active,Degraded,Rebuilding,R/WProtected,Missing,Offline, and Failed.
Active Thediskarrayisfunctionalandfullyprotected (RAIDlevel5, 6,and10)withallphysicaldisksin theActivestate.
Degraded
Thediskarray’sprotectionagainstdiskfailures isdegraded oritsperformanceisdegraded.When oneormorephysicaldisksinthediskarrayareintheFailed.state,thearray isstillfunctionalbut mightnolongerfullyprotectedagainst diskfailures. Whenallphysicaldisksinthediskarrayarein theActive.state,thearrayisnotperformingoptimallybecauseof aproblemwiththeI/Oadapter’s nonvolatilewritecache.
Rebuilding
Dataprotectionisbeingrebuiltonthisdiskarray. R/WProtected
Thediskarraycannotprocessa readnorwriteoperation.Adiskarraymay beinthisstate becauseof acache,deviceconfiguration,orany otherproblemthatcouldcausea dataintegrity exposure.
Missing
Thediskarraywasnotdetected bythehost operatingsystem. Offline Thediskarrayhasbeenplaced offlinedueto unrecoverableerrors.
Failed Thediskarrayisnolongeraccessiblebecauseof diskfailuresorconfigurationproblems.
Physical
Disk
States
Thefifthcolumnintheprecedingoutputdisplaysthestate ofthedevice.Thesixpossiblestatesfor physicaldisksare:Active,Failed,Offline, Missing,R/WProtected,andFormatRequired..
Active Thediskisfunctioningproperly.
Failed TheIOAcannotcommunicatewiththediskor thediskisthecauseof thediskarraybeinginthe degradedstate.
Offline Thediskarrayhasbeenplaced offlinedueto unrecoverableerrors. Missing
Thediskwasnotdetected bythehost operatingsystem. R/WProtected
Thedevice cannotprocessareadnorwriteoperation.Adiskmaybeinthis statebecauseofa cache,deviceconfiguration,oranyotherproblemthatcouldcauseadataintegrity exposure. FormatRequired
Thediskunitmust beformattedto becomeusableonthisIOA.
I/O
Adapter
States
Thefifthcolumnintheprecedingoutputdisplaysthestate oftheI/OAdapter.Thethree possiblestatesfor I/OAdaptersare:Operational,NotOperational,andNotReady.
Operational
TheIOAisfunctional. NotOperational
NotReady
TheIOArequiresa microcodedownload.
Auxiliary
Cache
TheAuxiliaryWriteCacheadapter(AWC)providesaduplicate,non-volatilecopyof writecachedataof the RAIDcontrollerto whichitisconnected.Protection ofdataisenhancedbyhaving twobatterybacked (non-volatile)copies ofwritecacheeach storedonseparateadapters. Ifafailureoccursto thewritecache portionoftheRAIDcontroller,or theRAIDcontrolleritselffailsinsucha waythatthewritecachedata is notrecoverable,theAWCadapterprovidesabackupcopyofthewritecachedata topreventdataloss duringtherecovery ofthefailedRAIDcontroller.Thecachedataisrecoveredtothenew replacement RAIDcontrollerandthenwrittenouttodiskbeforeresumingnormaloperations.TheAWC adapterisnota failoverdevicethatcankeep thesystemoperationalbycontinuingdiskoperationswhentheattached RAIDcontrollerfails.Thesystem cannotusetheauxiliarycopyofthecachefor runtimeoperationseven if onlythecacheontheRAIDcontrollerfails.TheAWCadapterdoes notsupportanyotherdevice
attachmentandperformsnoother tasksthan communicatingwiththeattachedRAIDcontrollerto receive backupwritecachedata.Thepurposeof theAWCistominimizethelengthofanunplannedoutage,due toa failureof aRAIDcontroller,bypreventinglossofcritical datawhichmayhave otherwiserequireda systemreload.
Shoulda failureof eithertheRAIDcontrolleror theAuxiliaryCacheoccur,it isextremelyimportant thatthe theMaintenanceAnalysisProcedures(MAPs)fortheerrorsintheLinuxerrorlogbefollowedprecisely. Neededservice informationcanbefound intheProblemDeterminationandRecoverychapter.
TheRAIDcontrollerandtheAWCadaptereachrequireaPCI-Xslotandarerequiredtobeinthesame partition.ThetwoadaptersareconnectedbyaninternalSCSIconnection.This dedicatedSCSIbusruns inUltra4mode (U320).
IftheAWCadapteritselffailsorthelinkbetweenthetwo adaptersfails,theRAIDcontrollerwillstop cachingoperations,destage existingwritecachedatatodisk,andrunina performancedegradedmode. OncetheAWCisreplacedorthelink isreestablished, theRAIDcontrollerwillautomatically recognizethe AWC,synchronizethecachearea,resumenormalcachingfunction,andresumewritingtheduplicate cachedatatotheAWC.
TheAWCadapteristypicallyusedinconjunctionwithRAIDprotection.RAIDfunctionsarenotaffectedby theattachmentofanAWC.Because theAWCdoesnotcontrolotherdevicesoverthebusand
communicatesdirectly withit’sattached RAIDcontrollerdirectlyovera dedicatedSCSIbus,ithaslittle,if any,performanceimpactonthesystem.
Chapter
2.
PCI-X
SCSI
RAID
Controller
Software
ForthePCI-XSCSIRAIDControllerto beidentifiedandconfiguredbyLinux,therequisitedevicesupport softwaremust beinstalled.Softwarefor thePCI-XSCSIRAIDControllerconsistsof adevicedriver anda setofutilities.Thedevicedriverisusuallycompiledasa kernelmodulenamedipr.ko.Theuserutilities areusuallypackagedinanRPMcallediprutils.TherequisitesoftwareforthePCI-XSCSIRAID
Controllerisoften preinstalledaspartofthenormalLinuxinstallation.However, ifthesoftwarepackageis notinstalled,softwareverificationwillfail.Themissingpackage(s)canbeinstalledfromyour Linux
operatingsystem CD-ROM.Ifyou aremissingcomponentsor neednewerversions,obtainthem fromyour Linuxdistributororonlineat:http://sourceforge.net/projects/iprdd/
Note: Systems usingthe2.4Linuxkernelrequireversion1.xof theiprdevicedriverandiprutils.
Systems usingthe2.6Linuxkernelrequireversion2.xof theipr devicedriverandiprutils.
ThePCI-XSCSIRAIDControllerexecutesonboardmicrocode.Althoughaversion ofcontrollermicrocode maybedistributedalongwithLinux,thisdoesnotnecessarilyrepresentthemost recentversionof
microcodeavailableforthecontroller.Theiprconfigutility canbeused toupdatethemicrocodebeing usedbythecontroller.Formoreinformationregardingiprconfig, see“UpdatingthePCI-XSCSIRAID ControllerMicrocode”onpage 24.
Verifying
the
Installation
of
the
PCI-X
SCSI
RAID
Controller
Software
Toverifythattheiprdevicedriver forthePCI-XSCSIRAIDControllerisinstalled,type:modinfo ipr | grep version
Thefollowingisanexampleofthedatathatdisplaysonyourscreen:
version: 2.0.9 DA9C6A0AA78C5D2B9D947A1
Thefollowingtabledescribestheminimumiprdevice driverversionrequiredforeach supportedadapter.
Table1.Minimumiprdevicedriversupport
Card Type
MinimumSupportedkernel.org Version
MinimumSupportedRedHat EnterpriseLinuxVersion
MinimumSupportedSuSE EnterpriseLinuxVersion iprdriver
version kernelversion
iprdriver
version RHELVersion
iprdriver
version SLESVersion
2780 2.0.10 2.6.8 2.0.11 RHEL4 2.0.10 SLES9 5702, 1974 2.0.10 2.6.8 2.0.11 RHEL4 2.0.10 SLES9 5703, 1975 2.0.10 2.6.8 2.0.11 RHEL4 2.0.10 SLES9 5709, 1976 2.0.10 2.6.8 2.0.11 RHEL4 2.0.10 SLES9 571A, 1912 2.0.13 2.6.12 2.0.11.1 RHEL4U2 2.0.10.3 SLES9SP2 571B, 1913 2.0.13 2.6.12 2.0.11.1 RHEL4U2 2.0.10.3 SLES9SP2 571B, 1913 2.0.13 2.6.12 2.0.11.1 RHEL4U2 2.0.10.3 SLES9SP2
571E 2.0.13 2.6.12 2.0.11.1 RHEL4U2 2.0.10.3 SLES9SP2
Table1.Minimumiprdevicedriversupport (continued)
Card Type
MinimumSupportedkernel.org Version
MinimumSupportedRedHat EnterpriseLinuxVersion
MinimumSupportedSuSE EnterpriseLinuxVersion iprdriver
version kernelversion
iprdriver
version RHELVersion
iprdriver
version SLESVersion
573D, 1908 2.0.13 2.6.12 2.0.11.1 RHEL4U2 2.0.10.3 SLES9SP2 571F/ 575B 2.1.0 2.6.15 2.0.11.2 RHEL4U4 2.1.2 SLES10
Toverifythattheiprconfigutilityisinstalled,typethecommand:
iprconfig --version
Outputfromthiscommandwillindicateiftheutilityisinstalled,andif so,version informationwillbe displayed.
Overtime, itmaybecome necessarytoinstallsoftwareupdatesinordertohave thevery latestavailable levelofdevicesoftwaresupportforthePCI-XSCSIRAIDController.Updatesto thedevicesupport softwarearepackaged,distributed,andinstalledthroughthesamemechanismsused forotherportionsof theLinuxdistribution.
Chapter
3.
Common
PCI-X
SCSI
RAID
Controller
Tasks
Theinstructionsinthischapterpertaintothevarioustasksthatcanbeperformedinordertomanagedisk arrays.
Using
iprconfig
TheinterfaceforworkingwiththePCI-XSCSIRAIDControllerisiprconfig.
Tostarttheiprconfigutility, typethecommand:
iprconfig
Themainmenuwilldisplayoptionsforconfiguring andmanagingthePCI-XSCSIRAIDController.
IBM Power RAID Configuration Utility
Select one of the following:
1. Display hardware status 2. Work with disk arrays
3. Work with disk unit recovery 4. Work with SCSI bus configuration 5. Work with driver configuration 6. Work with disk configuration 7. Download microcode 8. Analyze Log Selection: e=Exit
Viewing
the
Status
of
Disks
and
Disk
Arrays
Theiprconfigutilityoffersaneasywaytoviewthestatusofalldevicescontrolledbytheiprdevice driver.
Toviewinformationabout thedisksanddiskarraysonyoursystem,dothefollowing: 1. Runtheiprconfigutilitybytypingiprconfig.
Display Hardware Status
Type option, press Enter.
1=Display hardware resource information details
OPT Name PCI/SCSI Location Description Status
--- --- --- ---
0000:01:01.0/0: PCI-X SCSI Adapter Operational 0000:41:01.0/1:
PCI-X SCSI RAID Adapter Operational sda 0000:41:01.0/1:0:3:0 Physical Disk Active sdb 0000:41:01.0/1:0:4:0 Physical Disk Active sdc 0000:41:01.0/1:1:3:0 Physical Disk Active sdd 0000:41:01.0/1:1:4:0 Physical Disk Active sde 0000:41:01.0/1:1:5:0 Physical Disk Active 0001:61:01.0/2:
PCI-X SCSI RAID Adapter Operational sdf 0001:61:01.0/2:0:3:0 Physical Disk Active sdg 0001:61:01.0/2:0:9:0 Physical Disk Active sdh 0001:61:01.0/2:255:0:0 RAID 5 Disk Array Active 0001:61:01.0/2:0:4:0 RAID 5 Array Member Active 0001:61:01.0/2:0:5:0 RAID 5 Array Member Active 0001:61:01.0/2:0:6:0
RAID 5 Array Member Active 0001:61:01.0/2:0:8:0 RAID 5 Array Member Active
e=Exit q=Cancel r=Refresh t=Toggle
v
Thefirstcolumnisaninputfield usedto selectadeviceto displayadditionalinformation.
v Thesecondcolumnof outputisthedevice’sresourcename.
v Thethirdcolumnof outputisthedevice’sPCI/SCSILocationcode.Theformatof thisfieldisPCI
Location/SCSIHost:SCSIBus:SCSITarget:SCSILun.
v Thefourthcolumnof outputisthedevice’sdescription.Typing a’t’willtogglethisfield tobethe
device’sVendor/ProductID.
v Thefifthcolumnof outputisthedevice’shardwarestatus.For anoverviewofthepossiblediskanddisk
arrayhardware statuses,see“DiskArrayOverview”onpage10.
Toviewinformationonaspecificdevice, selectthedesireddevicewitha1andpressEnter.Outputsimilar tothefollowingwillbedisplayed:
Disk Unit Hardware Resource Information Details
Manufacturer. . . : IBM
Product ID. . . : IC35L036UCDY10-0 Firmware Version. . . : 53323846 (S28F) Serial Number . . . : E3V0E77B
Capacity. . . : 36.40 GB Resource Name. . . . .. . . : /dev/sda
Physical location:
PCI Address . . . : 0000:41:01.0 SCSI Host Number . . . : 1
SCSI Channel . . . : 0 SCSI Id. . . : 3 SCSI Lun . . . : 0 Extended Details FRU Number . . . : 08K0293 EC Level . . . : H32224
Press Enter to continue.
e=Exit q=Cancel f=PageDn b=PageUp
Thepreviousscreenshowsanoverviewof aparticularpieceof hardwareonyoursystem.Multiplepages ofinformationmaybeavailable.
Viewing
Disk
Array
Status
Toviewthediskarraystatus,dothefollowing: 1. Runtheiprconfigutilitybytypingiprconfig.2. SelectWorkwithdiskarrays.
3. SelectDisplay diskarraystatus.TheDisplayDiskArrayStatusscreenwillappear.
Thefollowingissampleoutputthatwilldisplay:
Display Disk Array Status
Type option, press Enter.
1=Display hardware resource information details
OPT Name PCI/SCSI Location Description Status
--- --- --- ---
sdh
0001:61:01.0/2:255:0:0 RAID 5 Disk Array Active 0001:61:01.0/2:0:4:0 RAID 5 Array Member Active 0001:61:01.0/2:0:5:0 RAID 5 Array Member Active 0001:61:01.0/2:0:6:0 RAID 5 Array Member Active 0001:61:01.0/2:0:8:0 RAID 5 Array Member Active
Disk
Array
Status
Thelastcolumnintheprecedingoutputdisplaysthestatusofeachdiskarray.Thesestatesaredescribed in“DiskArraysStates”onpage11.
Physical
Disk
Status
Thelastcolumnintheprecedingoutputdisplaysthestatusofeachphysicaldisk.Thesestatesare describedin“PhysicalDiskStates”onpage11.
Formatting
Disks
for
Use
in
PCI-X
SCSI
RAID
Disk
Arrays
BeforeadiskcanbeusedinaPCI-XSCSIRAIDDiskArray,it mustbeformattedfor AdvancedFunction. Beforeadiskisrecognizedasastandalonedisk,itmust beformatted toJBOD.Stepsfor performingboth oftheseactionsarecontainedinthis section.
Formatting
to
Advanced
Function
ToformatadiskforAdvancedFunction,dothefollowing: 1. Runtheiprconfigutilitybytypingiprconfig.2. SelectWorkwithdiskarrays.
3. SelectFormatdeviceforRAIDfunction.
4. Fromthelist ofeligiblediskunits,choosethedisksyouwanttoformat forAdvancedFunctionand
pressEnter.
Attention: Continuingwiththisoptionwillformatthedisks.Alldataonthediskswillbelost. Some
disksrequirethattheirmicrocodebeupdatedto thelatestlevelbeforebeingformattedfor Advanced Function.Thesediskswillnotshowuponthelistof choices.Insomecases,errorsmaybeloggedin the/var/log/messagesfile.For moredetailedinformation,viewthatlogfile.
5. Amessagewilldisplayaskingif youwantto continue.Toproceedwiththeformat,typec toconfirm.To
returntothepreviousmenuwithoutformattingthedisks,typeq.
Note: If youarenotusingadevicemapper(for example,LVM,md,orscsidev),resource names(such
as/dev/sdb)maychangewhenthesystemisrebooted.Thiscouldaffectkernelcommandline entriesandfstab entries.
Aftertheformattingiscomplete,thediskswillbereadyforuse inPCI-XSCSIRAIDDiskArrays.
Formatting
to
JBOD
Toformatadiskbackto JBODformatandallowLinuxto useitasastandalonedisk,dothefollowing: 1. Runtheiprconfigutilitybytypingiprconfig.
2. SelectWorkwithdiskarrays.
3. SelectFormatdeviceforJBODfunction(512).
4. Fromthelist ofeligiblediskunits,choosethedisksyouwanttoformat toJBODandpressEnter.
Attention: Continuingwiththisoptionwillformatthedisks.Alldataonthediskswillbelost.
5. Amessagewilldisplayaskingif youwantto continue.Toproceedwiththeformat,typec toconfirm.To
returntothepreviousmenuwithoutformattingthedisks,typeq.
Note: If youarenotusingadevicemapper(for example,LVM,md,orscsidev),it isadvisedthatyou
reboot thesystemnow.Resourcenames(suchas/dev/sdb)maychangeasaresultof reformatting.Thiscouldaffectkernelcommandlineentriesandfstabentries.
Note: If adiskarrayiscurrently beingrebuiltorsynched,thatdiskarraycannotbedeleted.Adiskarray
thatiscurrentlyinusecanbedeleted.Makesurethediskarrayisnotbeingused beforedeleting it. Topreservethedataonthediskarray,youmustfirstbackupalldatathatyouwishtosave.Todeletethe array,dothefollowing:
1. Runtheiprconfigutilitybytypingiprconfig.
2. SelectWorkwithdiskarrays.
3. SelectDeletea diskarray.
4. Fromthelisteddiskarrays,selecttheoneyou wishto deletethenpressEnter.
Attention: Alldataontheselecteddriveswillbelostwhenthediskarrayisdeleted. Ifyouaresure
youwantto deletethediskarray,pressEnter.Ifyoudonotwantto deletethediskarray,typeq to cancel.
Note: If youarenotusingadevicemapper(for example,LVM,md,orscsidev),resource names(such
as/dev/sdb)maychangewhenthesystem isrebooted.Thiscouldaffectkernelcommandline entriesandfstab entries.
Adding
Disks
to
an
Existing
Disk
Array
ThePCI-XSCSIRAIDControllersupportsaddingdisksto existingRAIDlevel5or 6diskarrays.This featurecanbeused todynamicallyincreasethecapacityof adiskarraywhilepreservingexistingdata in thediskarray.Extensiveuseof thisfeature,however,willresultinaperformancepenaltybecausethe datawillnotberestriped.
Toadddiskstoanexistingarray, dothefollowing: 1. Runtheiprconfigutilitybytypingiprconfig.
2. SelectWorkwithdiskarrays.
3. SelectAdda devicetoa diskarray.
4. SelectthePCI-XSCSIRAIDDiskArraytowhichyouwantto addthedisk(s).
5. Selectthedisk(s)to beincludedintothediskarray thenpressEnter.
Attention: Alldataontheselecteddriveswillbelostwhenthedisk(s)areaddedto theexisting
array.
6. Ifyouaresureyouwanttoincludethedisk(s),pressEnter.Ifyou donotwanttoincludethedisk(s)in
thediskarray,type qtocancel.
Ifaparticulardiskisnotincludedinthelist, itmaynotbeacandidatethatcanbeaddedto thearray becauseof oneormoreof thefollowing reasons:
v Thedisk’scapacityislessthanthatofthesmallestdiskalreadyinthearray.
v Thediskhasnotbeen formattedforAdvancedFunction.
v ThediskbelongstoanotherdiskarrayorisconfiguredasaHotSpare.
Using
Hot
Spare
Disks
HotSparedisksareusedto automaticallyreplacefaileddisksina RAIDenvironment.HotSparedisksare usefulonlyif theircapacityisgreater thanorequaltothatof thesmallestcapacitydiskinanarraythat becomesDegraded.In ordertoassigna diskfor useasahotspare,it mustbeformattedfor Advanced Function.
Creating
Hot
Spare
Disks
TocreateHotSparedisks, dothefollowing: 1. Runtheiprconfigutilitybytypingiprconfig.2. SelectWorkwithdiskarrays.
3. SelectCreateahotspare.
4. Selecttheadapterforwhich youwantto createhotsparesandpressEnter.
5. SelectthedisksthatyouwanttodesignateashotsparesandpressEnter.
Attention: Alldataontheselecteddriveswillbelostwhenthedisksareconfiguredashotspares.
6. Ifyouaresureyouwanttoconfigurethedisksashotspares,pressEnter.Ifyoudonotwantto
configurethedisksashotspares,typeq tocancel.
Deleting
Hot
Spare
Disks
TodeleteHot Sparedisksandfreethem foruseinotherdiskarrays,dothefollowing: 1. Runtheiprconfigutilitybytypingiprconfig.
2. SelectWorkwithdiskarrays.
3. SelectDeletea hotsparedevice.
4. SelecttheHotSparedisksto bedeleted andpressEnter.
5. Ifyouaresureyouwanttoreconfigurethedisks,pressEnter.Ifyoudonotwantto reconfigurethe
disks,typeq tocancel
Viewing
and
Changing
PCI-X
SCSI
RAID
Controller
Bus
Settings
Note: Themaximumspeedandbuswidthareautomatically negotiatedbasedonthecharacteristicsof the
attached devices.Ifyouarenotsuccessfulrunningat thesespeeds, youcanreducetheadapter speedbychangingtheMaximumbusthroughput.
Note: Thedriver willdefaultto amaximumof160MB/suntil theiprinitutilityrunssuccessfully.After
iprinitcompletes,maximumconfiguredbusspeedisenabled.Eachtimethesystem isbooted,
iprinitisrunautomatically.
Toviewandchange settingsfor thePCI-XSCSIRAIDController,dothefollowing: 1. Runtheiprconfigutilitybytypingiprconfig.
2. SelectWorkwithSCSIbusconfiguration.
3. Selecttheadapterthatyouwanttochangesettingsonor viewbytyping1 thenpressingEnter.The
Change SCSI Bus Configuration
Current Bus configurations are shown. To change
setting hit "c" for options menu. Highlight desired
option then hit Enter
c=Change Setting
Adapter Location: 0000:01:01.0
BUS 0
Maximum Bus Throughput . . : 320 MB/s BUS 1
Wide Enabled . . . : Yes Maximum Bus Throughput . . : 320 MB/s
Press Enter to Continue e=Exit q=Cancel
4. Followthedirectionsonscreento changesettings,ifdesired.
Theprecedingscreendisplaysinformationabout thecontrollerbus.Thefollowingisinformationregarding attributesdisplayed:
MaximumBusThroughput
Thisattributeisusedto limitthemaximumspeedinmegabytespersecond(MB/s)thatthe adapterSCSIbuswilluse.Speed isnegotiatedbetweentheadapterandSCSIdevices.The optionsavailablewilldifferdependingonyourhardware configuration.
WideEnabled
Specifieswhether thecontrollerwillnegotiatewiththedevicefora 16-bitdatatransferwidthonthe SCSIbus.IfYesisspecified,whichisthedefaultvalue,thecontrollernegotiatesfor 16-bitdata transfers.Yescanbespecifiedwhenan8-bitdeviceisconnectedthroughan8-bitcabletothe SCSIconnector.TheNooptionsetsthedatatransferwidthto8 bits.UsetheNooptionwhen olderdevicesareconnectedthatmaynothandletheWideDataTransfer Request,WDTR,
messagesproperly.TheWDTRmessagesareused todetectthedatatransferwidthcapabilitiesof adevice.
Note: If adevicethatrequires16-bitdata transferisattached,youwillnothave theoptionto
disable widetransfermode.
HostSCSIID
SpecifiestheSCSIIDtouse forthecontroller.Thedefaultvalueis7.Change thiswhenin multi-initiatorconfigurations.
Setting
Bus
Speed
at
Boot
Ifrunninga2.6kernel,amoduleparameter isprovided toallowfor settingthemaximumbus speedat boottimebefore theiprinitutilityruns.Normallythis shouldnotbeneeded,butif youarehavingdifficulty bootingdueto running160MB/s,thebusspeedcan belowered to80MB/sbyusing ipr.max_speed=0on thekernelcommandline.
Creating
a
PCI-X
SCSI
RAID
Disk
Array
Adiskarrayiscreatedusingasetof disksthatareformatted forAdvancedFunction.Diskarrayswithdata redundancy(RAIDlevel5, RAIDlevel6, andRAIDlevel10)willbeunprotecteduntil parityisfinished beingbuilt.Fordiskarrayswithnodataredundancy(RAID0),noparityiscalculated andthearrayisbuilt immediately.TocreateaPCI-XSCSIRAIDDiskArray,dothefollowing:
1. Runtheiprconfigutilitybytypingiprconfig.
2. SelectWorkwithdiskarrays.
3. SelectCreateadiskarray.
4. Selectthecontrollerunderwhichyouwouldlike tocreateadiskarrayandthenpressEnter.
5. Selectthediskunitsto beincludedinthediskarraythenpressEnter.Ascreensimilarto thefollowing
willappear:
Select Protection Level and Stripe Size
Default array configurations are shown. To change
setting hit "c" for options menu. Highlight desired
option then hit Enter
c=Change Setting
Protection Level . . . : RAID 5
Stripe Size . . . : 64 k
Queue Depth (default = 12). . . : 12
Press Enter to Continue
e=Exit q=Cancel
6. Selecttheprotectionleveldesiredforthearray.Formore informationaboutselectinganappropriate
RAIDlevel,see “SupportedRAIDLevels” onpage4.
7. Selectthestripesizedesiredinkilobytesfor thearray. Formoreinformationabout selectingan
appropriatestripesize,see“Stripe-UnitSize”onpage 9. 8. PressEntertocontinue.
Attention: Alldataontheselecteddriveswillbelostwhenthediskarrayiscreated. Ifyouaresure
youwantto createthediskarray,pressEnter.Ifyoudonotwantto createthediskarray,typeqto cancel.
Ifyouchooseto createthediskarray,astatusscreendisplaysuntil theoperation iscompleted.Ifyouwish tocreateadditionaldiskarrays,youmay leavethestatusscreenbytypinge.Whena diskarrayhasbeen built,it willbeavailableto LinuxaswouldanynormalSCSIdisk.Tofind theresource namethathasbeen assignedtothediskarray,referto theDisplayDiskArrayStatusscreen.
Note: If youarenotusingadevicemapper(for example,LVM,md,orscsidev),it isadvisedthatyou
reboot thesystemnow.Resourcenames(suchas/dev/sdb)maychangeasaresultofcreatinga new diskarray.This couldaffect kernelcommandlineentriesandfstab entries.
Chapter
4.
PCI-X
SCSI
RAID
Controller
Maintenance
Thischapterprovidesinformationto assistinensuringoptimalperformanceof yourPCI-XSCSIRAID ControllerunderLinux2.6.
Do’s
and
Don’ts
TohelpavoidPCI-XSCSIRAIDControlleranddiskarrayproblems,usethefollowingtips:
v Always performanormalsystem shutdownbeforephysicallyreplacingormovingtheRAIDadapteror
members ofdiskarrays.Anormalshutdownof thesystemwillflushtheadapter’swritecacheand removedependenciesbetweentheadapterandthephysicaldisks.Using themodprobe -ripr
commandwillhavethesameeffect asasystemshutdown.For systemsthatsupportPCIhotplug,PCI hotplugwillalsohave thesameeffect asa systemshutdown.
Note: Adiskthatisa failedmemberof aDegradeddiskarraycanbereplacedandthediskarray
rebuiltwhile thesystem continuestorun.
v Youcan physicallymovedisksfromoneadaptertoanother.However, ifthephysicaldisksaremembers
of adiskarray,besureto moveallthedisksasagroup. Priorto attemptingadiskmovement,ensure thatthediskarrayisnotinanDegradedstate becauseof adiskfailure.
v Whenphysicallyremovingdisksthataremembers ofadiskarrayandthereisnoneedtopreserve data
andnointentto usethediskarrayagain,deletethediskarraybefore removingthedisks.Thisaction avoidsdiskarray-relatedproblemsthenexttime thesedisksareused.
v Always usetheDevice ConcurrentMaintenancescreentoremoveandreplacea physicaldisk.For
instructionsonhowtoremoveandreplacea disk,see“ReplacingPhysicalDisks”onpage49.
v If adiskarrayisbeingusedasaboot deviceandthesystem failsto bootbecauseofasuspected disk
arrayproblem,boot intoRescuemode.Linuxerrorlogs,theiprconfigutility,andothertoolsare availableto helpdetermineandresolve theproblem withthediskarray.
v
Donotattempttocorrectproblemsbyswappingadaptersanddisksunlessyouaredirectedto dosoby
theserviceprocedures.Thisislikelyto maketheproblemsworse.Foradditionalinformationregarding problemdetermination,seeChapter5,“ProblemDeterminationandRecovery,”onpage55.
v Donotconfuse theCache DirectoryCard,whichisasmallrectangularcardwitharoundbutton-shaped
battery,fora CacheCard.Thenonvolatilewritecachememoryisintegratedinto themain adaptercard orintegratedintotheremovablecachecardonthenewest adapters.Thememoryitselfisnot
removable,andisbattery-backedbythelarge,rechargeableCache BatteryPack.TheCacheDirectory Card containsonly asecondarycopyofthewritecachedirectory, nocachedata, andshouldneverbe removedexceptunder veryspecificcasesasdescribedintheMAPs.
v DonotunplugorexchangeaCache BatteryPackwithout followingtheproceduresasoutlined inthis
section orintheMAPs. v
If multipleerrorsoccuratapproximatelythesametime,look atthemasawholeto determineif there
Updating
the
PCI-X
SCSI
RAID
Controller
Microcode
Diskdrivesattachedto thisRAIDadaptermust containthelatestmicrocodelevel.Thelatestmicrocodelevelfor yourdrivemodelandPCI-XSCSIRAIDControllercanbedownloadedfrom http://www14.software.ibm.com/webapp/set2/firmware.
Microcodecanbeupdatedusingtheiprconfigutility.Theiprconfigutilityallowsfor multiplelevels of adapteranddevice microcodetoexistonthesystem atthesametime.Aftertheadapteror deviceto updatemicrocodeisselected,allavailablemicrocodelevelsarepresentedtochooseto download.
Note: For eitheranAuxiliaryCacheadapterorastorageI/OadapterwhichattachestoanAuxiliaryCache
adapter,it isrecommendedthatboththestorageI/OadapterandAuxiliaryCache adapterbe updated.
IfyoudownloadednewmicrocodefromtheWebsite,followtheinstructionsintheReadmefileorDesc
linkontheWebsiteto installit.Iftherearenoinstructionsavailable,install itasfollows: 1. Installthepackagebytyping:
rpm -ivh –ignoreos pci.1069B166.09-0509008e-1.Linux.noarch.rpm
PressEnter.
2. Updatethemicrocodebytyping:
iprconfig
PressEnter.
3. SelectDownloadmicrocodeandpressEnter.
4. Selectthedeviceor adapteryou wishto updatebytypinga 1thenpressingEnter.Multipledevices
maybeselected.
5. Selectthemicrocodeimagetobedownloadedtothedeviceoradapterbytypinga1 thenpressing
Enter.
6. PressEnteragainto confirmthedownload.
7. Ifmultipledeviceswere selected,repeatfromStep5forthenext device.
Separating
a
Removable
Cache
Card
From
the
Base
Card
on
Type
2780
and
571E
Cards
YouwillneedaT-10TORXtoolto completethis procedure.
Attention: Toavoidlossof cachedata,donotdisconnectthecachebatteryduringthisprocedure.
Allcardsaresensitivetoelectrostaticdischarge.See“HandlingStaticSensitiveDevices”onpagexi.
Attention: Thisprocedure shouldonlybeperformedif directedfromanisolationprocedure ora
maintenanceanalysis procedure(MAP).
Toseparatetheremovablecachecardfromthebase card,dothefollowing:
1. Labelbothsidesofthecardbeforeseparatingthecachecardfromthebasecard.
2. Areyouservicinga 2780adapterora571Eadapter?
v Ifyouareservicinga2780adapter,gotostep 3.
4. RemovethreescrewsA fromthebasecardasshown.Notethethreaddifferencesofthescrewsfor
correctreplacementinthefollowingsteps.Thetwoscrewsthatweresecured intotheplastic handle havea differentthreadpatternfromthemachinethreadscrewthatwasremovedfromthefrontofthe card.
5. Turnthecardover.
6. RemovefourscrewsB fromtheremovablecachecardasshown.
7. Toseparatethecards,turnthejackscrewcounterclockwiseuntilyoufeelthecardsseparate.
8. Separatethebasecardfromtheremovablecachecard.
Important: Afterseparatingthebasecardfromthecachecard,turnthejackscrewclockwiseto
reseatit.thengoto “AttachtheRemovableCacheCardto theBaseCard”onpage31.
Thisendsthisprocedure.
9. Placethe571EadapteronanESDprotectivesurfaceandorientateitasshown instep10.Notethe
positionofthecardhandle.
10. RemovescrewAfromthebasecardasshown.
11. Turnthecardover.
12. LoosenscrewB onefullturn.Donotremovethis screw.
13. RemovethefourscrewsCfromtheremovablecachecardasshown.
Note: Observethekeyholeslot locatedbelowscrewB.Instep15onpage30whenaskedto
separatethecachecard,youwillneedtohave thekeyholeslotclear theretainingscrew.
14. Turnthejackscrewcounterclockwisetobeginseparatingthecards.Continueturningthejackscrew
untilyou feelthecardsseparate.
15. Carefullyslidetheremovablecachecard towardstheplastichandle.Slightlyliftupthebackportionof
thecachecardfreeingthecachedirectory card’skeyholeslotfromitsretaining screw.
16. Separatethebasecardfromtheremovablecachecard.
Note: Afterseparatingthebasecardfromthecachecard,turnthejackscrew clockwisetoreseatit.
Attach
the
Removable
Cache
Card
to
the
Base
Card
Toattachtheremovablecachecardtothebase card,dothefollowing:1. Makesurethejackscrewisturnedclockwiseuntil itisseated.Ifthejackscrewisnotfullyseated,
thecardswillnotconnecttogetherproperlyanddamagemay occur.
2. Areyouservicinga 2780adapterora571Eadapter?
v If youareservicinga2780adapter,gotostep 3.
v
If youareservicinga571Eadapter,goto step7 onpage33.
3. Alignthenattachthe2780base cardtotheremovablecachecardbydoingthefollowing:
a. Alignthecardconnector.
Note: Tohelp alignthecardconnector,viewthejackscrewthroughthehole.
4. AttachfourscrewsAto theremovablecachecardasshown.
5. Turnthecardover.
6. AttachthethreescrewsBto thebasecardasshown.Notethethreaddifferencesinthescrewsas
pointedoutinstep4onpage25ofthepreviousseparationprocedure.
Thisendsthisprocedure.
7. Attachthe571Ebase cardtotheremovablecachecardbydoingthefollowing:
a. Aligntheremovablecachecardwiththebase cardandplacethehole inthecachedirectorycard
overthescrewB.
b. Slidetheremovablecachecardawayfromtheplastichandletoline upthecardconnector.(view
thejackscrewthroughtheholeto helpalignment).
8. AttachthefourscrewsCto theremovablecachecardasshown.
9. TightenscrewB onthecachedirectory card.
10. Turnthecardover.
11. AttachthescrewA tothebasecard.
Thisendsthisprocedure.
Replacing
the
Cache
Directory
Card
Attention: Allcardsaresensitiveto electrostaticdischarge.See“HandlingStaticSensitiveDevices”on
pagexi.Thisprocedureshouldonly beperformed ifdirectedfromanisolationprocedureora maintenance analysisprocedure(MAP).
v For non-571Ecards,goto“Toreplacea non-571Ecachedirectory card.”
v For 571Ecards,goto“Toreplacethe571ECacheDirectorycard”onpage38.
Attention: Allcardsaresensitiveto electrostaticdischarge.See“HandlingStaticSensitiveDevices”on
pagexi.
v Toreplaceanon-571Ecachedirectory card,dothefollowing:
1. Removethecardusingthecardremoveandreplaceprocedure forthemodelorexpansionuniton
whichyou areworking.
2. LocatethecachedirectorycardB.ItisasmallrectangularcardmountedontheI/Ocard(seethe
followingfiguresfor youradapter).
3. Unseat theconnectoronthecachedirectory cardbywigglingthetwocornersfurthestfromthe
mountingpegusinga rockingmotion.Then,pivotthecachedirectory cardbackoverthemounting pegto disengagethecachedirectory cardfromthemountingpeg.
4. Installthereplacementcardbyseating itontheconnectorandmountingpeg.
B
RZAR6586-1
IPHAU609-0
5. Thisendstheprocedure.
v Toreplacethe571ECacheDirectorycard,dothefollowing:
1. Locatetheretainingscrewfor thecachedirectorycardB.
2. Loosentheretainingscrew locatedonthecachedirectorycard onefull turn.Donotremovethe
retaining screw.
3. Unseat thecachedirectory cardfromtheconnector.Tounseatandremovethecachedirectorycard,
pullthecachedirectory cardinthedirectionoftheplasticcardhandle.Youmightneedto pullthe cardononesidethentheotherto helpdisengagetheconnector.Whenyouhave disengagedthe connector,liftthecachedirectorycardupandofftheretainingscrew.
4. Installthereplacementcachedirectory card.Toinstall thereplacement cachedirectorycard,placeit
downovertheretainingscrew, thenslideit awayfromtheplasticcardhandletofullyseatit inthe connector.
5. Tighten theretaining screw.
6. Thisendstheprocedure.
Rechargeable
Battery
Maintenance
Rechargeablebatterymaintenancetasksdescribedinthissectioninclude displayingrechargeable battery information,forcingarechargeable batteryerror,andreplacingtherechargeableCacheBattery Pack.