Chapter 4. Continuous availability and manageability
4.3 Serviceability
4.3.1 Detecting
The first and most crucial component of a solid serviceability strategy is the ability to accurately and effectively detect errors when they occur. Although not all errors are a guaranteed threat to system availability, those that go undetected can cause problems because the system does not have the opportunity to evaluate and act if necessary. POWER processor-based systems employ IBM System z® server-inspired error detection
mechanisms that extend from processor cores and memory to power supplies and hard drives.
Service processor
The service processor is a separately powered microprocessor, separate from the main instruction processing complex. The service processor provides the capabilities for:
POWER Hypervisor (system firmware) and Hardware Management Console connection surveillance
Several remote power control options
Reset and boot features
Environmental monitoring
The service processor monitors the server’s built-in temperature sensors, sending instructions to the system fans to increase rotational speed when the ambient temperature is above the normal operating range. Using an architected operating system interface, the service processor notifies the operating system of potential environmental-related
problems so that the system administrator can take appropriate corrective actions before a critical failure threshold is reached.
The service processor can also post a warning and initiate an orderly system shutdown when:
– The operating temperature exceeds the critical level (for example, failure of air conditioning or air circulation around the system).
– The system fan speed is out of operational specification, for example, because of multiple fan failures.
– The server input voltages are out of operational specification. The service processor can immediately shut down a system when:
– Temperature exceeds the critical level or if the temperature remains above the warning level for too long.
– Internal component temperatures reach critical levels. – Non-redundant fan fails.
Placing calls
On systems without a Hardware Management Console, the service processor can place calls to report surveillance failures with the POWER Hypervisor, critical environmental faults, and critical processing faults even when the main processing unit is inoperable
Mutual Surveillance
The service processor monitors the operation of the POWER Hypervisor firmware during the boot process and watches for loss of control during system operation. It also allows the POWER Hypervisor to monitor service processor activity. The service processor can take appropriate action, including calling for service, when it detects the POWER Hypervisor firmware has lost control. Likewise, the POWER Hypervisor can request a service processor repair action if necessary
Availability
The auto-restart (reboot) option, when enabled, can reboot the system automatically following an unrecoverable firmware error, firmware hang, hardware failure, or environmentally induced (AC power) failure
Fault Monitoring
Built-in self-test (BIST) checks processor, cache, memory, and associated hardware required for proper booting of the operating system, when the system is powered on at the initial install or after a hardware configuration change (for example, an upgrade). If a non-critical error is detected or if the error occurs in a resource that can be removed from the system configuration, the booting process is designed to proceed to completion. The errors are logged in the system nonvolatile random access memory (NVRAM). When the operating system completes booting, the information is passed from the NVRAM into the system error log where it is analyzed by error log analysis (ELA) routines. Appropriate actions are taken to report the boot time error for subsequent service if required
Concurrent access to the service processors menus of the Advanced System Management Interface
This allows nondisruptive abilities to change system default parameters, interrogate service processor progress and error logs, set and reset server indicators, (Guiding Light for midrange and high-end servers, Light Path for low end servers): indeed, access all service processor functions without having to power-down the system to the standby state. This approach allows the administrator or service representative to dynamically access the menus from any Web browser-enabled console attached to the Ethernet service network concurrent with normal system operation.
Managing the interfaces for connecting uninterruptible power supply systems to the POWER processor-based systems, performing timed power-on (TPO) sequences, and interfacing with the power and cooling subsystem
Error checkers
IBM POWER processor-based systems contain specialized hardware detection circuitry that is used to detect erroneous hardware operations. Error checking hardware ranges from parity error detection coupled with processor instruction retry and bus retry, to ECC correction on caches and system buses. All IBM hardware error checkers have distinct attributes:
Continual monitoring of system operations to detect potential calculation errors
Attempt to isolate physical faults based on run time detection of each unique failure
Ability to initiate a wide variety of recovery mechanisms designed to correct the problem. The POWER processor-based systems include extensive hardware and firmware
Fault isolation registers
Error checker signals are captured and stored in hardware fault isolation registers (FIRs). The associated logic circuitry is used to limit the domain of an error to the first checker that encounters the error. In this way, run-time error diagnostics can be deterministic so that for every check station, the unique error domain for that checker is defined and documented. Ultimately, the error domain becomes the field-replaceable unit (FRU) call, and manual interpretation of the data is not normally required.
First-failure data capture (FFDC)
First-failure data capture (FFDC) is an error isolation technique, which ensures that when a fault is detected in a system through error checkers or other types of detection methods, the root cause of the fault will be captured without the need to re-create the problem or run an extended tracing or diagnostics program.
For the vast majority of faults, a good FFDC design means that the root cause will be detected automatically without intervention by a service representative. Pertinent error data related to the fault is captured and saved for analysis. In hardware, FFDC data is collected from the fault isolation registers and from the associated logic. In firmware, this data consists of return codes, function calls, and so forth.
FFDC
check stations
are carefully positioned within the server logic and data paths to ensure that potential errors can be quickly identified and accurately tracked to an FRU.This proactive diagnostic strategy is a significant improvement over the classic, less accurate “reboot and diagnose” service approaches.
Figure 4-6 Schematic of a FIR implementation
Fault isolation
The service processor interprets error data captured by the FFDC checkers (saved in the FIRs or other firmware-related data capture methods) in order to determine the root cause of the error event.
Root cause analysis might indicate that the event is recoverable, meaning that a service action point or need for repair has not been reached. Alternatively, it could indicate that a service action point has been reached, where the event exceeded a pre-determined threshold or was unrecoverable. Based upon the isolation analysis, recoverable error threshold counts may be incremented. No specific service action could be necessary when the event is recoverable.
When the event requires a service action, additional required information is collected to service the fault. For unrecoverable errors or for recoverable events that meet or exceed their service threshold, meaning that a service action point has been reached, a request for service is initiated through an error logging component.