Switchover and Failover Operations This chapter describes how the broker manages databases during switchover and
FS_FAILOVER_OBSERVER PRESENT
Primary SYNCHRONIZED Maximum Availability
YES
Standby SYNCHRONIZED Maximum Availability
Fast-Start Failover
In the following example, assume the network between the primary database and the observer has failed. In this case, the FS_FAILOVER_STATUS and FS_FAILOVER_ OBSERVER_PRESENT columns will appear as shown in the following table and fast-start failover will not occur:
5.5.7.3 What Happens if the Observer Fails?
If the primary and target standby databases stay connected but the connection to the observer is lost, then the broker reports that the configuration is not observed. The configuration and database status report that the observer is not running and return one of the following status messages:
ORA-16658: unobserved fast-start failover configuration
ORA-16820: fast-start failover observer is no longer observing this database
While the configuration is in the unobserved state, fast-start failover cannot happen. Therefore, the primary database can continue processing transactions, even if the target standby database fails. The configuration status returns the SUCCESS status after the observer reestablishes its connection to the primary database, which then notifies the target standby database.
5.5.7.4 Stopping the Observer
You may want to stop the observer when you no longer want to use fast-start failover (see Section 5.5.5, "Disabling Fast-Start Failover") or if you want to move the observer to a different host machine (see Section 5.5.7.5, "Moving the Observer to Another Computer").
To stop the observer when fast-start failover is enabled, the primary database and target standby database must be connected and communicating with each other. Stopping the observer does not disable the fast-start failover. However, fast-start failover cannot occur when the target standby database is in the unobserved state.
To stop the observer when fast-start failover is not enabled, the primary database must be running.
You can stop the observer while connected to any database in the broker configuration that has network connectivity to the primary database, as follows:
■ Using Enterprise Manager
Primary TARGET UNDER LAG LIMIT Maximum Performance
YES
Standby TARGET UNDER LAG LIMIT Maximum Performance
YES
Database FS_FAILOVER_STATUS FS_FAILOVER_OBSERVER_PRESENT
Primary SYNCHRONIZED NO Standby PRIMARY UNOBSERVED YES
See Also: Section 5.5.4, "Viewing Fast-Start Failover Configuration Statistics and Status"
Database FS_FAILOVER_STATUS Protection Mode
FS_FAILOVER_OBSERVER_ PRESENT
Fast-Start Failover
Choose the Stop Observer option on the first page of the fast-start failover wizard and click Continue at the bottom of the page. See the Enterprise Manager online help system for more information.
■ Using DGMGRL
Issue the following command:
DGMGRL> STOP OBSERVER;
See the STOP OBSERVER command on page 8-58 for more information.
5.5.7.5 Moving the Observer to Another Computer
To move the observer to another computer:
1. Stop the observer from any computer system in the broker configuration, as described in Section 5.5.7.4.
2. Start the observer on the new computer system, as described in Step 8 of
Section 5.5.2.
There is no need to disable fast-start failover when you move the observer.
5.5.7.6 How the Observer Maintains Fast-Start Failover Configuration Information
The observer persistently maintains information about the fast-start failover
configuration in a binary file created in the working directory where you started the observer. By default, the observer creates this file in the current working directory when it is started and names the file fsfo.dat. This file contains connect descriptors to both the primary and the target standby databases.
Ensure this file cannot be read by unauthorized users.
Once the observer is started, you cannot change the file's name and location. However, you can change the name or the location of the file if you start the observer using the DGMGRL START OBSERVER command and include the FILE qualifier. See the
START OBSERVER command on page 8-54 for more information.
If you want to use one Oracle home to start multiple observers, with each observer monitoring a different fast-start failover configuration, use the FILE qualifier to specify a unique observer configuration file location for each configuration to be monitored. If you want to capture any logging generated by the observer, use the LOGFILE option and ensure that file name is unique as well. For example:
% dgmgrl -logfile $ORACLE_HOME/rdbms/log/config1.log DGMGRL> CONNECT /@primary1;
DGMGRL> START OBSERVER FILE=$ORACLE_HOME/dbs/config1.dat;
% dgmgrl -logfile $ORACLE_HOME/rdbms/log/config2.log
Note: The observer does not stop immediately when you issue STOP OBSERVER command. When the broker receives the STOP OBSERVER request, it informs the observer the next time the observer contacts the broker.
Note: If the observer is stopped abnormally (for example, by typing CTRL/C), restart it and reference the existing fsfo.dat file with the FILE qualifier.
Fast-Start Failover
DGMGRL> CONNECT /@primary2;
DGMGRL> START OBSERVER FILE=$ORACLE_HOME/dbs/config2.dat;
5.5.8 Reinstating the Former Primary Database in the Broker Configuration
If a fast-start failover was initiated because the primary database had crashed or lost connectivity with the observer and target standby database, the observer
automatically attempts to reinstate the former primary database as a standby database, if the FastStartFailoverAutoReinstate configuration property is set to TRUE.
Reinstatement restores high availability to the broker configuration so that, in the event of a failure of the new primary database, another fast-start failover can occur. The reinstated database acts as the fast-start failover target for the new primary database, making a subsequent fast-start failover possible. The new standby database is a viable target of a failover when it begins receiving redo data received from the new primary database.
To allow the observer to automatically reinstate the former primary database, the database must be started and mounted, but it cannot be opened. The broker reinstates the database as a standby database of the same type as the former standby database of the new primary database.
If the former primary database cannot be reinstated automatically, you can manually reinstate it using either the DGMGRL REINSTATE command or Enterprise Manager. Step-by-step instructions for manual reinstatement are described in Section 5.4.3.
5.5.8.1 Requirements
Reinstatement is supported only after failover in a broker configuration. It also requires Flashback Database to be enabled on both the primary and target standby databases. Section 5.5.1 provides complete information about all of the fast-start failover and reinstatement requirements.
5.5.8.2 Restrictions on Reinstatement
The broker cannot automatically reinstate the former primary database if:
■ A fast-start failover occurred because a user-configurable condition was detected or was requested by an application by calling the DBMS_DG.INITIATE_FS_ FAILOVER function.
■ FastStartFailoverAutoReinstate is set to FALSE
■ Another failover or switchover occurred after the fast-start failover completed but
before the former primary database restarted
■ Fast-start failover was disabled
■ The observer cannot connect to the former primary database
■ The former primary database cannot connect to the new primary database
■ The former primary database and the new primary database are not configured in the same fast-start failover environment
■ The former primary database was disabled because of a manual failover when fast-start failover was disabled
Note: Standby databases that are disabled during switchover, manual failover, or fast-start failover will not be automatically reinstated.
Fast-Start Failover
If automatic reinstatement fails, the broker will log errors and the former primary database will remain in the mounted state. At this point, you can either:
■ Disable fast-start failover (described in Section 5.5.5) and attempt to open the former primary database
■ Manually reinstate the former primary database, as described in Section 5.4.3
5.5.8.3 How the Broker Handles a Failed Reinstatement
If a failure occurs once a reinstatement operation (automatic or manual) is underway, the broker logs the appropriate information in the broker configuration files and "broker log" files. The former primary database is disabled. Most in-progress failures cannot be restarted (for example, archived redo log file corruption on the primary database). You must manually reenable the database as a standby database.
5.5.9 Shutting Down Databases In a Fast-Start Failover Environment
Perform the following steps if you need to shut down the primary or standby databases:
1. Shut down the observer and wait for the FS_FAILOVER_OBSERVER_PRESENT column in the V$DATABASE fixed view to contain the value "NO" for both the primary and target standby databases. This ensures that a fast-start failover will not occur while you are shutting down the primary database.
2. Shut down the primary database and the target standby database using either DGMGRL SHUTDOWN command or the SQL*Plus SHUTDOWN statement. When restarting the databases, you may restart them in any order. When both databases have been restarted, you may restart the observer.
See Also: Section 5.4.3.2, "How to Re-create and Reenable a Disabled Database"