MIMIX
Availability
Version 7.1
MIMIX Operations - 5250 User Guide January 2014
Version: 7.1.19.00
© Copyright 1999, 2014 Vision Solutions®, Inc. All rights reserved.
The information in this document is subject to change without notice and is furnished under a license
agreement. This document is proprietary to Vision Solutions, Inc., and may be used only as authorized in our license agreement. No portion of this manual may be copied or otherwise reproduced without the express written consent of Vision Solutions, Inc.
Vision Solutions provides no expressed or implied warranty with this manual.
The following are trademarks or registered trademarks of their respective organizations or companies: • MIMIX and Vision Solutions are registered trademarks and AutoGuard, Data Manager, Director, Dynamic
Apply, ECS/400, GeoCluster, IntelliStart, Integrator, iOptimize, iTERA, iTERA Availability, MIMIX AutoNotify, MIMIX Availability, MIMIX Availability Manager, MIMIX DB2 Replicator, MIMIX Director, MIMIX dr1, MIMIX Enterprise, MIMIX Global, MIMIX Monitor, MIMIX Object Replicator, MIMIX Professional, MIMIX Promoter, OMS/ODS, RecoverNow, Replicate1, RJ Link, SAM/400, Switch Assistant, Vision AutoValidate, and Vision Suite are trademarks of Vision Solutions, Inc.
• Double-Take Share, Double-Take Availability, and Double-Take RecoverNow—DoubleTake Inc.
• AIX, AIX 5L, AS/400, DB2, eServer, IBM, Informix, i5/OS, iSeries, OS/400, Power, System i, System i5, System p, System x, System z, and WebSphere—International Business Machines Corporation.
• Adobe and Acrobat Reader—Adobe Systems, Inc. • HP-UX—Hewlett-Packard Company.
• Teradata—Teradata Corporation. • Intel—Intel Corporation.
• Java, all Java-based trademarks, and Solaris—Sun Microsystems, Inc. • Linux—Linus Torvalds.
• Internet Explorer, Microsoft, Windows, and Windows Server—Microsoft Corporation. • Mozilla and Firefox—Mozilla Foundation.
• Netscape—Netscape Communications Corporation. • Oracle—Oracle Corporation.
• Red Hat—Red Hat, Inc. • Sybase—Sybase, Inc.
• Symantec and NetBackup—Symantec Corporation. • UNIX and UNIXWare—the Open Group.
All other brands and product names are trademarks or registered trademarks of their respective owners. If you need assistance, contact Vision Solutions’ CustomerCare team at:
CustomerCare Vision Solutions, Inc.
Telephone: 1.800.337.8214 or 1.949.724.5465 Email: [email protected]
Who this book is for... 11
What is in this book ... 11
The MIMIX documentation set ... 11
Sources for additional information... 13
How to contact us... 14
Chapter 1 MIMIX overview 15 MIMIX concepts... 17
Product concepts... 17
System role concepts ... 18
Journaling concepts ... 19
Configuration concepts... 20
Process concepts ... 21
Additional switching concepts ... 22
Best practices for maintaining your MIMIX environment ... 23
Authority to products and commands... 23
Accessing the MIMIX Main Menu... 24
Chapter 2 MIMIX policies 26 Environment considerations for policies... 27
Policies in environments with more than two nodes or bi-directional replication. 27 When to disable automatic recovery for replication and auditing ... 28
Disabling audits and recovery when using the MIMIX CDP feature ... 29
Setting policies - general ... 29
Changing policies for an installation ... 29
Changing policies for a data group... 30
Resetting a data group-level policy to use the installation level value ... 30
Policies which affect an installation ... 31
Changing retention criteria for procedure history ... 31
Policies which affect replication... 32
Errors handled by automatic database recovery ... 33
Errors handled by automatic object recovery ... 34
Policies which affect auditing ... 36
Policies for auditing runtime behavior ... 36
Policies for submitting audits automatically ... 37
When automatically submitted audits run... 38
Changing auditing policies ... 41
Changing when automatic audits are allowed to run... 41
Changing scheduling criteria for automatic audits... 41
Changing the selection frequency of priority auditing categories ... 42
Changing the audit level policy when switching ... 43
Changing the system where audits are performed... 43
Changing retention criteria for audit history... 43
Restricting auditing based on the state of the data group ... 44
Preventing audits from running ... 45
Disabling all auditing for an installation ... 46
Disabling all auditing for a data group ... 46
Disabling automatically submitted audits... 46
Policies for switching with model switch framework ... 48
Policy descriptions... 50
Chapter 3 Checking status in environments with application groups 60 Checking application group status ... 60
Resolving problems reported in the Monitors field ... 61
Resolving problems reported in the Notifications field ... 63
Resolving problems reported in Status columns ... 64
Resolving a procedure status problem ... 64
Resolving an *ATTN status for an application group... 65
Resolving other common status values for an application group ... 66
Status for Work with Node Entries ... 66
Status for Work with Data Resource Group Entries ... 68
Verifying the sequence of the recovery domain ... 70
Changing the sequence of backup nodes ... 71
Examples of changing the backup sequence ... 73
Chapter 4 Working with status of procedures and steps 77 Displaying status of procedures ... 78
Displaying status of the last run of all procedures ... 78
Displaying available status history of procedure runs ... 79
Resolving problems with procedure status... 80
Responding to a procedure in *MSGW status... 81
Resolving a *FAILED or *CANCELED procedure status... 82
Displaying status of steps within a procedure run ... 83
Resolving problems with step status ... 85
Responding to a step with a *MSGW status ... 87
Resolving *CANCEL or *FAILED step statuses ... 88
Acknowledging a procedure ... 89
Running a procedure... 90
Resuming a procedure ... 91
Overriding the attributes of a step ... 91
Canceling a procedure ... 92
Chapter 5 Monitoring status with MIMIX Availability Status 93 Checking replication status from the MIMIX Availability Status display ... 95
Checking audit and notification status from the MIMIX Availability Status display.... 96
Checking status of supporting services from the MIMIX Availability Status display.. 96
Chapter 6 Working with data group status 98 The Work with Data Groups display... 99
Problems reflected in the Audits/Recov./Notif. field ... 101
Problems reflected in the Data Group column ... 101
Resolving problems highlighted in the Data Group column... 102
Manager problems reflected in the Source and Target columns... 103
Replication problems reflected in the Source and Target columns ... 103
Setting the automatic refresh interval ... 104
Working with the detailed status of data groups... 105
Displaying data group detailed status ... 105
Identifying replication processes with backlogs... 115
Data group status in environments with journal cache or journal state ... 117
Resolving a problem with journal cache or journal state ... 119
Chapter 7 Working with audits 121 Auditing overview ... 122
Components of an audit ... 122
Phases of audit processing ... 123
Object selection methods for automatic audits... 123
How priority auditing determines what objects to select... 124
How audits are submitted automatically ... 124
Audit status and results ... 125
Audit compliance ... 125
Guidelines and considerations for auditing ... 126
Auditing best practices ... 126
Considerations for specific audits... 127
Recommendations when checking audit results ... 127
Displaying audit runtime status ... 129
Running an audit immediately ... 131
Resolving audit problems ... 133
Checking the job log of an audit ... 135
Ending audits... 136
Displaying audit history ... 137
Audits with no selected objects ... 139
Working with audited objects... 139
Displaying audited objects from a specific audit run ... 141
Displaying a customized list of audited objects ... 141
Working with audited object history... 142
Displaying the audit history for a specific object... 143
Displaying audit compliance... 144
Determining whether auditing is within compliance... 145
Displaying scheduling information for automatic audits ... 147
Chapter 8 Working with system-level processes 149 Displaying status of system-level processes... 149
Resolving *ACTREQ status for a system manager ... 151
Checking for a system manager backlog ... 151
Starting a system manager or a journal manager ... 152
Ending a system manager or a journal manager ... 152
Starting collector services ... 152
Ending collector services... 153
Starting target journal inspection processes ... 153
Ending target journal inspection processes... 154
Displaying status of target journal inspection ... 155
Displaying results of target journal inspection ... 156
Displaying details associated with target journal inspection notifications... 157
Displaying messages for TGTJRNINSP notifications... 157
Displaying notifications... 160
What information is available for notifications ... 160
Detailed information... 161
Options for working with notifications ... 162
Notifications for newly created objects ... 163
Displaying recoveries ... 164
What information is available for recoveries... 165
Detailed information... 166
Options for working with recoveries ... 166
Orphaned recoveries ... 167
Determining whether a recovery is orphaned... 167
Removing an orphaned recovery ... 168
Chapter 10 Starting and ending replication 169 Before starting replication... 171
Commands for starting replication... 171
What is started with the STRMMX command... 171
STRMMX and ENDMMX messages... 172
What is started by the default START procedure for an application group ... 172
Choices when starting or ending an application group... 172
What occurs when a data group is started ... 174
Journal starting point identified on the STRDG request ... 175
Journal starting point when the object send process is shared ... 175
Clear pending and clear error processing ... 175
Starting MIMIX... 179
Starting an application group... 180
Starting selected data group processes ... 181
Starting replication when open commit cycles exist ... 183
Checking for open commit cycles... 183
Resolving open commit cycles ... 183
Before ending replication... 184
Commands for ending replication... 184
Command choice by reason for ending replication ... 184
Additional considerations when ending replication... 186
Ending immediately or controlled ... 186
Controlling how long to wait for a controlled end to complete ... 187
Ending all or selected processes... 187
When to end the RJ link ... 188
What is ended by the ENDMMX command ... 188
What is ended by the default END procedure for an application group ... 189
What occurs when a data group is ended ... 190
Ending MIMIX... 192
Ending with default values... 192
Ending by prompting the ENDMMX command... 192
After you end MIMIX products ... 193
Ending an application group... 194
Ending a data group in a controlled manner ... 195
Ending selected data group processes ... 198
What replication processes are started by the STRDG command... 199
What replication processes are ended by the ENDDG command ... 203
Chapter 11 Resolving common replication problems 207 Working with message queues ... 208
Working with the message log ... 209
Working with user journal replication errors ... 210
Working with files needing attention (replication and access path errors)... 210
Working with journal transactions for files in error... 213
Placing a file on hold ... 214
Ignoring a held file ... 214
Releasing a held file at a synchronization point ... 215
Releasing a held file ... 215
Releasing a held file and clearing entries... 216
Correcting file-level errors ... 216
Correcting record-level errors... 217
Record written in error ... 217
Working with tracking entries ... 219
Accessing the appropriate tracking entry display ... 219
Holding journal entries associated with a tracking entry ... 221
Ignoring journal entries associated with a tracking entry... 222
Waiting to synchronize and release held journal entries for a tracking entry .... 222
Releasing held journal entries for a tracking entry ... 223
Releasing and clearing held journal entries for a tracking entry... 223
Removing a tracking entry... 223
Working with objects in error ... 224
Using the Work with DG Activity Entries display ... 225
Retrying data group activity entries ... 227
Retrying a failed data group activity entry ... 227
Determining whether an activity entry is in a delay/retry cycle ... 228
Removing data group activity history entries... 229
Chapter 12 Starting, ending, and verifying journaling 230 What objects need to be journaled... 231
Authority requirements for starting journaling... 232
MIMIX commands for starting journaling... 233
Journaling for physical files ... 235
Displaying journaling status for physical files ... 235
Starting journaling for physical files ... 235
Ending journaling for physical files ... 236
Verifying journaling for physical files ... 237
Journaling for IFS objects... 238
Displaying journaling status for IFS objects ... 238
Starting journaling for IFS objects ... 238
Ending journaling for IFS objects ... 239
Verifying journaling for IFS objects... 240
Ending journaling for data areas and data queues... 242
Verifying journaling for data areas and data queues ... 243
Chapter 13 Switching 244 About switching ... 244
Planned switch ... 245
Unplanned switch ... 246
Switching application group environments with procedures... 247
Switching data group environments with MIMIX Model Switch Framework ... 248
Switching an application group... 250
Switching a data group-only environment ... 251
Switching to the backup system ... 251
Synchronizing data and starting MIMIX on the original production system ... 252
Switching to the production system ... 252
Determining when the last switch was performed ... 253
Checking the last switch date ... 253
Problems checking switch compliance... 254
Performing a data group switch... 255
Switch Data Group (SWTDG) command... 257
Chapter 14 Less common operations 259 Starting the TCP/IP server ... 260
Ending the TCP/IP server... 261
Working with objects ... 262
Displaying long object names... 262
Considerations for working with long IFS path names ... 262
Displaying data group spooled file information... 262
Viewing status for active file operations ... 263
Displaying a remote journal link ... 264
Displaying status of a remote journal link... 265
Identifying data groups that use an RJ link ... 267
Identifying journal definitions used with RJ ... 268
Disabling and enabling data groups ... 269
Procedures for disabling and enabling data groups ... 270
Determining if non-file objects are configured for user journal replication... 271
Determining how IFS objects are configured ... 271
Determining how data areas or data queues are configured ... 272
Using file identifiers (FIDs) for IFS objects ... 273
Operating a remote journal link independently... 274
Starting a remote journal link independently ... 274
Ending a remote journal link independently ... 274
Chapter 15 Troubleshooting - where to start 276 Gathering information before reporting a problem ... 278
Obtaining MIMIX and IBM i information from your system ... 278
Reducing contention between MIMIX and user applications... 279
Data groups cannot be ended ... 280
Verifying a communications link for system definitions ... 281
Data groups cannot be started ... 285
Cannot start or end an RJ link... 286
Removing unconfirmed entries to free an RJ link... 286
RJ link active but data not transferring ... 287
Errors using target journal defined by RJ link... 288
Verifying data group file entries... 289
Verifying data group data area entries ... 289
Verifying key attributes ... 289
Working with data group timestamps ... 291
Automatically creating timestamps ... 291
Creating additional timestamps ... 291
Creating timestamps for remote journaling processing ... 292
Deleting timestamps ... 293
Displaying or printing timestamps ... 293
Removing journaled changes... 294
Performing journal analysis ... 295
Removing journal analysis entries for a selected file ... 297
Appendix A Interpreting audit results - supporting information 299 Interpreting results for configuration data - #DGFE audit... 300
When the difference is “not found” ... 302
Interpreting results of audits for record counts and file data ... 303
What differences were detected by #FILDTA... 303
What differences were detected by #MBRRCDCNT ... 304
Interpreting results of audits that compare attributes ... 306
What attribute differences were detected ... 306
Where was the difference detected ... 308
What attributes were compared ... 309
Appendix B IBM Power™ Systems operations that affect MIMIX 310 MIMIX procedures when performing an initial program load (IPL) ... 310
MIMIX procedures when performing an operating system upgrade... 311
Prerequisites for performing an OS upgrade on either system ... 312
MIMIX-specific steps for an OS upgrade on the backup system... 313
MIMIX-specific steps for an OS upgrade on the production system with switching 315 MIMIX-specific steps for an OS upgrade on the production system without switch-ing... 316
MIMIX procedures when upgrading hardware without a disk image change ... 318
Considerations for performing a hardware system upgrade without a disk image change... 318
MIMIX-specific steps for a hardware upgrade without a disk image change... 319
Hardware upgrade without a disk image change - preliminary steps ... 319
Hardware upgrade without a disk image change - subsequent steps ... 320
Hardware upgrade with a disk image change - subsequent steps ... 323 Handling MIMIX during a system restore ... 325 Prerequisites for performing a restore of MIMIX ... 325
Who this book is for
The MIMIX Operations - 5250 book describes how to perform routine operational tasks and basic troubleshooting for MIMIX® Enterprise™ and MIMIX® Professional™ from a 5250 emulator.
What is in this book
The MIMIX Operations - 5250 book provides these distinct types of information: • A summary of concepts within MIMIX
• Application group and data group status and troubleshooting • Audit status, troubleshooting, scheduling, and history
• Procedures for starting, ending, and switching replication • Procedures for starting, ending, and verifying journaling
• Procedures for handling MIMIX when performing operations such as IPLs or hardware and operating system upgrades.
The MIMIX documentation set
The following documents about MIMIX® Availability™ products are available: Using License Manager
License Manager currently supports MIMIX® Availability™, iTERA Availability™, and iOptimize™. This book describes software requirements, system security, and other planning considerations for installing software and software fixes for Vision Solutions products that are supported through License Manager. The preferred way to obtain license keys and install software is by using Vision AutoValidate™ and the product’s Installation Wizard. However, if you cannot use the wizard or AutoValidate, this book provides instructions for obtaining licenses and installing software from a 5250 emulator. This book also describes how to use the
additional security functions from Vision Solutions which are available for License Manager and MIMIX and implemented through License Manager.
MIMIX Administrator Reference
This book provides detailed conceptual, configuration, and programming information for MIMIX® Enterprise™ and MIMIX® Professional™. It includes checklists for setting up several common configurations, information for planning what to replicate, and detailed advanced configuration topics for custom needs. It also identifies what information can be returned in outfiles if used in automation. MIMIX Operations with IBM i Clustering
This book is for administrators and operators in an IBM i clustering environment who either use the basic support for IBM i clustering provided within MIMIX or who use MIMIX® Global™ to integrate cluster management with MIMIX logical
focuses on addressing problems reported in MIMIX status and basic operational procedures such as starting, ending, and switching.
MIMIX Operations - 5250
This book provides high level concepts and operational procedures for managing your high availability environment using MIMIX® Enterprise™ or MIMIX®
Professional™ from a 5250 emulator. This book focuses on tasks typically performed by an operator, such as checking status, starting or stopping replication, performing audits, and basic problem resolution.
Using MIMIX Monitor
This book describes how to use the MIMIX Monitor user and programming interfaces available with MIMIX® Enterprise™ or MIMIX® Professional™. This book also includes programming information about MIMIX Model Switch Framework and support for hardware switching.
Using MIMIX Promoter
This book describes how to use MIMIX commands for copying and reorganizing active files. MIMIX Promoter is available with MIMIX® Enterprise™ and as no-charge feature for MIMIX® Professional™.
MIMIX for IBM WebSphere MQ
This book identifies requirements for the MIMIX for MQ feature which supports replication in IBM WebSphere MQ environments. This book describes how to configure MIMIX for this environment and how to perform the initial
Sources for additional information
This book refers to other published information. The following information, plus additional technical information, can be located in the IBM System i and i5/OS Information Center.
From the Information center you can access these IBM Power™ Systems topics, books, and redbooks:
• Backup and Recovery • Journal management
• DB2 Universal Database for IBM Power™ Systems Database Programming • Integrated File System Introduction
• Independent disk pools • OptiConnect for OS/400 • TCP/IP Setup
• IBM redbook Striving for Optimal Journal Performance on DB2 Universal Database for iSeries, SG24-6286
• IBM redbook AS/400 Remote Journal Function for High Availability and Data Replication, SG24-5189
• IBM redbook Power™ Systems iASPs: A Guide to Moving Applications to Independent ASPs, SG24-6802
The following information may also be helpful if you replicate journaled data areas, data queues, or IFS objects:
• DB2 UDB for iSeries SQL Programming Concepts • DB2 Universal Database for iSeries SQL Reference
How to contact us
For contact information, visit our Contact CustomerCare web page.
If you are current on maintenance, support for MIMIX products is also available when you log in to Support Central.
This book provides operational information and procedures for using MIMIX® Enterprise™ and MIMIX® Professional™ through its 5250 emulator user interface. For simplicity, this book uses the term MIMIX to refer to the functionality provided by either product unless a more specific name is necessary.
MIMIX® Availability™ version 7.1 provides high availability for your critical data in a production environment on IBM Power™ Systems through real-time replication of changes and the ability to quickly switch your production environment to a ready backup system. These capabilities allow your business operations to continue when you have planned or unplanned outages in your System i environment. MIMIX also provides advanced capabilities that can help ensure the integrity of your MIMIX environment.
Replication: MIMIX continuously captures changes to critical database files and objects on a production system, sends the changes to a backup system, and applies the changes to the appropriate database file or object on the backup system. The backup system stores exact duplicates of the critical database files and objects from the production system.
MIMIX uses two replication paths to address different pieces of your replication needs. These paths operate with configurable levels of cooperation or can operate independently.
• The user journal replication path captures changes to critical files and objects configured for replication through a user journal. When configuring this path, shipped defaults use the remote journaling function of the operating system to simplify sending data to the remote system. In previous versions, MIMIX DB2 Replicator provided this function.
• The system journal replication path handles replication of critical system objects (such as user profiles, program objects, or spooled files), integrated file system (IFS) objects, and document library object (DLOs) using the system journal. In previous versions MIMIX Object Replicator provided this function.
Configuration choices determine the degree of cooperative processing used between the system journal and user journal replication paths when replicating database files, IFS objects, data areas, and data queues.
your replication environment. MIMIX automatically detects and corrects potential problems during replication and auditing. MIMIX also helps to ensure the integrity of your MIMIX configuration by automatically verifying that the files and objects being replicated are what is defined to your configuration.
MIMIX is shipped with these capabilities enabled. Incorporated best practices for maintaining availability and switch-readiness are key to ensuring that your MIMIX environment is in tip-top shape for protecting your data. User interfaces allow you to fine-tune to the needs of your environment.
Analysis: MIMIX also provides advanced analysis capabilities through the MIMIX portal application for Vision Solutions Portal (VSP). When using the VSP user interface, you can see what objects are configured for replication as well as what replicated objects on the target system have been changed by people or programs other than MIMIX. (Objects changed on the target system affect your data integrity.) You can also check historical arrival and backlog rates for replication to help you identify trends in your operations that may affect MIMIX performance.
Uses: MIMIX is typically used among systems in a network to support a hot backup system. Simple environments have one production system and one backup system. More complex environments have multiple production systems or backup systems. MIMIX can also be used on a single system.
You can view the replicated data on the backup system at any time without affecting productivity. This allows you to generate reports, submit (read-only) batch jobs, or perform backups to tape from the backup system. In addition to real-time backup capability, replicated databases and objects can be used for distributed processing, allowing you to off-load applications to a backup system.
The topics in this chapter include:
• “MIMIX concepts” on page 17 summarizes key concepts that you need to know about MIMIX.
• “Best practices for maintaining your MIMIX environment” on page 23 summarizes recommendations from Vision Solutions.
• “Authority to products and commands” on page 23 identifies authority levels to MIMIX functions when additional security features provided by Vision Solutions are used.
MIMIX concepts
The following subtopics organize the basic concepts associated with MIMIX® into related groups. More detailed information is available in the MIMIX Administrator Reference book.
Product concepts
MIMIX installation - The network of IBM Power™ Systems systems that transfer data and objects among each other using functions of a common MIMIX product. A MIMIX installation is defined by the way in which you configure the MIMIX product for each of the participating systems. A system can participate in multiple independent MIMIX installations.
Replication - The activity that MIMIX performs to continuously capture changes to critical database files and objects on a production system as they occur, send the changes to a backup system, and apply the changes to the appropriate database file or object on the backup system.
Switch - The process by which a production environment is moved from one system to another system and the production environment is made available there. A switch may be performed as part of a planned event such as for system maintenance, or an unplanned event such as a power or equipment failure. MIMIX provides customizable functions for switching.
Audits - Audits are predetermined programs that are used to check for differences in replicated objects and other conditions between systems. Audits run and can correct detected problems automatically. Policies control when audits run and many other aspects of how audits are performed. Additional auditing concepts and
recommendations are described in the auditing chapter of this book.
Automatic recovery - MIMIX provides a set of functions that provide the ability to automatically correct problems detected in a MIMIX installation during database replication, object replication, and auditing. During these activities, when MIMIX detects any of a set of scenarios known to interfere with maintaining your MIMIX environment, it will automatically start recovery actions to correct them. Through policies, you have the ability to disable automatic recovery in any of these areas at the installation or data group level.
Application group - A MIMIX construct used to group and control resources from a single point in a way that maintains relationships between them. The use of
application groups is best practice for MIMIX® Professional™ and MIMIX® Enterprise™ and required for MIMIX® Global™.
Data group - A MIMIX construct that is used to control replication activities. A data group is a logical grouping of database files, data areas, objects, IFS objects, DLOs, or a combination thereof that defines a unit of work by which MIMIX replication activity is controlled. A data group may represent an application, a set of one or more
Prioritized status - MIMIX assigns a priority to status values to ensure that problems with the highest priorities, those for detected problems or situations that require immediate attention or intervention, are reflected on the highest level of the user interface. Additional detail and lower priority items can be viewed by drilling down to the next level within the interfaces. Those interfaces are the Work with Systems display and depending on your configuration, either the Work with Application Groups display or the Work with Data Groups display.
Policies - A policy is a mechanism used to enable, disable, or provide input to a function such as replication, auditing, or MIMIX Model Switch Framework. For most policies, the initially shipped values apply to an installation. However, policies can be changed and most can also be overridden for individual data groups. Policies that control when audits are automatically performed can be set only for each specific combination of audit rule and data group.
Notifications - A notification is the resulting automatic report associated with an event that has already occurred. The severity of a notification is reflected in the overall status of the installation. Notifications can be generated by a process, program, command, or monitor. Because the originator of notifications varies, it is important to note that notifications can represent both real-time events as well as events that occurred in the past but, due to scheduling, are being reported in the present.
Recoveries - This term recovery is used in two ways. The most common use refers to the recovery action taken by a replication process or an audit to correct a detected difference when automatic recovery polices are enabled. The second use refers to a temporary report that provides details about a recovery action in progress that is created when the recovery action starts and is removed when it completes.
System role concepts
MIMIX uses several pairs of terms to refer to the role of a system within a particular context. These terms are not interchangeable.
Production system and backup system - These terms describe the role of a system relative to the way applications are used on that system.
A production system is the system currently running the production workload for the applications. In normal operations, the production system is the system on which the principal copy of the data and objects associated with the application exist.
A backup system is the system that is not currently running the production workload for the applications. In normal operations, the backup system is the system on which you maintain a copy of the data and objects associated with the application. These roles are not always associated with a specific system. For example, if you switch application processing to the backup system, the backup system temporarily becomes the production system.
A source system is the system from which MIMIX replication activity between two systems originates. In replication, the source system contains the journal entries. Information from the journal entries is either replicated to the target system or used to identify objects to be replicated to the target system.
A target system is the system on which MIMIX replication activity between two systems completes.
Management system and network system - These terms define the role of a system relative to how the products interact within a MIMIX installation. These roles remain associated with the system within the MIMIX installation to which they are defined. One system in the MIMIX installation is designated as the management system and the remaining one or more systems are designated as network systems. A management system is the system in a MIMIX installation that is designated as the control point for all installations of the product within the MIMIX installation. The management system is the location from which work to be performed by the product is defined and maintained. Often the system defined as the management system also serves as the backup system during normal operations.
A network system is any system in a MIMIX installation that is not designated as the management system (control point) of that MIMIX installation. Work definitions are automatically distributed from the management system to a network system. Often a system defined as a network system also serves as the production system during normal operations.
Journaling concepts
MIMIX uses journaling to perform replication and to support newer analysis functionality.
Journaling and object auditing - Journaling and object auditing are techniques that allow object activity to be logged to a journal. Journaling logs activity for selected objects of specific object types to a user journal. Object auditing logs activity for all objects to the security audit journal (QAUDJRN, the system journal), including those defined to a user journal. MIMIX relies on these techniques and the entries placed in the journal receivers for replicating logged activity.
Journal - An IBM i system object that identifies the objects being journaled and the journal receivers associated with the journal. The system journal is a specialized journal on the system which MIMIX uses.
Journal receiver - An IBM i system object that is associated with a journal and contains the log of all activity for objects defined to the journal.
Journal entry - A record added to a journal receiver that identifies an event that occurred on a journaled object. MIMIX uses file and record level journal entries to recreate the object on a designated system.
MIMIX uses remote journaling for transferring data to be replicated from the source system to the target system.
Configuration concepts
MIMIX configuration provides considerable flexibility to enable supporting a wide variety of customer environments. Configuration is implemented through sets of related commands. The following terms describe configuration concepts.
Definitions - MIMIX uses several types of named definitions to identify related configuration choices.
• System definitions identify systems that participate in a MIMIX installation. Each system definition identifies one system.
• Transfer definitions identify the communications path and protocol to be used between systems.
• Journal definitions identify journaling environments that MIMIX uses for replication Each journal definition identifies a system and characteristics of the journaling environment on that system.
• Data group definitions identify the characteristics of how replication occurs between two systems. Each data group definition determines the direction in which replication occurs between the systems, whether that direction can be switched, and the default processing characteristics for replication processes. • Application group definitions identify whether the replication environment does or
does not use IBM i clustering. When clustering is used, the application group also defines information about an application or proprietary programs necessary for controlling operations in the clustering environment.
Data group entries - A data group entry is a configuration construct that identifies a source of information to be replicated by or excluded from replication by a data group. Each entry identifies at least one object and its location on the source system. Classes of data group entries are based on object type. MIMIX uses data group entries to determine whether a journal entry should be replicated. Data groups that replicate from both the system journal and a user journal can have any combination of data group entries.
Remote journal link (RJ link) - An RJ link is a MIMIX configuration element that identifies an IBM i remote journaling environment used by user journal replication processes. An RJ link identifies the journal definitions that define the source and target journals, primary and secondary transfer definitions for the communications path used by MIMIX, and whether the IBM i remote journal function sends journal entries asynchronously or synchronously.
Tracking entries - Tracking entries identify objects that can be replicated using advanced journaling techniques and assist with tracking the status of their replication. A unique tracking entry is associated with each IFS object, data area, and data queue that is eligible for replication using advanced journaling. IFS tracking entries identify eligible, existing IFS objects while object tracking entries identify eligible, existing data areas and data queues.
Process concepts
The following terms identify MIMIX processes. Some, like the system manager, are required to allow MIMIX to function. Others, like procedures, are used only when invoked by users.
Replication path - A replication path is a series of processes used for replication that represent the critical path on which data to be replicated moves from its origin to its destination. MIMIX uses two replication paths to accommodate differences in how replication occurs for user journal and system journal entries. These paths operate with configurable levels of cooperation or can operate independently.
• The user journal replication path captures changes to critical files and objects configured for replication through a user journal. When configuring this path, shipped defaults use the remote journaling function of the operating system to simplify sending data to the remote system. The changes are applied to the target system.
• The system journal replication path handles replication of critical system objects (such as user profiles, program objects, or spooled files), integrated file system (IFS) objects, and document library object (DLOs) using the system journal. Information about the changes are sent to the target system where it is applied. System manager - The system manager is a pair of communications jobs between the management system and a network system which must be active to enable replication. The system manager monitors for configuration changes and
automatically moves any configuration changes to the network system. Dynamic status changes are also collected and returned to the management system. The system manager also gathers messages and timestamp information from the network system and places them in a message log and timestamp file on the management system. In addition, the system manager performs periodic maintenance tasks, including cleanup of the system and data group history files.
Journal manager - The journal manager is a job on each system that MIMIX uses to maintain the journaling environment on that system. By default, MIMIX performs both change management and delete management for journal receivers used by the replication process.
Collector services - A group of jobs that are necessary for MIMIX to track historical data and to support using the MIMIX portal application within the Vision Solutions Portal. One or more collector service jobs collect and combine MIMIX status from all systems.
node to be recognized by the other nodes in the cluster. MIMIX integrates starting and stopping cluster services into status and commands for controlling processes that run at the system level.
Target journal inspection - A MIMIX process that reads a journal on a system being used as the target system for replication. The process identifies people or processes other than MIMIX that accessed replicated objects on the target system. Users can access the resulting information from the Replicated Objects portlet within the MIMIX portal application in Vision Solutions Portal.
Procedures and steps - Procedures and steps are a highly customizable means of performing operations for application groups. A set of default procedures for each application group provide the ability to start, end, perform pre-check activity for switching, and switch the application group. Each operation is performed by a procedure that consists of a sequence of steps and multiple jobs. Each step calls a predetermined step program to perform a specific sub-task of the larger operation. Steps also identify runtime attributes for handling before and after the program call within the context of the procedure.
Log space - A MIMIX object that provides an efficient storage and manipulation mechanism for replicated data that is temporarily stored on the target system during the receive and apply processes.
Additional switching concepts
The following concepts are specific to switching.
Environments configured with application groups perform switching through procedures.
Planned switch - An intentional change to the direction of replication for any of a variety of reasons. You may need to take the system offline to perform maintenance on its hardware or software, or you may be testing your disaster recovery plan. In a planned switch, the production system (the source of replication) is available. When you perform a planned switch, replication is ended on both the source and target systems. The next time you start replication, it will be set to replicate in the opposite direction.
Unplanned switch - A change the direction of replication as a response to a problem. Most likely the production system is no longer available. When you perform an unplanned switch, you must initiate the switch from the target system. Replication is ended on the target system. The next time you start replication, it will be set to replicate in the opposite direction.
MIMIX Model Switch Framework - A set of programs and commands that provide a consistent framework to be used when performing planned or unplanned switches in environments that do not use application groups. Typically, a model switch framework is customized to your environment through its exit programs.
Best practices for maintaining your MIMIX environment
MIMIX is shipped with default settings that incorporate many best practices for maintaining your environment. Others may require changing policies and adopting best practices within your organization. Best practices include:
• Allow MIMIX to automatically correct differences detected during database and object replication processes that would otherwise result in errors. If MIMIX is unable to perform the recovery, the problem is reported as a replication error (a file is placed in held error or an object is in error).
• Allow MIMIX to automatically perform audits and to automatically recover any differences detected by audits. Best practice is to allow regularly scheduled audits of all objects configured for replication and daily audits of prioritized categories of replicated objects. User interfaces summarize audit results and indicate whether MIMIX is unable to recover an object.
• Perform all audits with the audit level set at level 30 immediately prior to a planned switch to the backup system and before switching back to the production system. • Perform switches on a regular basis. Best practice is to switch every three to six
months. You need to set aside time for performing planned switches.
Environments that continue to use MIMIX Switch Assistant can use policies so that compliance with regular switching is automatically reported in the user interface.
Authority to products and commands
If your MIMIX environment takes advantage of the additional security available in the product and command authority functions which Vision Solutions provides through License Manager, you may need a higher authority level in order to perform MIMIX daily operations.
A MIMIX administrator can change your authorization level to commands and displays. Authorization levels typically fall into these categories:
• Viewing information requires display (*DSP) authority. • Controlling operations requires operator (*OPR) authority.
• Creating or changing configuration requires management (*MGT) authority. For example, consider audits. You can view an audit if you have display authority, perform audits if you have operator authority, and change policies that affect how auditing is performed if you have management authority.
Accessing the MIMIX Main Menu
The MIMIX command accesses the main menu for a MIMIX installation. The MIMIX Main Menu has two assistance levels, basic and intermediate. The command defaults to the basic assistance level, shown in Figure 1, with its options designed to simplify day-to-day interaction with MIMIX. Figure 2 shows the intermediate assistance level. The options on the menu vary with the assistance level. In either assistance level, the available options also depend on the MIMIX products installed in the installation library and their licensing. The products installed and the licensing also affect subsequent menus and displays.
Accessing the menu - If you know the name of the MIMIX installation you want, you can use the name to library-qualify the command, as follows:
Type the command library-name/MIMIX and press Enter. The default name of the installation library is MIMIX.
If you do not know the name of the library, do the following: 1. Type the command LAKEVIEW/WRKPRD and press Enter.
2. Type a 9 (Display product menu) next to the product in the library you want on the Vision Solutions Installed Products display and press Enter.
Changing the assistance level - The F21 key (Assistance level) on the main menu toggles between basic and intermediate levels of the menu. You can also specify the the Assistance Level (ASTLVL) parameter on the MIMIX command.
Figure 1. MIMIX Basic Main Menu
Note: On the MIMIX Basic Main Menu, options 5 (Start or complete switch using Switch Asst.) and 10 (Availability Status) are not recommended for
installations that use application groups.
Figure 2. MIMIX Intermediate Main Menu
Each MIMIX policy is a mechanism used to enable, disable, or provide input to a function such as replication, auditing, or MIMIX Model Switch Framework. A policy may also determine how you are notified about certain problems that may occur. For most policies, the initially shipped values apply to an installation. However, policies can be changed and most can also be overridden for individual data groups. When a policy is set for a data group, it takes precedence over the installation policy. Some policies, such as ones that control when audits are automatically submitted, apply to individual audit rules for specific data groups.
Policies must be changed from the management system. Changing policies requires that you have management-level authority to the Set MIMIX Policy (SETMMXPCY) command.
You can set policies from a command line or from the Work with Audits, the MIMIX Availability Status, and the Work with DG Definitions displays.
The topics in this chapter include:
• “Environment considerations for policies” on page 27 describes additional considerations for setting policies for environments with more than two nodes or bi-directional replication. Also, applications and features can conflict with policy-controlled automatic recovery functions.
• “Setting policies - general” on page 29 provides basic procedures for changing policies. Other topics in this chapter include more in-depth procedures for specific policy-controlled functionality.
• “Policies which affect an installation” on page 31 identifies the policies that are set for an installation and which cannot be overridden by a data group-level setting. Also, this includes procedures for changing retention criteria for procedure history. • “Policies which affect replication” on page 32 identifies the policies associated
with automatic error detection and correction during replication and identifies the common object and file error situations that can be automatically recovered. • “Policies which affect auditing” on page 36 identifies policies that influence audit
runtime behavior and control scheduling for automatically submitted audits. Shipped audits and their descriptions and default scheduling details are included. • “Changing auditing policies” on page 41 provides additional information and
procedures for changing policies associated with auditing. This includes changing the auditing level before switching, changing automatic audit scheduling,
changing audit history retention, restricting auditing based on the state of data groups, and disabling auditing.
• “Policies for switching with model switch framework” on page 48 identify the policies associated with model switch framework and includes instructions for changing these policies.
Environment considerations for policies
Default settings for policies are chosen to address the needs of a broad set of customer environments. However, in more complex environments, you need to consider the effect of policies. Also, applications and other MIMIX features in some environments can conflict with automatic recovery actions during replication and with auditing.
Policies in environments with more than two nodes or bi-directional
repli-cation
Policy values may affect data throughout your entire environment, not just a single installation or data group. This is of particular concern in environments that have more than two systems (nodes) or which have replication occurring simultaneously in more than one direction (bi-directional). Specifically, be aware of the following:
• In these environments, the value *DISABLED for the Objects only on target policy is recommended. When the policy is disabled, audits will detect that objects exist only on the target system but will not attempt to correct them. The commands used by an audit are aware of all objects on the target system, not just those which originate from the source system of the data group associated with the audit. In these environments, the values *DELETE and *SYNC must be used with care. When the policy value is Delete, audits will delete objects which may have originated from systems not associated with the data group being audited. When the policy value is Synchronize, audits will synchronize the objects to the source system of the data group being audited, which may not be the source system from which they originated.
• Synchronization of user profiles and authorization lists associated with an object will occur unless the user profiles and authorization lists are explicitly excluded from the data group configuration. In the environments mentioned, this may result in user profiles and authorization lists being synchronized to other systems in your configuration. This behavior occurs whenever any of the automatic recovery policies are enabled (database, object, audit). To prevent this from occurring, you must explicitly exclude the user profiles and authorization lists from replication for any data group for which you do not want them synchronized.
rules to be run from the management system.
• In environments with three or more systems in the same installation, you need to evaluate each pair of systems. For each pair of systems, evaluate the directions in which replication is permitted. If any pair of systems supports simultaneous bi-directional replication, determine the winning system in each pair and determine the direction to be audited. Set the audit level policy to permit auditing for the data group that replicates in the chosen direction. Disable auditing for the data group which replicates in the other direction. You may also want to consider changing the values of the Run rule on system policy for the installation or the audited data groups to balance processing loads associated with auditing.
• In environments that permit multiple management systems in the same
installation, in addition to evaluating the direction of replication permitted within each pair of systems, you must also consider whether the systems defined by each data group are both management systems. If any pair of systems supports simultaneous bi-directional replication, choose the winning system and change the Audit level policies for each data group so that only one direction is audited. You may need to change the Run rule on system policy to prevent certain data groups from being audited from specific management systems.
When to disable automatic recovery for replication and auditing
At times, you may need to disable automatic recoveries during replication and auditing for certain data groups because a feature in use or an application being replicated may interact with auditing in an undesirable way.Features - Do not use automatic recoveries during auditing and replication in any data group that is using functions provided by the MIMIX CDP™ feature. This feature, which requires an additional license key, permits you to perform operations
associated with maintaining continuous data protection. By configuring a recovery window for a data group, you introduce an automatic delay into when the apply processes complete replication. By setting a recovery point for a data group, you identify a point that, when reached, will cause the apply processes to be suspended. In both cases, source system changes have been transferred to the target system but have not been applied. In such an environment, comparisons will report differences and automatic recoveries will attempt recovery for items that have not completed replication. To prevent this from occurring, disable comparisons and automatic recoveries for any data group which uses the MIMIX CDP feature. For details, see
To exclude a data group from audits, use the instructions in “Preventing audits from running” on page 45.
Disabling audits and recovery when using the MIMIX CDP feature
The functions provided by the MIMIX CDP™ feature1 create an environment in which source system changes have been transferred to the target system but have not been applied. Any data group which uses this feature must disable automatic comparisons and automatic recovery actions for the data group.
Do the following from the management system:
1. From the command line type SETMMXPCY and press F4 (Prompt).
2. For the Data group definition, specify the full three-part name of the data group that uses the MIMIX CDP feature.
3. Press Enter to see all the policies and their current values. 4. For Automatic object recovery, specify *DISABLED.
5. For Automatic database recovery, specify *DISABLED. 6. For Automatic audit recovery, specify *DISABLED. 7. For Audit level, select *DISABLED.
8. To accept the changes, press Enter.
Setting policies - general
Policies must be changed from the management system. Changing policies requires that you have management-level authority to the Set MIMIX Policy (SETMMXPCY) command.
The following procedures describe the basic procedures for setting policies.
Changing policies for an installation
This procedure changes a policy value at the installation level. The installation level value will overridden if a data group level policy has been specified with a value other than *INST.
Do the following from the management system:
1. From the command line type SETMMXPCY and press F4 (Prompt). 2. Verify that the value specified for Data group definition is *INST. 3. Press Enter to see all the policies and their current values.
4. Specify a value for the policy you want. Use F1 (Help) to view descriptions of possible values.
5. To accept the changes, press Enter.
Changing policies for a data group
Do the following from the management system:
1. From the command line type SETMMXPCY and press F4 (Prompt). 2. For the Data group definition, specify the full three-part name. 3. Press Enter to see all the policies and their current values.
4. Specify a value for the policy you want defined for the data group. Use F1 (Help) to view descriptions of possible values.
5. To accept the changes, press Enter.
Resetting a data group-level policy to use the installation level value
Do the following from the management system:1. From the command line type SETMMXPCY and press F4 (Prompt). 2. For the Data group definition, specify the full three-part name. 3. Press Enter to see all the policies and their current values. 4. For the policy you want to reset, specify *INST.
Policies which affect an installation
While many policies can be set for an installation, the policies in Table 1 cannot be overridden for an individual data group. At the data group level, these policies always have a value of *INST.
Changing retention criteria for procedure history
The procedure history retention policy determines how long to retain historical
information about procedure runs that completed, completed with errors, or that failed or were canceled and have been acknowledged.
Environments configured with application groups use procedures to control
operations such as starting, ending, or switching. History information for a procedure includes timestamps indicating when the procedure was run and detailed information about each step within the procedure. The policy specifies how many days to keep history information and the minimum number of runs to keep. You can specify a different number of runs to keep for switch procedure runs than what is kept for other types of procedures.
Each procedure run is evaluated individually against the policy and its history information is retained until the specified minimum days and minimum runs are both met. When a procedure run exceeds these criteria, system manager cleanup jobs will remove the historical information for that procedure run from all systems. The values specified at the time the cleanup jobs run are used for evaluation.
To change the procedure history retention policy for the installation, do the following: 1. From the command line type SETMMXPCY and press F4 (Prompt).
2. Verify that the value *INST is specified for the Data group definition prompt: 3. Press Enter to see all the policies and their current values.
4. Locate the Procedure history retention policy. The current values are displayed. Specify values for the elements you want to change.
5. To accept the changes, press Enter.
Table 1. Policies that can be set only at the installation level and shipped default values.
Policy Shipped Values – Installation
Independent ASP library ratio 5 Procedure history retention
• Minimum days
• Minimum runs per procedure • Min. runs per switch procedure
Policies which affect replication
Table 2 identifies the policies which can affect replication and their shipped default values.
MIMIX can automatically attempt to correct problems it encounters during replication when the policies for Automatic system journal recovery and Automatic user journal recovery are enabled. The following topics identify what errors can be recovered in this way:
• “Errors handled by automatic database recovery” on page 33 • “Errors handled by automatic object recovery” on page 34
Table 2. Policies associated with replication and shipped default values.
Policy Shipped Values Replication Processes Installation Data Groups System Journal User Journal
Data group definition *INST Name1
1. A data group definition value of *INST indicates the policy is installation-wide. A name indicates the policies are in effect only for the specified data group.
Yes Yes Automatic system journal
recovery
*ENABLED *INST1 Yes2
2. When this policy is enabled, the other policies in the same column are in effect unless otherwise noted.
–
Automatic user journal recovery
*ENABLED *INST – Yes2
System journal recovery notify on success
*YES *INST Yes –
User journal recovery notify on success
*YES *INST – Yes
DB apply cache *DISABLED *INST – Yes
Access path maintenance3 • Optimize for DB apply • Maximum number of jobs
3. This policy is available only on systems running service pack 7.1.15.00 or higher. When running on earlier levels, the Parallel AP maintenance provides similar functionality. For more information about both access path maintenance functions, see the MIMIX Administrator Reference book.
*DISABLED 99
*INST *INST
– Yes
Synchronize threshold size 9,999,999 *INST Yes Yes Number of third delay retry
attempts
100 *INST Yes –
Errors handled by automatic database recovery
MIMIX can detect and correct the most common file error situations that occur during database replication. When the Automatic database recovery policy is enabled, database replication processes detect the types of errors listed in Table 3. When an error is detected, MIMIX automatically attempts to correct the error by starting a job to perform an appropriate recovery action.
The recovery action also sends a report of a recovery in progress to the user
interface. The reports are on the Work with Recoveries display (WRKRCY command). When the recovery action completes, the report is removed.
The DB rcy. notify on success policy determines whether a successful recovery generates an informational notification.
Only when all recovery options are exhausted without success is a file placed in hold error (*HLDERR) status. Recovery actions that end in an error do not generate a separate error notification because the error is already reflected in MIMIX status.
Table 3. Errors detected and corrected during database replication when automatic database recovery is enabled.
Error Description
File level errors - and -
Unique-key record level error
Typically invoked when there is a missing library, file, or member. Also invoked when an attempt to write a record to a file results in a unique key violation. Without database autonomics, these conditions result in the file being placed in *HLDERR status.
Record level errors Invoked when the database apply process detects a data-level issue while processing record-level transactions.
Without database autonomics, any configured collision resolution methods may attempt to correct the error. Otherwise, these conditions result in the file being placed in *HLDERR status.
Errors on IFS objects configured for user journal replication
Invoked during the priming of IFS tracking entries when replicated IFS objects are determined to be missing from the target system. Priming of tracking entries occurs when a data group is started after a configuration change or when Deploy Data Grp. Configuration (DPYDGCFG) is invoked.
Errors on data area and data queue objects configured for user journal replication
Invoked during the priming of object tracking entries when replicated data area and data queue objects are determined to be missing from the target system. Priming of tracking entries occurs when a data group is started after a
configuration change or when the Deploy Data Grp. Configuration (DPYDGCFG) is invoked.
Errors when DBAPY cannot open the file or apply transactions to the file
Errors handled by automatic object recovery
MIMIX can detect and correct the most common object error situations that occur during replication. When the Automatic object recovery policy is enabled, object replication processes detect the types of errors listed in Table 4. When an error is detected, MIMIX automatically attempts to correct the error by starting a job to perform an appropriate recovery action.
Unless the object is explicitly excluded from replication for a data group, the
autonomic recovery action will synchronize the object to ensure that it is on the target system.
Note: Object automatic recovery does not detect or correct the following problems: • Missing spooled files on the target system.
• Files and objects that are cooperatively processed. Although the files and objects are not addressed, problems with authorities for cooperatively processed files and objects are addressed.
• Activity entries that are “stuck” in a perpetual pending status (PR, PS, PA, or PB).
The recovery action also sends a report of a recovery in progress to the user interface. In a 5250 emulator, the reports are on the Work with Recoveries display (WRKRCY command). When the recovery action completes, the report is removed. The Obj. rcy. notify on success policy determines whether a successful recovery generates an informational notification.
Only when all recovery options are exhausted without success is an activity entry placed in error status. Recovery actions that end in an error do not generate a separate error notification because the error is already reflected in MIMIX status.
Table 4. Errors detected and recoveries attempted by object autonomics during object replication
Error Description
Missing objects on target system1
An object (library-based, IFS, or DLO) exists on the source system and is within the name space for replication, but MIMIX detects that the object does not exist on the target system. Without object automatic recovery, this results in a failed activity entry.
Notes:
• Missing spooled files are not addressed.
• Missing objects that are configured for cooperative processing are not synchronized. However, any problems with authorities (*AUTL or *USRPRF) for the missing objects are addressed.
Missing parent objects on target system1
Any operation against an object whose parent object is missing on the target system. Without object autonomics, this condition results in a failed activity entry due to the missing parent object.
Missing *USRPRF objects on target system1
Missing *AUTL objects on target system1
Any operation that requires a authority list (*AUTL) that does not exist on the target system.Without object autonomics, this results in authority issues that cause replication errors.
In-use condition
Applications which hold persistent locks on objects can result in object replication errors if the configured values for delay/retry intervals are exceeded. Default values in the data group definition provide approximately 15 minutes during which MIMIX attempts to access the object for replication. If the object cannot be accessed during this time, the result is activity entries with errors of Failed Retrieve (for locked objects on the source system) and Failed Apply (for locked objects on the target system) and a reason code of *INUSE.
Notes:
1. The Number of third delay/retries policy and the Third retry interval policy determine whether automatic recovery is attempted for this error.
2. Automatic recovery for this error is not attempted when the objects are configured for cooperative processing.
1. The synchronize command used to automatically recover this problem during replication will correct this error any time the command is used.
Table 4. Errors detected and recoveries attempted by object autonomics during object replication
