RTL uses full incremental compilation to control memory consumption and you can control the net-change database size to reduce memory consumption.
SQT Memory Consumption Control for RTL
Control the maximum memory consumed by unpacked commands in the DSI SQT cache during transaction profiling in RTL .
The SQT thread monitors the memory consumed by commands unpacked by the RTL transaction profiling processes and referenced by the DSI SQT cache.
When Replication Server is replicating using RTL, the maximum amount of memory consumed by the DSI thead is the sum of dsi_sqt_max_cache_size, sqt_max_prs_size , and
dsi_cdb_max_size. Setting dsi_sqt_max_cache_size, sqt_max_prs_size , and
dsi_cdb_max_size smaller would reduce memory consumption but reduce replication performance. Tune your replication environment for optimum memory consumption and performance. See the Replication Server Reference Manual > Replication Server Commands to configure the parameters.
Net-Change Database Size
Reduce memory consumption by the net-change database by triggering the net-change database to flush data to the replicate database once the net-change database size reaches a threshold size.
Memory consumption refers to Replication Server data structures such as the net-change database, and the data that the structures store. Net-change databases are in-memory data structures. Net-change database memory consumption can increase drastically when Replication Server compiles commands applied on a table with a large number of columns, or tables with large text and image datatype values. For example, compiling 1,000,000 rows
in a table with 100 columns may consume approximately 10 times more memory than compiling the same number of rows in a table with 10 columns. Replication performance suffers when there is insufficient memory available for other processes and modules. Replication Server uses the values you set for dsi_cdb_max_size and
dsi_compile_max_cmds to control memory consumption. You can use dsi_cdb_max_size to control the maximum net-change database size that Replication Server can generate. Once the size reaches the threshold you set, Replication Server stops compiling new commands and transactions into the compiled transaction that Replication Server is building in the net-change database, performs the bulk apply of the compiled group to the replicate database, clears the net-change database, and releases the memory consumed by the net-change database. The number of of net-change database instances that Replication Server generates depends on the values you set with dsi_cdb_max_size and memory_limit. The estimated memory requirements for a replication system using RTL is the number of replicate connections multiplied by dsi_cdb_max_size.
Full Incremental Compilation
Full incremental compilation improves replication performance for real-time loading (RTL) by reducing memory consumption during the processing of large compilable transactions that contain many commands.
Full incremental compilation can compile large transactions containing mixed insert, delete, or update operations. Replication Server uses full incremental compilation to apply a large compilable transaction to the replicate database, using multiple in-memory net-change database instances. Full incremental compilation divides a large transaction into a sequence of segments. Each segment consists of a group of commands.
Replication Server compiles each segment and creates a dedicated net-change database in which to to store one segment. Replication Server instructs the net-change database instance to send and apply the segment to the replicate database. Replication Server then closes the net-change database instance and releases the memory consumed. Replication Server creates another net-change database instance for the next transaction segment and continues to create and close net-change database instances in sequence for all the segments.
Therefore, instead of consuming a single large portion of memory for a large net-change database instance to hold a large transaction, full incremental compilation reduces the memory requirement to the memory consumed by a single smaller net-change database instance containing just a segment of the transaction. Full incremental compilation divides the memory requirement by the number of net-change database instances used. For example, when full incremental compilation applies a large transaction with 10 net-change database instances, the memory requirement is approximately one-tenth of the requirement without full incremental compilation.
Memory Control Parameters and Replication Server Processing
Setting dsi_cdb_max_size to Different Values
Examples that show Replication Server applying a transaction with 100,000 updates on two tables. Table1 has 100 columns and requires approximately 4GB of memory, and Table2 has 10 columns requiring approximately one-tenth the memory—400MB.
dsi_cdb_max_size
Value (MB)
Table Name
Impact on Replication Processing
1024 (default) Table1 Prerequisite: Set memory_limit in Replication Server to a value large enough to allow the construction of 1GB net-change data- bases.
Replication Server uses 4 1GB net-change database instances to apply the transaction.
1024 (default) Table2 Prerequisite: Set memory_limit in Replication Server to a value large enough to allow the construction of 400MB net-change databases.
Replication Server uses 1 400MB net-change database instance to apply the transaction.
4096 Table1 Prerequisite: Set memory_limit in Replication Server to a value large enough to allow the construction of 4GB net-change data- bases.
Replication Server uses 1 4GB net-change database instance to apply the transaction.
4096 Table2 Prerequisite: Set memory_limit in Replication Server to a value large enough to allow the construction of 400MB net-change databases.
Replication Server uses 1 400MB net-change database instance to apply the transaction.
Incremental Parsing for RTL
To further reduce memory consumption, enable incremental parsing by the DSI Scheduler thread by setting dsi_incremental_parsing on.