M4: With a replication schema that meet the requirements of a static specification of the database usage, the virtually fully replicated database does not make data objects available for unspecified data accesses. This is a disadvantage, compared to a fully repli-cated database. To address this problem for virtual full replication, we introduce adaptive segmentation. With adaptive segmentation, we will allow unspecified data accesses for soft transactions. Data objects that are not allocated to nodes are loaded on demand and the database is re-segmented based on the new information that becomes available at execution time. The approach in this part of our method is still under development, and below we present our current findings and the related challenges.
The segmentation information at each node keep tracks of which segments are allocated to the node. This includes the data objects in a segment, the consistency model used by the segment, the required deadlines for replication, and a list of all the nodes that the segment
is allocated to. When an access arrives that use a data object that is not available at the node, processing is initiated for for loading this data object from another node.
The representation for where data can be found is available by listing all data objects in the database and their residence and segment assignment and store at each node. This is a simple approach that does not scale on the size of the database. This representation need to be elaborated in the work to come. One approach is to store information only of the data objects available at the node. Missing data can still be detected, but finding nodes where data can be loaded from then need a (bounded) search approach.
For adaptiveness, we will primarily consider the following changes to the replication schema: Unspecified accesses, added and removed nodes, added and removed data objects.
Also, the level of fault tolerance is considered. We can apply a minimum degree of repli-cation for a segment, and when a node fail, this degree needs to be restored again. It is a similar problem to making data available for un-specified accesses, and we expect to use the same approach to restore minimum degree of replication.
The segmentation algorithm we have presented [53] is expected to be efficient enough for adapting segmentation online. Segment tables (based on the updated property sets) can be updated when new information arrives, in O(o log o) time.
We propose to use incremental recovery [45] for loading segments that contain the data objects that are detected as missing at a node. On-demand loading of segments will delay transactions while loading a missing segment onto a node, and the timeliness will depend on the timeliness of the network, which makes segmentation adaptation unsuitable for hard transactions.
To maintain scalability and to reduce storage usage over time, segments also need to be unloaded from nodes when not used anymore. The load and unload policies is application dependent. Some applications may not need to unload segments, but will reach an equilibrium where all segments ever needed is available, and are continuously in use. Other applications, with mode changing characteristics, may need to switch data sets when changing the mode of execution. The combination of the frequency of changes and the available storage may cause ’thrashing’ that keeps the database system busy with loading and unloading data, without leaving processing time for database transactions.
This is an open problem that need to be studied in this part of our work.
Reconfiguration of segments immediately take effect at the local node, and as soon as data has been loaded to the local node, the pending transaction can continue its execution.
Subsequent transactions to the loaded segment will not need to wait, but will execute timely at the node. Reconfiguration of segments is assumed to always be initiated at the node of an access, then propagated to all other nodes that needs the information about
the change. In the DeeDS database, such propagation can be done by using the database replication mechanism, detached from the update of the local segment knowledge. Since such propagation has eventual consistency, the other nodes does not get the information at the same time. Therefore user updates of data objects, in a segment that is being loaded to a node, needs to be forwarded from the node where the data is loaded from as long as not all nodes has received the propagated change of segmentation. The problem of propagating and forwarding of updates to nodes that loaded to new nodes, is closely connected to the issue of representation of information about segments at nodes, and will be addressed together.
The conditions for maintaining scalability is an important issue in this part of the work. Applications that require most the data at all nodes will closely resemble a fully replicated database, and be less scalable than a database where the database client have requirements of a low degree of replication. We need to find out more details for which types applications virtual full replication has benefits, and for which applications it means an overhead only.
4.4.1 Adaptive segmentation with pre-fetch
M5: With adaptive recovery of segments to nodes, virtual full replication can make data available at request by soft transactions. For hard transactions, data must be available at the time the transaction executes at the node. We extend adaptive segmentation with pre-fetch of data to nodes, to make data available at a certain time point or repeating time points separated by an interval. This enables adaptation for hard transactions as well.
One condition for this is that the time instant of an access, or the frequency of repeating accesses, can be specified. Another condition that is that segments must be recovered in bounded time, which imply that the system uses a real-time network.
Since many real-time systems run transactions periodically or at least with some re-peating patterns, it may additionally be possible to detect the patterns for the accesses.
Having simple mechanisms for detection of periodic accesses, we may be able to detect when data needs to be available. This can be extended to include advanced pattern detec-tion and also pattern predicdetec-tion. It seems unlikely that detecdetec-tion and predicdetec-tion can be correct all the time, and for this reason it may only be used to improve for soft real-time transactions. Pre-fetch of data by advanced pattern detection and prediction is an optional step in our methodology.
To maintain scalability, data objects that are not accessed for a long time may need to be removed from nodes. This need to be put in relation to the set up time for data objects. Adding and removing data objects from nodes has some resemblance with both
virtual memory, cache coherency and garbage collection and our approach will relate to these concepts. Removal of objects is critical for hard transactions and must be done in a way that guarantees availability at the time of access by the hard transaction.
Replicas of data objects that have been assigned to nodes, either by a pre-specified static segmentation or by pre-fetching of data for hard transactions, cannot be removed if there is a risk that removal jeopardizes timeliness of a hard transaction. Pinning of such data objects is one approach to prevent those data objects from being removed from memory by a generic removal policy based on known accesses. Removal may also mean to move data from memory to disk at the same nodes. Pinning of object require an explicit specification, in a similar manner as static segmentation or as an explicit specification of time instants for accesses by hard transactions.