Abstract
Large, spatial databases may be required to process both intense query and intense update loads. For example, a data warehouse which contains spatial data may be required to answer difficult, analytical queries at the same time as it accepts massive amounts of new data, and any downtime to merge new data into existing data organizations may be unacceptable. Traditional, incremental spatial access methods (like the popular R-tree and its variants) may be unacceptable for use in such an environment since they support relatively slow update rates, and may lead to a relatively poor global data organization resulting in slow query evaluation. In this paper, we present the details T2SM, which is an instantiation of the linear file template, for use with spatial data. In T2SM, data are organized as a set of ongoing, external memory sorts based on the STR algorithm. T2SM exhibits query performance similar to (and sometimes superior to) a bulk-loaded R-Tree, but at the same time is truly incremental and can maintain exceptional update rates.
1 Introduction
An important trend in hard disk technology over the last several decades has been the continued exponential decrease of sequential transfer costs and disk capacity with respect to seek times. Accord-ing to one study [1], disk transfer rate and capacity have increased historically at around 25% per year, while seek times decrease at a rate of only around 10%. Thus, with each passing year, fewer seeks are available per byte of disk storage, and sequential I/O becomes more attractive.
This trend has many important implications for data indexing and storage organizations. One implication is that it becomes more and more feasible (and desirable) to maintain on-disk organiza-tions as 100% packed structures. If each insertion into an index requires one or more seeks (definitely the case in classical file structures such as the B-Tree or R-Tree [3]) it may be preferable to batch insertions and periodically rebuild the entire structure using massive, sequential I/O. This bulk-loading has the added benefit that packed structures can maintain a careful, global data organiza-tion that reduces seeks during query evaluaorganiza-tion, by allowing a careful, on-disk organization of data where related disk blocks are placed sequentially on disk.
However, the major drawback of this approach is that is essen-tially a variation on the simplest sorting algorithm: periodically, scan the structure and insert a new set of data objects into it. Such a
method is O(n2) in time complexity. Thus, it may be impractical for a growing database that is gigabytes or terabytes in size.
In response to this, we have previously developed a new para-digm for storage organization called the linear file [4]. In a linear file, a database is organized as a large set of fully indexed, ongoing external sorts. Each ongoing sort is referred to as a subindex. Essentially, each subindex is a small fraction of the database that is bulk-loaded individually as needed, so as to not require down-time for massive index rebuilds. Data within a subindex are fully packed, and maintained in a careful, sorted order to reduce seeks during query evaluation. The linear file data organization is a generic template, and may be used with many data types. In this paper, we detail how the linear file can be instantiated to produce T2SM, a storage manager for use with spatial data.
The paper is organized as follows. In Section 2, we briefly describe the linear file template. In Section 3, we describe how the linear file can be instantiated to produce T2SM. In Section 4 we give an overview of some experiments that suggest that T2SM can produce an organization that supports processing of range queries as well as a bulk-loaded R-Tree (though T2SM is a truly incremen-tal structure). We conclude the paper in Section 5.
2 The Linear File Template
We now briefly describe the linear file template. A much more complete description can be found elsewhere [4].
Data in a linear file are organized into subindexes, which are make of two or more levels. Levels are different than levels in a classical, hierarchical file in that they are fully packed and contain no pointers, only data. Each subindex contains a lowest level, L0, that is by definition stored in main memory. Each subsequent level is multiplicatively larger than the previous level by some factor F (in practice the value for F is on the order of 50).
When a data object which is inserted into the linear file is added into a subindex, it is first buffered in L0for the appropriate subin-dex. For example, consider Figure 1 (for which F = 3). We first insert data items 11 and 12, which are buffered in main memory in L0. Next, we insert item 1, at which point L0overflows (that is, we run out of buffer memory for this subindex). Then, in a single sequential write, the items 1, 11, and 12 are written to a new level, L1, which is located on disk (b). Then next time that L0overflows, L1 is re-read in a single, sequential scan from disk, merged and packed with the items from L0, and then written out to disk in a single, sequential write, leaving (d). L1is allowed to grow F times larger than L0, but in the next overfill of L0, it reaches its capacity. L1is read sequentially from disk, merged and packed with the new items from L0, and then written to disk as a new level, L2(shown in (f)). More items are added and eventually, L1overflows once again. Then, L1 and L2 are read sequentially from disk, merged and packed with L0, and written out in a single, sequential write to L2 (shown in (k)). The process can easily be generalized to accommo-date an arbitrary number of levels.
Subindexes are queried as well as inserted into, and so when placed on disk they are prepended with a directory to facilitate fast lookup during query evaluation. This directory contains (region, Permission to make digital or hard copies of all or part of this work
for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Copyright 2001 ACM XXXXXXXXX/XX/XX ...$5.00.
Maintaining a Large Spatial Database with T2SM
Christopher Jermaine, Edward Omiecinksi, Wai Gen Yee
Georgia Institute of Technology, College of Computing
ptr) pairs for each of the disk pages in the structure, as shown in Figure 2 (for spatial data, the regions will be MBRs, or Minimum Bounding Rectangles). Note that the subindex levels are placed contiguously on disk. This ensures that queries over data within a single subindex require only short, forward seeks to evaluate.
In memory for each subindex, we need to store L0for the subin-dex, as well as pointers to each of the directory pages that precede the subindex, and pointers to the end of each subindex level. A rep-resentation of a linear file containing three subindexes is given above in Figure 3. In practice, there may be many thousands of subindexes (or more) in a linear file.
We have implemented the linear file as a formal template in C++. In order to instantiate the linear file for a specific type of data (like spatial data) one needs to implement the following, core oper-ations for the specific type of data (among others):
•Pack (array of data objects). This operation is used by the lin-ear file to perform the merge and pack operation during inser-tion into a subindex. This operainser-tion must do two things. First, it must arrange the array of data objects into a set of disk-page-sized segments (the merge). Next, it must order those pages, and return them to the calling function (the pack).
•Partition (array of data objects). This operation is used to split subindexes when they become too large. An additional degree of freedom in a linear file is that the partition need not be a binary partition, as in a B-tree or R-tree. For spatial data, it might make sense to partition the data many ways in order to mimic the underlying spatial distribution of the data.
•ChooseSubindexForInsertion (data object). This operation chooses a subindex which will receive the new data object. Since there is no minimum size for a subindex, it is possible to return NULL, in which case a brand new subindex is created for the data object.
•IsConsistent (query region, subindex). This operation is used by the linear file to determine if a subindex could possibly con-tain any records corresponding to the given query region.
11 12 1 11 12 1 11 12 4 5 1 4 5 10 11 12 2 3 1 4 5 10 11 12 4 5 8 10 11 1 2 3 12 15 18 1 2 3 4 5 8 10 11 12 13 15 18 1 2 3 4 5 8 10 11 12 14 17 13 15 18 1 2 3 4 5 8 10 11 12 9 13 14 15 17 18 1 2 3 4 5 8 10 11 12 4 5 6 7 8 1 2 3 9 10 11 12 13 14 15 16 17 18 (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k)
Figure 1: Subindex insertion.
L1 L2
L0
L1 L0
L2
Figure 2: Subindex stored on disk. The header is followed by L1, which is followed by L2.
used portion unused portion
directory low address high address MBR ptr MBR ptr MBR ptr MBR ptr L1 L2 (MBR, ptr)(L1ptr, L2ptr, endptr) (MBR, ptr)(MBR, ptr)(MBR, ptr)(L1ptr, L2ptr, endptr) (MBR, ptr)(MBR, ptr)(L1ptr, L2ptr, endptr) directory directory directory L L L L1 2 L1 2 L1 2 subindex 2: subindex 1: subindex 3: memory main hard disk
Figure 3: Overall linear file organization.
low address high address hard disk
Figure 4: The steps for packing a region of a subindex. First, the data are packed into pages, shown in (b). Then these pages are ordered using a Hilbert curve (c) and written to disk (d).
(a) (b)
(c)
3 T2SM
In this section, we detail how the linear file is instantiated to pro-duce a high-performance spatial indexing structure called T2SM. For each of the four linear file functions listed above that must be instantiated, we briefly detail how this is done in T2SM.
3.1 Pack()
The implementation of this routine is very straightforward. The process is shown above in Figure 4. First, the data are organized into pages, using a version of the STR algorithm [5]. Next, the pages are sorted according to a Hilbert curve. Finally, the pages are returned to the linear file template and written to disk by the tem-plate in the order specified by T2SM. Due to the clustering pro-vided by the Hilbert curve, queries can often be evaluated often without seeks, by long, sequential scans over multiple pages.
3.2 Partition()
In a linear file (and hence in T2SM), an overfull subindex is dealt with in the obvious way: by splitting the subindex into smaller sub-indexes, much as is done in an R-tree when a node in the tree becomes overfull. However, there are two key differences between splitting a node in an R-tree and splitting apart a subindex:
• First, a subindex is much larger in size than a tree node. The largest acceptable size for a subindex is somewhere on the order of tens of megabytes, which could possibly encompass millions of data objects. This contrasts to the case of a tree node, which will have on the order of hundreds of objects. Thus, a key consideration is the asymptotic running time of the splitting algorithm. A quadratic splitting algorithm (as is typi-cally used in an R-tree) will not be acceptable in this situation. • Second, we are very concerned that the structure of the sys-tem’s subindexes mimic the clustering structure of the underly-ing data. The splittunderly-ing of subindexes provides the best opportunity for ensuring that this is the case. The problem with applying an R-tree split algorithm in this case is that R-tree splitting is phrased as a bipartition problem, whereas within a subindex we have no idea a priori of the number of clusters in the underlying data. Also, due to the insertion algorithm of the previous section, subindexes may effectively cover irregular areas. T2SM subindex splitting is then a multi-partition prob-lem that is very close to a clustering probprob-lem from machine learning or data mining. Given a subindex, it must be parti-tioned into many smaller, non-overlapping subindexes. Given this, we will now discus the multi-partition problem as it applies to subindex splitting in T2SM. First, we define few terms. Assuming n is the dimensionality of the data:
Definition 1: A rectangle is an ordered triple (low, high, D) where low and high are vectors of real numbers of length n and D is the data objects from the subindex falling in that rectangle. Definition 2: The aspect ratio AR of a rectangle r is defined as
follows. Let S = . Then
AR (r) = max (S) / min (S).
Definition 3: Given a rectangle r, we define the following
areas. First, .
Next, .
Definition 4: The union operation on rectangles is defined as follows. If t = , then t.low[i] = min (r.low[i], s.low[i])
t.high[i] = max (r.high[i], s.high[i]) t.D = .
Definition 5: Two overlapping rectangles r and s have
inciden-tal overlap if there is at least one object in r or s that spans the
overlap between them in at least one dimension. An example of overlap that is incidental is given in Figure 5.
Definition 6: The multi-partition problem is defined as fol-lows. Given a set of parameters ( , mc, mn), and a dataset D, a partition of D into rectangles is defined as acceptable if the number of rectangles is no greater than mn, and if for each con-tained rectangle r, AR (r) < and count (r.D) < mc, and if any overlap exists among the rectangles in the partition, then it is only incidental. Then, the multi-partition problem searches for the acceptable partition P of D into rectangles such that
is minimized over all such acceptable partitions. Computationally, the multi-partition problem as it has been defined is a rather difficult one. It differs from standard formulations of the clustering problem (such as k-means clustering [2]) in that not only are we worried about the partitioning of data objects into clusters, but we are also concerned with the relationship of the structure of each cluster with the other clusters. Namely, we require only incidental overlap among the clusters, which compli-cates matters considerably. For example, if we attempt a straight-forward modification of a classical clustering algorithm like a single-linkage or multiple-linkage algorithm [2] to solve this prob-lem directly, the computational complexity is increased by a factor of the size of the subindex, from O (s2) to O (s3), where s is the number of data objects in the subindex. This is because when com-bining two clusters, we cannot just consider the two clusters; we must also see whether their combination produces a cluster that overlaps any other clusters, and requires another scan through the current set of clusters. This is problematic, since even an O (s2) algorithm is too costly for use with a large subindex.
With this in mind, the algorithm we use for partitioning the data in a subindex into several subindexes in accord with the multi-par-tition problem that we have developed is a rather novel one. While specific experimental or analytic evaluation is beyond the scope of this paper, we have found this heuristic algorithm to be very suc-cessful in partitioning low-dimensional data to closely match the clustering structure of the underlying data. It is comprised of five steps, which we describe now. The steps are as follows:
1) First, the objects are “gridified”. In this step, the objects in the subindex (of which there are potentially millions) are forced into a regular grid comprised of a target number of rectangles t, so that each rectangle in the grid contains one or more objects from the subindex. t is a number which is typically several orders of magnitude smaller in size than the number of objects in the subin-r.high i[ ]–r.low i[ ] (i∈{1…n})
{ }
r 1 = max r.high i{ [ ]–r.low i[ ],i∈{1…n}} r 2 (r.high i[ ]–r.low i[ ]) i=
∏
1...n = r∪s ∧ i∈{1…n} ( ) ∀ ∧ r.D∪s.D Γ Γ r 2 r∑
∈PFigure 5: The effect of different grid resolutions on the number of gridified rectangles for the first step of the solution to the multi-partition problem.
dex; a typical value for t might be 10,000, and we might have 100 to 1,000 breaks along each dimension for two-dimensional data.
The fundamental issue when gridifying the objects in the subin-dex is determining the resolution for the grid in order that the resulting number of grid rectangles is close to t. The effect of dif-ferent choices of resolution on the number of gridified rectangles is shown above in Figure 5. We use a basic, binary search algorithm to determine the correct resolution. First, a very fine resolution is picked for the gridification; a good choice for such an initial reso-lution is one that might produce 10t grid rectangles. The objects in the subindex are partitioned according to this resolution, and the resulting number of grid rectangles is determined. Assuming that this number is much larger than t, a very coarse resolution is cho-sen (one that partitions the dataspace into t rectangles), and again the subindex objects are partitioned. Assuming this time that the number of resulting grid rectangles is less than t, a middle resolu-tion is chosen, and the process is repeated in a binary search until the resolution produces a resulting number of grid rectangles that is very close to the target t. The complexity of this step is O (size (S) log (t)) where S is the subindex to be split. The result of the step, applied, on a real dataset is shown above in Figure 6 (a).
We stress that the point of this step is data reduction and it is only a precursor to the clustering, since nearly any algorithm is too costly to use with the millions of items that may be in a subindex. We also point out that the potential for serious error in the eventual multi-partition due to the gridification is small. While the breaks chosen by this step may not be ideal, the algorithm still chooses 100 to 1,000 or more potential cuts along each dimension, assum-ing two-dimensional data. With c cuts along a dimension, we stress that no rectangle in the optimal multi-partition can have a bound-ary more than 1 / c of the width of that subindex from one of the
cuts chosen. This is a small distance, with generally only a corre-spondingly small introduction of error.
2) In the next step, the rectangles in the grid are grouped. First, a rectangle r from the grid is picked at random, where the number of items in r.D is less than mc. Next, all rectangles riare sorted in ascending order according to the value . This produces a list of rectangles, where r1 denotes r (the first rectangle in this sorted list) and ridenotes the ith rectangle in this sorted list. Let Ri
denote . We choose Ri to maximize
where . Let Rnew =
. Finally, we remove any rectangles in the
grid covered by Rnew and add Rnew to the grid.
This grouping is repeated (choosing a new rectangle for r each time) until all rectangles in the grid have either been grouped, or have been picked as r and considered for grouping. The result of the grouping step is shown above in Figure 6 (b).
3) The next step is the overlap expulsion step. Each rectangle r in the grid is chosen in succession. If r is overlapped by any other rectangles in the grid (and the observed overlap is not only inci-dental), then portions of r are repeatedly sliced off until no rectan-gles overlap r. Each rectangle in turn is chosen and the overlaps are expelled. The result of the step is shown in Figure 6 (c).
4) Next, any rectangle r having count (r.D) > mcis split along its longest dimension. This is repeated until no rectangles contain too many objects. In this way, rectangles which would produce subindexes that are too large are broken up. The result of this step is shown in Figure 6 (d).
5) Finally, any rectangle r having AR (r) > is split, again along its longest dimension, just as in step 4 above. The result of this step is shown above in Figure 6 (e).
After these steps have been performed once, the steps 2-5 are repeated iteratively and in order until a stable state is reached. The final result over the sample dataset is shown above in Figure 6 (f).
3.3 ChooseSubindexForInsertion()
When we add new data, a subindex must be chosen to accept each new data item added to the file. The primary concern during this selection process is reducing overlap among subindexes within the partition, since it is well understood that overlap is one of the pri-mary causes of degradation of performance in multidimensional indexing. To deal with this, we use an algorithm that is part of the following theorem:
Theorem 1. Regardless of dimensionality and assuming point
data, there exists a contiguous, right angular, minimum bounding shape (MBS) for all data within each subindex (as shown in Figure 7), such that each MBS is overlap-free with respect to the MBRs
Figure 6: The steps in our solution to the multi-partition prob-lem, which is used to split apart a subindex in a multiway split.
(b) (c) (e) (f) (d) (a) × r∪ri 1 r1∪r2∪…∪ri count R( i.D) Ri 1 --- count R( i.D)≤mc r r∈grid∧
∪
r covered by RiFigure 7: A partitioning of a 2-D plane into non-overlapping, right-angled, minimum-bounding regions of the type guaran-teed by the insertion algorithm of T2SM.
for all other subindexes in the file, if the following algorithm is used for insertion:
Algorithm. We maintain something resembling a partial ordering
of subindexes (though it is not a true partial ordering). This data structure is a simple lookup table of assertions of the form “Subin-dex 132 BEFORE Subin“Subin-dex 34 EXCEPT MBR”. To find a subin-dex that can accommodate d, subinsubin-dexes are considered in the following order:
1) All subindexes whose MBR covers d (if any) are considered first. From this set we choose one subindex to accept d subject to the following criteria. If subindex A is chosen, then for every other subindex B whose MBR also covers d, either of the following must hold:
A is before B in the ordering and d does not fall in the MBR in the EXCEPT clause of the relating A and B assertion, or:
B is before A and d does fall in the MBR in the EXCEPT por-tion of the asserpor-tion.
2) Otherwise, if no subindexes cover d, determine if there are any subindexes that could swallow the object d without increasing the amount of overlap volume among all subindex MBRs. If so, we choose one and it accepts d. If there is only one such subindex, d is inserted into it. By convention, if there are more than one, then d is inserted into the subindex in that group such that the resulting MBR has an aspect ratio which is closest to 1.
3) If there are no such subindexes, then we choose the subindex A that can accommodate d such that the total additional overlap incurred is minimized (though which subindex we choose is unim-portant in the proof of the theorem). Then, for each subindex B whose MBR now overlaps A’s new MBR:
(a) if B is before A in the ordering, we do nothing additional (b) if A is before B in the ordering, or their order is undefined (that is, there is no entry concerning them), we add or amend the entry in the ordering to be: “Subindex A BEFORE Subindex B EXCEPT ((Subindex A’s MBR after insertion) intersect (Subindex B’s MBR))”.
Proof Outline. The proof relies on the ordering of subindexes. If A
is before B in the ordering, then the MBRs of A and B overlap. Any objects falling in this overlap region belong to A, except for any area that B got to first, and A simply overran (marked in the EXCEPT clause). B is considered the owner of such an “overrun” area, which is always a simple rectangle. Since one and only one subindex ever owns a given spatial region, the theorem holds. A logical question is: Why is this theorem important? Why add this additional complexity? The answer is that while this theorem does not guarantee that the MBRs of subindexes do not overlap, it guarantees something almost as useful: we know that the regions that subindexes hold data in do not overlap. Since we can quickly and easily compute the region that is actually owned by the subin-dex using the set of assertions, we effectively eliminate overlap among subindexes. We point out that since subindexes are packed using a packing algorithm, there is likely to be little overlap within a subindex, and hence little overlap at all.
Still, in addition to overlap another important consideration in such data organization ia the shapes of the data regions. The strangely-shaped regions that can be produced by this algorithm will tend to increase query time, as strangely-shaped regions increase the probability that a query region will require that many subindexes be searched. Thus, an additional constraint that we place during insertion is that we do not allow overlap volume among MBRs produced in step (3) above to exceed , where c is some constant (we have used the value 0.9 for c with good success), n is the dimensionality of the data, and m is the
median size (in terms of volume) of all subindex MBRs. Intu-itively, we choose the above expression to bound the size of MBR overlap by a percentage of the size of an average subindex. The exponent comes into play because it can be shown (but is beyond the scope of this paper) that with increasing dimensionality, the amount of overlap incurred when we attempt to partition a square-shaped spatial region into a few smaller, square-square-shaped spatial regions increases exponentially. If there are no subindexes meeting this criterion, a new subindex is created and d is inserted into it as the only data object.
3.4 IsConsistent()
The implementation of the IsConsistent routine for T2SM is very straightforward. Assuming point data and a point query, we first determine all subindexes whose MBR intersects. Then from that set, we determine the single subindex S where there is no entry in the ordering of subindexes of the form “S BEFORE A EXCEPT MBR” where MBR encompasses P. The fact that there is exactly one such subindex (for point data) follows from Theorem 1. This subindex is then returned.
The extension of this algorithm for use with for non-point data, and non-point queries, is straightforward. We also point out that since subindexes most likely number in the thousands (at 10 MB each, 10 GB of data is held in only 1,000 subindexes), the CPU computation required above is not too great. If it is still too expen-sive, one could imagine using a small, in-memory R-tree to index the MBRs and assertions and greatly speed this part of the query computation.
4 Evaluation of T2SM
Due to space constraints, we are able only to give a brief synopsis of our benchmarking for T2SM in this paper. For a much deeper set of experiments over T2SM, we point the reader to our descrip-tion of the linear file [4]. Our experimental methodology is given there as well.
We implemented the prototype for T2SM, with full concur-rency control in C++, under the Linux Mandrake distribution OS. All experiments in this section were carried out on a 566 MHz Pentium III computer, having 128 MB of RAM.
In this section, we deal with two different datasets:
•The Tiger dataset. We use the Tiger/Line GIS data for 5 states [6]: Texas, Louisiana, Kansas, Oklahoma, and Arkansas. While this dataset has been widely used in research on multidi-mensional indexing, we use a much larger sample of the data than has been customary. For these 5 states, there are 23,547,192 tuples in total. In this case, each tuple will be an entry in a secondary index, and so will contain a search key (a 32 B minimum bounding rectangle) and a 4B record ID (RID). The total amount of data in this dataset is 847.7 MB.
•The Random Walk dataset. This is another, synthetically gen-erated dataset. 100 particles are scattered throughout a field that is 10,000 units wide. Each particle has a certain amount of energy (distributed according to a Poisson distribution, with a mean of 5). The particles are tracked for 100,000 clock ticks, where at each clock tick the particle moves in a random direc-tion. The distance travelled per tick is governed by a Poisson distribution, with a mean distance equivalent to that particle’s energy. There are thus 10 million tuples in total. Each tuple has 120B of information about a particle’s position and motion at a given instant. Tuples are organized by position. The total size of the dataset is 1.20 GB. Clearly, the spatial distribution of data is very skewed.
1–cn
The first step in our experiments was to load each of the datasets into a T2SM disk partition. In all cases, disk-based buffer pages were sized to be 4096B, and data pages were 8192B. Maximum subindex size was set to be 30MB, and 2MB of core memory were allotted for use as main memory buffer pages. mc, the maximum subindex size after a split, was set to 30MB / 4, so the most com-mon split was 4-way. Data were loaded into the T2SM partition at the maximum rate possible.
The results of this load are shown below in Table 1. We point out the considerable speed with which insertions were accom-plished; much faster than in a comparable hierarchical structure. Even if only two seeks were required per insertion, at 10 ms per seek, we would expect only 50 insertions per second using such a structure; T2SM supported sustained insertion rates more than an order of magnitude greater.
We also tested query processing using T2SM. For comparison, we built STR packed R-Trees for both of the datasets. For further comparison, we also ran these queries over an obvious marriage of the STR algorithm and some of the ideas from T2SM. We not only packed the nodes of the tree using the STR algorithm, we orga-nized them globally on disk by sorting the nodes based on a Hil-bert curve. When evaluating queries using this modified STR-packed R-Tree, we added the further enhancement of traversing the tree in a breadth-first manner. We found the addresses of all nodes that needed to be read in at a given level in the tree, then
sorted the addresses and read the nodes in order of increasing disk addresses. In this way, all seeks were short, forward seeks and not longer, more expensive, random seeks all over the disk. In all, we tested several thousand range queries of differing sizes. The results are given above in Figure 8. T2SM uniformly outperformed the
packed R-Tree, and even outperformed the R-Tree with additional Hilbert organization. This is due to the skewed nature of the datasets. The clustering algorithms used in the T2SM Partition() algorithm, combined with the STR packing, yield an organization that is actually better than an STR packing alone (the quality of the STR packing declines somewhat with increasing data skew). We have conducted a much larger set of experiments with T2SM (included concurrent queries and loading), and the results pre-sented here are indicative of the performance of the structure.
5 Conclusions
In this paper, we have given the details of the T2SM instantiation of the linear file template, for use with spatial data. Since data in T2SM are organized into subindexes (indexed and ongoing sorts of the spatial data) insertion is accomplished very, very quickly. Since subindex levels are completely packed and carefully organized, performance in T2SM is comparable to that of an STR packed R-Tree. Still, the T2SM organization is truly incremental. For spatial data management where down-time for packing is unacceptable (due to database size or query load) and insertion loads are great, the incremental algorithms described here may be preferable to repacking a monolithic structure.
References
[1] Peter M. Chen, Edward L. Lee, Garth A. Gibson, Randy H. Katz, David A. Patterson: RAID: High-Performance, Reliable Secondary Storage. ACM Computing Surveys 26(2): 145-185 (1994)
[2] Brian Everitt. Cluster Analysis. New York: Halsted Press, 1993
[3] Antonin Guttman: R-Trees: A Dynamic Index Structure for Spatial Searching. SIGMOD Conference 1984: 47-57
[4] Chris Jermaine, Edward Omiecinski, Wai-Gen Yee: Out From Under the Trees. Georgia Institute of Technology College of Computing Technical Report GIT-CC-01-24.
[5] Scott T. Leutenegger, J. M. Edgington, Mario A. Lopez: STR: A Simple and Efficient Algorithm for R-Tree Packing. ICDE 1997: 497-506
[6] United Stated Bureau of the Census. TIGER/Line 1995. AZ, AR, KS, LA, OK, TX. Washington, D.C: U.S. Department of Commerce, Bureau of the Census, Geography Division, 1996.
Random W
alk
T
iger
Figure 8: Results of query experiments.
Table 1: Data Insertion
Dataset Tiger Random Walk
Record size 36B 120 B
Main mem. buffer page size 600B 600 B # records inserted 23,547,192 10,000,000
Total data size 847.7 MB 1.20 GB
Required disk space 995.7 MB 1.42 GB
Percent storage overhead 14.9% 14% Sustained insert rate (recs/sec) 2472 recs/sec 669 recs/sec Sustained insert rate (KB/sec) 89.0 KB/sec 80.2 KB/sec
Total number subindexes 84 106
