Signature generation

The basic idea behind signature generation of topological signatures, as described in Geva and De Vries [2011], is to create term signature vectors consisting of positive, negative and neutral values for each term in the document, add the vectors together and flatten the result to a binary vector, with values greater than 0 being represented with a 1 bit and values less than or equal to zero being represented with a 0 bit.

This process is visualised in § 3.4.3.

4.4.1 Desired properties for generated signatures

The exact method by which the term signature vectors are created is not defined and left up to the implementer. For the purposes of creating TOPSIG signatures, it is necessary that the process used for generating term signatures has the following properties:

• The positions of the affected columns should be well distributed.

Like with a hash table, the effectiveness of a signature file is highly dependent on the quality of the hashing function. One difference between signature files and hash tables is that the density of the set bits matters in signature files and does not matter in hash tables, due to the fact that in a document signature many different hashes are overlaid onto the same space and overly dense hashes can increase the rate of collisions to a level at which the individual terms are difficult to identify in the final hash. Ideally,

this term signature density should be configurable. As the interactions between term density and the quality of the signatures created is complicated this is something we would like to investigate.

• The signs of column values should also be evenly distributed.

If two different signatures affect the same column and use the same sign, if those two signatures both appear in a document that is being indexed, the result is that both signatures reinforce each other in that column. Compare what happened in § 3.4.3 with the 7^th column; multiple signatures in both documents added to that column, resulting in a final value of +3 in the first document and +2 in the second document. However, as

§ 3.4.5 highlighted, collisions are useful and allow term frequency and term weighting to work within the signature model.

Note that this does not mean that any one signature needs to have a certain ratio of positive and negative values; a signature can be entirely composed of values of the same sign and still function, providing there are plenty of other opposite-signed signatures around to offset this. As with the positions, the poorer the distribution is, the lower the quality of the generated signatures will be.

• A term should always hash to the same signature.

In § 3.4.3 the term "the" appears in both documents and generates the same result in each: (0, 0, 0, 0, -1, 0, +1, 0). This is important, as otherwise it would not be possible to regenerate the signature from the term to make it possible to search for the term again.

This property could be violated under certain circumstances; for instance, if a disam-biguation engine is used to determine the context of a particular term, it could be used to hash the same term into different signatures (for instance, to hash the fruit “apple”

to a different signature from the technology company “Apple”); but in most cases this is a necessary property of signature generation.

This research looks at generating document signatures through random indexing [Sahlgren, 2005]; however, any approach that fulfils the above conditions (perhaps utilising a different dimensionality reduction approach, such as singular value decomposition [Deerwester et al., 1990]) could be used. These dimensionality reduction techniques typically have performance

and scalability issues compared to random indexing, making them potentially less feasible for this purpose.

4.4.2 Reproducibility

As explained previously, it is essential that a given term produce the same signature each time. Failing to meet this requirement would render signature search futile.

There are at least two ways in which the reproducibility requirement can be satisfied:

Stored signatures

The signature for a term can be generated the first time that term is seen and then stored. It could then be retrieved for subsequent instances of that term appearing in the collection while indexing, and also retrieved for searching the collection.

This approach allows for a great deal of flexibility in the process by which term signatures are generated. Any sort of random number generator could be used, as could any source of entropy be used to seed it. It does, however, come with a number of issues:

• The term signatures table will need to be stored, not only while the collection is being indexed, but for all subsequent searches. This requires it to be used everywhere the signature data is used and effectively makes it a permanent attachment to the signature data file.

• Collections that were indexed independently will not be compatible and there will be no way of reconciling them without entirely re-indexing one of the collections. This further limits the ways signature files can be used.

• The vocabulary for a given collection can be very large. The Wikipedia XML corpus (§ B.2) has a vocabulary of over 2 million terms, approaching the number of documents in the collection, and bigger collections can have even larger vocabularies. Storing these could have serious ramifications for the storage (on-disk and in memory) of large collections.

• Indexing a collection requires that each time a document is indexed, read-write access to a table or associative array that stores the mappings of terms to term signatures is

needed, and it is necessary that this same table be used for every document that is indexed.

This can have serious ramifications for parallel indexing. Parallel indexing will be discussed in detail later, but the pertinent issue is that every thread that is doing the indexing will need to lock down this table constantly just to do its job, preventing the other threads from getting work done.

Associative data structures that require only the pertinent parts of the table to be locked when storing/retrieving records do exist; however, this degree of synchronisation can still have serious negative effects on the performance of a system.

Seeded signatures

Alternatively, the term could be used to seed the pseudo-random number generator used for generating the term signature vector. When the term is seen again, even if this incarnation of the platform has never seen this term before and does not have the representation of the term stored, the signature generated by the pseudo-random number generator for this term will be identical.

What this means is that the random number generator used is effectively built into the format specification for the signature file and requires that the random number generator be available and work in the exact same way on every platform to ensure signatures are adequately portable. However, compared to the many downsides of storing the generated signatures, this is a clearly superior choice.

The use of a pseudo-random number generator seeded with the contents of the signature makes the term signature generator conceptually a hash function, albeit one with more exact requirements for the properties of the output result than a typical hash function.

4.4.3 Term signature generation

With the method of obtaining random data appropriate for generating the signature with having been chosen, the remaining problem is how to generate the term signatures using said random number generator. Being able to configure the signature density is also a desirable property, as described earlier.

To summarise the problem at hand, the term signature generator should utilise a pseudo-random number generator, seeded with the term the signature is being generated for, to generate a vector of a specified length with values randomly chosen from neutral, positive and negative based on the specified signature density. For example, with a signature density of ₁₀¹ , each value will have a ₁₀⁹ chance of being neutral (zero), a ₂₀¹ chance of being positive and a ₂₀¹ chance of being negative.

The most straightforward approach to meet this requirement is to simply iterate over the signature width, producing random numbers and comparing them against the desired signature density. The upside to this approach is that it is very simple to reason about and show that it demonstrates the desired properties. The downside is that it may not be the most efficient way to generate signatures.

4.4.4 Term signature performance

One issue potentially driving performance concerns in this area is the performance of the random number generator; use of a slow generator will directly impact indexing speed. How-ever, even if the random number generator is extremely efficient, it is easy to see that having to generate 1024 values (assuming 1024 bit signatures) for every term in every document is undesirable if it can be avoided.

• One potential step is simply to get more use out of each value the random number generator outputs. Consider a relatively common approach for generating random probabilities (values between 0.0 inclusive and 1.0 exclusive): you take a number output by the generator, for instance, an unsigned 32 bit value; move it into a floating-point type and then divide it by 2³²to produce a value in the desired range.⁵ This can then be compared with probability thresholds to determine whether to add a neutral, positive or negative value to the vector.

Hence, one potential optimisation is to reduce the precision of the generated random values and generate more of them with each call to a random number generator. For example, with the same random number call, two 16 bit numbers could be generated

5Conceptually. As we are concerned with performance it makes more sense to normalise the probability thresholds into the desired range and do the comparisons there so unnecessary floating point conversions are not performed.

instead. This would not necessarily mean that two values of the signature could be generated in the same time as one with the 32 bit random numbers, but it would almost certainly be faster. This could be considered trading signature quality for performance.

• An alternative approach, to avoid sacrificing signature quality at all while making more use out of the random values, is to constrain the signature density to values that can easily be selected with a random number generator producing small random numbers.

For example, if a signature density of ¹₈ was chosen, this would mean a ¹⁴₁₆ chance of a value being neutral, a₁₆¹ chance of it being positive and a ₁₆¹ chance of it being negative.

A 4 bit random number generator, producing values between 0 and 15 would be enough to manage that selection: if a value of 0–13 was returned it could mean neutral, while a value of 14 could mean positive and a value of 15 could mean negative. This means that a 32 bit random number generator would be able to cover eight columns with one call, requiring a total of 128 calls to the random number generator for a 1024 bit signature.

Naturally, this comes with the downside of restricting the available range of signature densities. In cases where a certain signature density produces superior results to the best signature density with a power of two denominator, this could end up being another speed-quality tradeoff.

• One other potential approach to reduce the number of calls to the random number generator is to take advantage of the fact that term signatures vectors are typically going to be quite sparse. This makes sense, as overly dense signatures will have to deal with a larger number of collisions and there is some evidence that this has an effect on the results; Geva and De Vries [2011] suggest a signature density that works out to be

1

6 in the scheme used here.

Hence, a superior approach for generating signatures is to instead choose the number of columns that will have non-0 values based on the signature density, randomly se-lect those columns and give them positive or negative values, randomly chosen. As choosing whether a value is positive or negative can be done with just one bit out of the random result, the number of calls to the random number generator could be reduced to the same fraction as the signature density; a signature density of ¹₆ should mean ¹₆ of the calls to the random number generator.

There is also further room for optimisation in there. If restricting the signature size

to a power of two is considered more acceptable than restricting the denominator of the signature density, a similar optimisation can be performed here. As an example, consider a 128 bit signature: it would take 7 random bits to decide which column to affect and 1 random bit to decide whether it should be set to a positive or negative value⁶. If the desired signature density is ¹₈, a total of 16 columns would need to be set to match this density. Setting these 16 columns would require a total of 8 × 16 (128) bits of random data, or only four calls to the random number generator. Although more typical signature lengths may not fit into the results returned by the random number generator as neatly, there is plenty of room for improvement over the na¨ıve approach.

Empirically comparing the performance of the different approaches is a simple task, as the signature generation algorithm can be implemented in a modular enough fashion to be easily swappable, either as a compiler flag or as a runtime argument. As a result, comparing the runtime performance of the signature generation methods is simply a matter of indexing the same collection multiple times using the different methods and timing them.

Another factor that needs to be measured is the search quality of the signature methods.

Although the different methods are fundamentally designed to produce signatures with the same statistical properties, they do generate signatures in different ways and hence will produce different signatures, simply by virtue of using the random numbers for different things. It is therefore important to ensure that the optimised signature generation approach does not significantly impact the quality of results. This is more difficult to judge.

To measure the search quality we will simply be using a standard set of topics with associated relevance judgements and measuring the P@10 (precision at 10 documents) of each method. The issue that arises here is that differences in the results may not necessarily show that certain signature generation methods produce higher quality signatures than others.

As all of these methods are random indexing methods there is automatically a certain level of randomness in the results, even though the same signature methods will consistently produce the same signatures for each term.

The impact of this element of random chance on the final results can be diminished by sampling multiple results by seeding the random number generator differently each time. The

6The sign bit could be omitted if, for example, the algorithm was constrained to output the first half of the bits as positive and the second half as negatives. For this research we are considering approaches that preserve the binomial distribution of bit signs, however.

Signature method Indexing time (s) P@10 σ(P@10)

Na¨ıve 503.631 0.3488 0.0083

Positional 125.682 0.354 0.015

Table 4.1: Comparison between indexing time of signature methods

random number generator is already seeded with the term the signature is being created for, so to introduce a level of variance suitable for sampling we introduce a second parameter into the seed, an optional signature seed which can then be varied to produce different results and reduce the effect the element of chance has on the final results.

Table 4.1 shows the results of tests performed by indexing the WSJ87-92 collection (§ B.1) with 1024 bit signatures. The two methods being tested are:

Na¨ıve

The method described in § 4.4.3, involving one call to the random number generator for each column in the signature.

Positional

The method described earlier, involving one call to the random number generator for each column being set, with that call determining the position of the column.

The P@10 score was averaged from a sample of 5 differently-seeded runs.

The positional signature generation approach performs much more efficiently than the na¨ıve approach, with the overall indexing time reduced to less than a quarter of the original time. The search quality is virtually unchanged; while the positional approach did produce slightly better results, the results are too close to matter in practice.

4.4.5 Term signature caching

In § 4.4.2, the concept of storing term signature vectors after their generation in an associative array or table is discussed. While this is a suboptimal model for various reasons as described in the same section, one advantage it provides is that terms do not need to be regenerated every time they are seen. Depending on factors such as the signature width and density, the performance of the random number generator etc. in certain cases it may be faster to look up

a signature that has already been generated rather than generating it again. This aspect can be leveraged in conjunction with seeded signatures to create a term signature cache.

Caching signatures solely to improve the performance of the signature indexer provides a number of advantages over storing signatures permanently:

• The table associating terms to the signatures they generate does not have to become a permanent attachment to the signature data. As the signatures can be regenerated without it, whenever the table becomes inconvenient to store it can be discarded.

• Signatures created by different instances of the application remain compatible.

• Parallel indexing causes no issues beyond the need to find an approach to caching signatures that does not require excessive synchronisation between threads. As there is no need to maintain consistency between the view of the term cache that each thread has, there is a large amount of room for flexibility in determining the most efficient approach.

One potential approach is to have each thread have its own copy of the signature cache that the other threads don’t even look at; while term signatures would need to be generated more times than they necessarily would due to threads not being able to use the cached versions created by other threads, the efficiency gains from avoiding the need to lock/unlock shared caches can easily make up for it.

Even without entirely isolating the individual caches like that, the fact that the threads can recreate the signatures on their own still grants plenty of flexibility in cases where it might simply be cheaper to create a signature again rather than retrieve it from a cache.

Another possibility is to take advantage of the fact that most of the common terms will have already appeared after indexing for a certain amount of time, presuming the documents in the collection are well distributed. Hence a long-term cache could be frozen a set portion of the way in and made read-only. Multiple threads can access a read-only data structure without any need for synchronisation. Individual threads can also keep smaller short-term caches for catching terms that are not common but can appear many times in those documents in which they do appear.

• It is not necessary for every term signature to be cached. The most benefit would come

• It is not necessary for every term signature to be cached. The most benefit would come