3.3 Evaluation
3.3.4 Checkpointing with MTR
Checkpointing removes the overhead of repeatedly creating MTRs with functional simulation when evaluating multiple microarchitectural configurations. However, it is more difficult to implement and requires large files on disk to hold the memory contents at each checkpoint. MTR snapshots can be added to architectural checkpoints to reduce the amount of detailed warming required. Because the MTR is microarchitecture-independent, many different con- figurations can be initialized from the single snapshot.
The MTR representation is highly compressible. A significant savings is achieved in the spatial dimension using the multilevel valid-bit scheme, discussed in Section 3.1.4. This eliminates the need to store addresses along with data: the address can be reconstructed based on offset within the snapshot and the empty page bit vector. In the temporal dimension, delta encoding may be used instead of full timestamps — although we show that this offers relatively minor benefit for short warming periods.
Scheme ID Number of blocks Page size
Block size
Number of CPUs (N)
Table 3.7. Header of compressed MTR.
To examine the extent to which the MTR can be compressed, we implemented several compression schemes. Each scheme uses a common header that contains the fields shown in Table3.7. The Scheme ID indicates which format is in use and allows proper interpretation of the structure. The number of blocks can be divided by the page size to discover the number of pages; several schemes require this number to parse the next block of the snapshot. Block size is used to infer the address represented by each MTR entry, and the number of CPUs is needed to determine the entry’s size.
The schemes are enumerated below:
1. All. Each block of the MTR is written to disk using the variable-length representation shown in Table 3.8. We assume 1 byte to store the CPU identifier (supporting up to 256 CPUs) and 4 bytes for a logical timestamp (supporting up to 232memory accesses in a warming period). Output is written at a byte-granularity as it is more easily compressed by general purpose compressors such as gzip and bzip2.
2. Accessed. The output begins with a vector denoting which pages of the MTR contain blocks that have been accessed. Only those pages indicated by the vector are written using the format of the All scheme.
3. Accessed unshared. Even in programs that share memory among several processors, many memory blocks are accessed by just a single processor. This scheme employs the alternate encoding shown in Table3.9 to reduce the space occupied by these common single-access cases.
4. Accessed horizontal. We begin with the vector of accessed pages described above. When outputting MTR records, write time is emitted explicitly, but read times are re- ported as deltas from the write time. The hope is that the read timestamp arrays become similar arrays of small numbers which would be easier to encode than mono- tonically growing timestamps.
5. Accessed vertical. We begin with the accessed page vector. The first valid record on each page is denoted the base. Additional write times on the same page are reported using a delta from the base write time. Additional reads are reported as deltas from the N base read times. This scheme should work well in the presence of locality at the page granularity.
Valid Remainder Record Size (Bytes)
Valid entries 1 < writer, last write time, read time array > 1+(1+4+4N)
Invalid entries 0 <> 1
Table 3.8. Variable length MTR records: default encoding.
Code Contents Record Size (bytes)
Invalid entries 0 <> 1
Single writer 1 < writer, last write time > 6
Single reader 2 < reader, last read time > 6
Default 3 < writer, last write time, read time array > 6+4N
Table 3.9. Variable length MTR records: Accessed unshared encoding.
6. Accessed vertical2. Same as Accessed vertical. However, after each block is emitted, its times become the new base time so that each block’s times are output relative to the previous block on the page. This should work well for unit-stride accesses.
7. Accessed both. This scheme combines the horizontal (temporal) and vertical (spatial) approaches. We record a base write time for the first valid block on a page. Read times for the first block are reported relative to the base write time. Write times for following blocks are also reported relative to the base write time. Read times for following blocks are reported as the difference between the actual read time and difference between the record’s write time and the base write time:
delta time = mtr[record].readtime[cpu] - base write time - mtr[record].writetime. We use two baselines, similar to the analysis in Section 4.4.1. The first is a trace of all memory accesses. Like the MTR, such a representation allows reconstruction of any cache or directory state, but unlike the MTR, the trace is likely to contain many references that are not actually necessary to warm-up state. Each reference is represented with the fields shown in Table3.10. Second, we consider a direct serialization of the contents of each CPU’s cache. Storing concrete cache state allows quick reconstruction, but only for the microarchitecture that has been stored. We assume a 4-way, 16 KB cache per CPU.
The cache state is serialized to a snapshot using a method similar to the MTR All scheme as shown in Table3.11. The encoded cache blocks are preceded by a header which provides the number of blocks, block size, and the number of bytes per way. A footer provides the LRU metadata for each set. Each of the three representations (MTR, Memory Trace, and Concrete Cache) is compressed with gzip and bzip2. The per-cpu cache snapshots are combined with
State Size (Bytes)
Load/Store 1
CPU 1
Address 4
Valid Remainder Record Size (Bytes)
Valid entries 1 < tag, dirty > 6
Invalid entries 0 <> 1
Table 3.11. Variable length encoding of cache blocks.
tarbefore compression so that the general-purpose compressor is able to take advantage of similarities between individual snapshots.
Each benchmark is run until it has completed one unit of work, advancing to each sample in an online mode. State snapshots are taken at the end of each warming period. The MTR is cleared, and the process repeats. Thus, the nth snapshot, snapn contains information about the memory accesses that have occurred between samples n− 1 and n. Reconstructing the state at sample n requires snap1 through snapn inclusive.
Figures3-19and3-20show compression ratio normalized to the size required to store a snapshot of eight 16 KB caches (compressed file size / compresses cache snapshot size). The bars are annotated with compression ratio when it is off the scale. The figures show that the compressed MTR always requires less storage than the memory trace. In several cases, the memory trace does surprisingly well when compared to the MTR and would likely improve if specialized compression were applied. However, if larger samples are desired, the self- compressing nature of the MTR is an advantage. Furthermore, while a trace is versatile, it requires time to simulate each of the memory accesses to warm a cache. MTR reconstruction, however, scales with the number of entries in the MTR.
With gzip compression, the MTR requires 6.6–7.1 times more storage than eight 16 KB caches (depending on the length of the fast-forward period), but is more flexible than the small concrete cache snapshot. With bzip2 compression, the MTR improves, requiring 6.3– 6.9 times the space of a concrete snapshot. Memory traces require 55–65 times more space than the cache snapshot, or 25–31 times the space when bzip2 is used. Interestingly, bzip2 is able to make the MTR’s All format compress to almost the same size as its Accessed format.
The simple Accessed unshared scheme proves almost as good as the techniques which use time deltas. This is likely due to the fact that, during relatively short warm-up periods, regular time strides are not nearly as common as singly-accessed blocks. Though the use of deltas reduces the number of bits that are needed to represent a timestamp, deltas do not skew the distribution enough to aid the entropy coding of gzip and bzip2. For example a bunch of large numbers such as “36114” can be encoded with just as few bits as an equally likely bunch of small numbers like “14.”
The amount of storage can be dramatically reduced — by a factor of 720 — by storing only memory access information for those blocks that will be accessed during an ensuing detailed sample point [125]. This requires that the simulation progress through the instruc- tions belonging to the detailed sample, record the required memory addresses, and include only information about these addresses in the preceding snapshot. Such a method excludes
references generated by incorrect speculation. In practice, few such accesses are observed, so the impact on performance estimation is small. This method of excluding portions of mem- ory was proposed to reduce the amount of memory values needed in a memory snapshot, but could similarly be applied to exclude the corresponding unneeded memory metadata from an MTR snapshot.