Cache

The new capability that the cache must provide is the ability to pin cache blocks that belong to a page being migrated by the DMA unit. My mechanism for pinning is based on a new PIN register, which holds the physical page address of the pinned page (see PPN in Figure 11). When the cache receives a message to pin a page, it stores the page number in this register and sets the valid bit (v) to indicate the register holds a pinned page. All blocks belonging to the pinned page are not candidates for eviction between page pinning (set v) and unpinning (clear v). Only one page can be pinned simultaneously.

the eviction process is modified. Read and write hits to the cache do not involve additional delay because PIN is not consulted during hits, and the tag and data array are unmodified. Pinning a page does affect the eviction procedure, but eviction can be performed in parallel with the memory access that brings new data into the cache. It involves only a few additional comparisons. Therefore, cache eviction with pinning does not add extra latency to a cache fill into the LLC.

Figure 13 shows the architecture of a 4-way set-associative cache and the way it is modified for pinning. Traditionally, addresses are divided in three parts for a cache access. The offset is used to identify what part of the cache block is being requested, while the index is used to select the correct set in the tag array. The tag is used to identify which way holds the correct block, if any. An address can also be divided into its page number and its page offset. Since my mechanism pins the whole page (as opposed to a single block), I keep the page number in PIN. In a typical cache configuration, the number of bits required for the page offset is greater than the number of bits for the cache offset (a page is usually larger than a cache block) and smaller than the number of bits for the cache offset and the cache index (cache size divided by associativity is usually much bigger than page size). Thus, only some of the bits are part of the page number. I term these bits high index bits (HIB), and the remaining index bits low index bits (LIB). As shown at the bottom of Figure13, the page number is formed from the cache tag bits and the HIB, while the page offset is composed of the LIB and the cache offset.

When a cache block needs to be evicted, the HIB of the requested address and of the pinned page are compared. If their values are equal and the PIN register’s valid bit is set, then this means one block in the cache set is potentially pinned. To determine if a block is actually pinned, the tag of the pinned page is compared with the tags from each way. If one of the tags match, the matching block belongs to the pinned page and should not be evicted. Note that it is unnecessary to access the tag array again because the tags for the set are already available from the original access that missed in the cache.

The result of the HIB and tag comparisons and the LRU bits for the set are fed into the LRU logic, which selects the way to evict. Assuming an LRU stack algorithm, the logic checks whether the LRU block is pinned, as determined by the tag comparisons. If it is, the

Figure 13: Cache organization with support for page pinning.

eviction algorithm simply chooses the second LRU block instead of the first one.

There are two observations to note about pinning. First, pinning affects only one cache block in a set because only one page is pinned at a time and each block of a page is mapped to a different set1_{. Therefore, there are always other blocks in a set that may be evicted to}

hold newly requested data. Second, PIN serves to avoid eviction of pinned blocks from the LLC. The caches “above” the LLC (e.g., L1 and L2) behave normally since holding blocks in the LLC does not influence their operation. The coherence protocol for these caches is similarly unaffected and needs no change for CPM.

Figure 14: Steps for hardware for CPM.

5.3.3 Concurrent Page Migration

CPM is initiated when the OS writes to the migration queue after determining what pages to migrate. To start a migration, the MM reads and removes the first entry in the migration queue. A migration is characterized by a 5-tuple of process ID, virtual page number, source physical page number, destination physical page number and the address of the PTE in memory. Figure 14gives an overview of CPM.

The first step pins the source physical page to the cache. To do this, the MM sends one message over the control portion of the interconnect to all cache slices with the physical page number of the page to be pinned (step 1.a). The caches write the page number into the PIN registers and set the PIN valid bits. If the destination page is a PCM address, the MM tells the caches to flush the source physical page (step 1.b).

In the second step, the MM programs the DMA unit to transfer the physical page to its new location. The DMA unit uses the migration buffer to temporarily hold data read from one memory before it is written to the other memory. When the copying finishes, the DMA unit signals the MM that the migration finished.

The third step atomically updates the PTE and TLB and flushes the page from the cache. To do this, the MM sends a message to all cores with the process ID and virtual

address of the page that is being migrated (step 3.a). The contents of this message are written to a core’s Pause on Access (POA) register and its valid bit is set. On every access, POA is compared to the requested address. Because the register holds a virtual address, the comparison is done simultaneously with address translation to hide the comparison latency. Once POA is set, if an executing program with the same process ID accesses the virtual address in the POA register, the core executing the program is stalled until the POA register is cleared. This action guarantees the PTE and TLB update and cache flush occur atomically. The application is allowed to execute as long as it only accesses other pages, since pinning prevents evictions of the migrating page caused by cache misses from other pages. After receiving acknowledgments from all cores that they updated their POA, the MM sends a message to all cores with the process ID, virtual page, PTE address and destination physical page of the migrating page (step 3.b). All cores that are executing the process under migration must update the entry, if present, in their TLB and invalidate the PTE from their private L1 caches. The PTE invalidation prevents the page table walker from reading a stale L1 value during TLB misses. The MM also writes the destination physical address into the page table by accessing the PTE directly in the corresponding L2 cache slice. In parallel with the update of the page table and TLB, the MM instructs the cache slices to flush the source physical address of the migrating page (step 3.b). Finally, the MM requests all cores to clear their POA valid bits and, concurrently, instructs the cache slices to unpin the page by clearing their PIN valid bits (step 3.c).