I will now introduce the remote memory access mechanism used in the Q-Machine implementation. The remote memory access mechanism is an im-portant component of the migration mechanism. Recall that the address space of ADAM is structured so that the processor node ID is the highest address bits; also, by convention, processor nodes occupy the even route addresses, and memory nodes occupy the odd route addresses. This allows processors and memory to be paired off into “preferred” pairs by the existence of a re-liable, in-order delivery cut-through network path between preferred pairs.
A local memory access is thus defined as a memory access where the node ID of the access capability is equal to the node ID of the preferred memory node. Local memory accesses are always serviced by the preferred
mem-ory node, and local memmem-ory allocation requests allocate data in the preferred memory node. The performance of accessing data in the preferred memory node is similar to that of an L3 cache access time on a contemporary proces-sor; please see section 6 for specific numbers.
Semantically, a preferred memory node is the target of allMML,MMSand EXCHqueue mappings, regardless of the access capability used to initialize the mapping. Thus, all remote requests are also routed from a processor node to the preferred memory node. When a remote request is initialized, the local virtual memory handler allocates local “shadow” pages for the remote capa-bility. Shadow pages serve two functions: first, they provide a method for storing the remote memory’s data locater pointer; second, they provide the infrastructure for caching immutable data. Shadow pages should never dis-place local memory pages when local memory is scarce. Hence, most of the shadow pages are not swapped into core or initialized when they are first al-located. The only exception is the first page. The first memory location of the first shadow page is the data locater pointer. This data locater pointer is initialized with the remote access capability. Note that the rest of the first shadow page’s space is marked as all invalid and all non-primary. Figure 4.1 illustrates the format of a remote capability in shadow space.
Figure 4.2 overviews the system level view of resolving a remote memory request. Remote requests are easily detected when a memory-mapped queue is initialized with its access capability: if the node ID of the access capability is not equal to the memory node’s ID, it must be a remote request. This remote request status is noted in the memory node’s access table tags (for more information on the memory node access table, please see section 5.3.2).
All requests from a processor node to a preferred memory node use the format of a transport packet without the physical layer route header and check-sums. More information on the transport protocol used in the Q-Machine can
data locater pointer remote
capability user base remote capability
user segment
first VM page is marked as "invalid" and "non-primary"
nonresident, unitialized pages
datamigformat.eps
remote capability system base
Figure 4.1: Format of a remote memory capability’s shadow space in local virtual memory space.
processor 2 preferred data memory space (handled by memory node 3)
1
3
5
7
9 shadow memory regions
processor 8 preferred data memory space (handled by memory node 9)
1
3
5
7
9 1. request
issued to memory via queue map 2. resolve data locater
pointer
3. send request to remote node all locations
non-resident, marked as non-primary on page-in
4. return data to requester via memory node,
cache immutable data processor 2
environment memory (thread space)
all forwarding is resolved here
local memory regions
2
remoteflow.eps
Figure 4.2: System level view of resolving remote memory requests.
be found in appendix C.2. These transport packets contain all the state re-quired to resolve the return address of the requester; thus, when forwarding a memory request, the memory node simply encapsulates the processor’s orig-inal request packet in forwarding headers and sends the encapsulated packet on to the remote memory node. Please see figure 4.3 for a more detailed il-lustration of how local and remote exchange (EXCH) mappings are handled.
TheEXCHoperation was chosen for illustrative purposes because it combines both a load and a store operation. A load operation uses exactly the load-half of theEXCHprotocol, and a store operation uses the store-half of theEXCH protocol, plus a store-acknowledge packet so that writes can be guaranteed to complete in program order.
P M
A Dout Din
cut-through interface
invalid locater pointer
EXCH, local memory case
P M
A Dout Din
valid locater pointer
EXCH, remote non-migrated memory case
M'
immutable memory cached locally
address/data bundled into single network packet
exchsimplecase.eps
EXCH queue mappings
Figure 4.3: Details of handling remote and localEXCHrequests.
Note that for compatibility with the migration mechanism to be outlined in the next section, all memory accesses, even stores, must check the valid and primary tag bits. If the existing value is invalid and the non-primary bit is set (as is the case when data has been migrated out), then the access table must be updated to forward future requests, and the current request must also be forwarded to the remote request queue. The overhead of tag checks on all requests, including stores, can be mitigated if dedicated hardware is provided in the memory implementation.