P3,.; .. ,, P2�_,, B2;-;, P2� .. r, B l :-: , P 1 " .r, P 1 "-''
( last) ( first)
there is a ti.nite set of val id reception orders at destina tion nodes X a nd Y, depending on the actual arrival time of the requests to the point of global ordering. Rule
1
d ictates that al l packets from node M (or N) to node X (or Y) must arrive at node X (or Y) i n the order in which they were transmitted . Ru le 2 dictates that, regardless of the relative order among the senders, messages destined to both receivers must be received in the same order. For example, if X receives B2", B l "', and B 1 " ' then Y should receive these packets in theRECEIVER RECEIVE PCT R E C EIVE ENABLED INTERRUPT ON RECEIVE SENDER TO 1/0
LOCAL COPY ON TRANSMIT ACKNOWLEDGE REQU EST TRANSMIT ENABLED UNDER E R ROR BROADCAST OR POINT-TO-POINT REQUEST ACKNOWLEDGE
- - - -· RECEIVE ENABLED UNDER E R RO R REMOTE READ RECEIVER LOAD MEMORY SPACE SPACE Figure 2
MEMORY C HAN NEL Page Control Attributes
Digital Technical )oumal Vol . 9 No. I 1997
same order. One arrival order congruent with both of these rules is the fol lowing:
at node X,
P3 � .,, P2" .,, P2" .,, B2,, B l ,�> B 1:-�, P l , .,, P l ," .,
(last) (first)
at node Y,
B 2 s, P2, .. " P l ,,l-) , B l ." , B l ", P l , ·r·
These rules are independent of a parrjcular intercon nection topology or implementation and must be obeyed in all generations of the M EMORY CHANNEL network.
On the lvlEMORY CHANNEL network, error han d l ing is a shared responsibility of the hardware and the cl uster management software. The hardware provides real -time precise error hand ling and strict packet ordering by discarding all packets in a particular path that fo l low an erroneous one. The software is respon sible f(>r recovering the network fi·om the faulty state back to i ts normal state and for retransmitting the lost packets.
Additional MEMORY CHANNEL Network Features Three additional features of the MEMORY CHANNEL network make it ideal f(>r cl uster interconnection:
l . A hardware- based barrier acknowledge that sweeps tbe nenvork and all its buffers
2.
A
fast, hardware-supported lock primitive 3. Node failure detection and isolationBecause of the three ordering rules, the MEMORY CHANNEL nenvork acknowledge packets are imple mented with little variation over ordinary packets. To request acknowledgment of packet reception, a node sends an ord inary packet marked with the request acknowledge attri bute.
The
packet is used to sweep clean the network queues in the sender destination path and to ensure that all previously transmitted pack ets have reached the destination . ln response to the reception of a M EMORY CHANNEL acknowledge request, the destination node transmits a M EMORY CHANN EL acknowledgment back to the originator. The arrival of the acknowledgment atthe
originating node signals that all preceding packets on that path have been successfu l ly received .MEMORY CHANNEL
locks
areimplemented
using a lock-acquire software data structure mapped as both incoming and outgoing by all nodes in the cluster. That is, each node wil l have a local copy of the page kept coherent by the mapping. To acq uire a l ock, a node writes to the shared data structure at an offset corresponding tO its node identi fier. !vl EM O RY CHANN EL ordering ruJes guarantee that the write order to the data structure-including the update ofthe copy local to the node that is setting the Jock is the same for all nodes. The node can t hen determine if it was the only bidder for the l ock, in which case the node has won the lock. If the node sees multiple bidders for the same lock, it resorts to an operating system-specific back-offand -retry algorithm. Thanks to the M EMORY CHANNEL !:,'llaranteed packet order ing, even under error the above mechanism ensures that at most one node in the cluster perceives that it was the first to write the lock data structure. To guarantee that data structures are never locked indefi nitely by a node that is removed from a c luster, the cluster manager software also monitors lock acquisi tion and release.
The M EMORY CHANNEL network supports a strong-consistency shared -memory model due to its strict packet ordering. In addition, the
1/0
operations used to access the M EMO RY CHANN EL arc ful l y integrated within the node's cache coherency scheme. Besides greatly simpl it)1ing the programming model, such consistency allows tor an implementation of spinlocks that does not saturate the memory system . For instance, whi l e a receiver is polling tor a tlag that signals the arrival of data ti·om the MEMO RY CHANNEL net\vork, the node processor accesses only the locally cached copy of the flag, which will be upd ated whenever the corresponding main memory location is written by a M EMORY CHANNEL packet. Unlike other networks, the MEMORY CHANNEL hardware maintains information on which nodes are currently part of the cluster. Through a collection oftimeouts, the MEMORY CHANN EL hardware con
tinuously monitors all nodes in the cl uster tor illegal behavior. When a tail ure is detected , the node is iso lated from the c luster and recovery software is invoked .
A
MEMORY CHANNEL cluster is equ ipped with software capable of reconfiguration when a node is added or removed ti-om the cl uster. The node is simply brought on-line or off-line, the event is broad cast to al l other nodes, and operation continues. Toprovide tolerance to net\vork fai lures, the cluster can be equipped with a pair of topologically identical
MEMORY CHANNEL nenvorks, one
f()l'
normal operation and the other for tailover. That is, when a nonrecoverable error is detected on the active MEMO RY CHANNEL nenvork, the sofuvare switches over to the stand by nenvork, in a man ner transparent to the appl ication .'"
The First-generation MEMORY CHANNEL Network The first generation of the J\II EMORY CHANNEL ncnvork consists of a node interrace card and a con centrator or hub. The interface card, ca lled an adapter, plugs into the
1/0
PCl. To send a packer, the CPU32