Interconnection Networks
54
0
0
Full text
(2) Agenda 1. 2. 3. 4. 5. 6. 7.. 2012-08-30. Introduction Static Networks Dynamic Networks Routing and Switching Communication Operations Practical Aspects Cray's High Speed Network. 2. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(3) 1. Introduction. Interconnection Networks (I) Data transfer between ● ●. Processing nodes Processors and memory. Abstract view: n inputs, m outputs Typical components: links, switches, interfaces Switching element. Left: Static network Right: Dynamic network. Network interface/ Switch. P. P. P. P. P. P. P. P. Processing node 2012-08-30. 3. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(4) 1. Introduction. Interconnection Networks (II) Topology – describes the geometric structure ●. ● ●. Graph – switches, processors, memory as nodes; connection links as edges Static networks = Direct or Point-to-point networks Dynamic networks = Indirect networks. Routing technology – defines how and along which path messages are transported ● ●. Routing = routing algorithm selects a path Switching strategy = defines segmentation of messages, mapping to a path, handling by switches and processors. Topology and Routing technology determine decisive the performance of the communication 2012-08-30. 4. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(5) 2. Static Networks. Criteria for Networks (I) Diameter Maximum distance between any two processing nodes Distance: Shortest Path (number of links) between two nodes Measure for the time to transfer messages between arbitrary nodes. 2012-08-30. 5. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(6) 2. Static Networks. Criteria for Networks (II) Degree Maximum degree of all processing nodes Degree of a node is the number of in- and outgoing links of a node Measure for the number of simultaneously active communication connections Measure for hardware efforts. 2012-08-30. 6. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(7) 2. Static Networks. Criteria for Networks (III) Connectivity Measure of the multiplicity of pathes between arbitrary processing nodes High connectivity lowers contention, increases reliability Arc (node) connectivity = number of links (nodes) to remove to separate the network in two disconnected networks 2012-08-30. 7. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(8) 2. Static Networks. Criteria for Networks (IV) Bisection Width, Bisection Bandwidth Bisection width = minimum number of links to remove to get two equal halves, Bisection bandwidth = minimum volume of communication between the halves. Bisection bandwidth = bisection width x channel bandwidth Measure of loading capacity: simultaneously transmission of „bisection width + 1“ messages may saturate the network Multicore – Bisection bandwidth per core 2012-08-30. 8. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(9) 2. Static Networks. Criteria for Networks (V) Cost Many criteria possible ● ●. Number of communication links, number of wires Bisection bandwidth. Technology Additional equipment. 2012-08-30. 9. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(10) 2. Static Networks. Criteria for Networks (VI) General Requirements Small diameter short distances for transmissions Small node degree reduction of the hardware efforts High bisection bandwidth high throughput High connectivity high reliability Good extensibility 2012-08-30. 10. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(11) 2. Static Networks. Topologies (I) Completely connected Diameter Degree. p−1. Arc connectivity. p−1. Bisection bandwidth Cost (# links). 3 0 - 0 8 -2 0 1 2. 11. 1. 2. p 2 p⋅ p−1 2. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(12) 2. Static Networks. Topologies (II) Star Diameter Degree. 12. p−1. Arc connectivity. 1. Bisection bandwidth. 1. Cost (# links). 2012-08-30. 2. p−1. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(13) 2. Static Networks. Topologies (III) Linear Array Lin. Array. Ring. p−1. p 〚 〛 2. Degree. 2. 2. Arc connectivity. 1. 2. Bisection bandwidth. 1. 2. p−1. p. Diameter Linear Array (no wraparound). Cost (# links) Ring. 2012-08-30. 13. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(14) 2. Static Networks. Topologies (IV) Diameter. Mesh. Degree. Mesh (d dimensions) Torus (d dimensions) p=r. 2012-08-30. 14. d. Mesh d. d⋅ p−1. Cost (# links). d. d⋅〚. p〛 2. 2⋅d. 2⋅d. d. 2⋅d. Arc connectivity Bisection bandwidth. Torus. p. d −1 d. d. d⋅ p− p. 2⋅p. d −1 d. d⋅p. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(15) 2. Static Networks. Topologies (V) Hypercube 101 1. 01. 111. 011 100. 00. 110. 10 000. 0-d. 1-d. 2-d. 010. 3-d. Diameter. log p. Degree. log p. Arc connectivity. log p . Bisection bandwidth 0101. 0001. 0111. 1101. 0011 0100. 0000. d. 11 001. 0. p=2. 1001 0110. 1111. 1100. 1000. 0010. Cost (# links). 1011. p 2 p⋅log p 2. 1110. 1010. 4-d 2012-08-30. 15. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(16) 2. Static Networks. Topologies (VI) (Static) Tree. k. p=2 −1 Compl. binary tree Diameter. Complete binary tree. 2012-08-30. 16. 2⋅log . n1 2. Degree. 3. Arc connectivity. 1. Bisection bandwidth. 1. Cost (# links). p−1. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(17) 2. Static Networks. Topologies (VII) Summary table Compl. Connect.. Star. Lin. Array. Ring. d⋅ p−1. Mesh. Torus. Hypercube. Binary Tree. 1. 2. p−1. p 〚 〛 2. Degree. p−1. p−1. 2. 2. 2⋅d. 2⋅d. log p . 3. Arc connectivity. p−1. 1. 1. 2. d. 2⋅d. log p . 1. 1. 1. 2. d −1 d. p 2. 1. p−1. p−1. p. p⋅log p 2. p−1. Diameter. Bisection bandwidth Cost (# links). 2012-08-30. 17. 2. p 2 p⋅ p−1 2. d. p. d. d⋅ p− p. d. d⋅〚. p〛. 2⋅p. 2. d −1 d. d⋅p. log p 2⋅log . n1 2. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(18) 4. Dynamic Networks. Criteria for Networks (I) Similar to static networks Processing in switches costs time seen as nodes Diameter = maximum distance between any node (in practice processing nodes) ●. Node and edge connectivity = number of nodes or edges to remove to get two networks Bisection bandwidth = any possible partitioning of processing nodes into two equal parts to consider induces partitioning of switching nodes with minimized number of crossed edges 2012-08-30. 18. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(19) 4. Dynamic Networks. Bus, Crossbar Bus. Crossbar. Simple, cheap. Complex, expensive. Constant distance. Non-Blocking. Good for Broadcasts. Realization hard for large p and high speed. Scaling limited by performance P2 C2. .... Pp Cp. 64. P1 .... P1 C1. Scaling limited by cost. Pp .... M1. Mm. Disk M1. 2012-08-30. 19. .... Mm. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(20) 4. Dynamic Networks. Multistage Interconnection Networks (I). .... Stage n. .... .... M2. Stage 2. P2. .... constuction rule ● degree of switching nodes Examples. M1. Stage 1. Characteristics. P1. .... Intermediate features between bus and crossbar ●. ● ● ● ●. omega network baseline network butterfly network benes network. Pp. Pm. Schematic view of a multistage interconnect network. Straight - Crossover. Upper and Lower Broadcast. Switch positions for a 2x2 switch 2012-08-30. 20. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(21) 4. Dynamic Networks. Multistage Interconnection Networks (II) Omega network ●. ● ●. Example of a blocking network log(p) stages Construction rule: j=. 000. 001. 001. 010. 010. 011. 011. 2i ,. p 0≤i≤ 2. 100. 100. 2i1− p ,. p ≤i≤ p−1 2. 101. 101. 110. 110. 111. 111. {. Number of switching nodes p ⋅log p 2. 2012-08-30. 21. . 000. (perfect shuffle). ●. . 8= 0 1 2 3 4 5 6 7 14702653. Realisation of an omega network with 2 x 2 switches. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(22) 3. Dynamic Networks. Tree-based networks (Dynamic) Tree Nodes at intermediate levels are switches, processing nodes are leafs Communication bottleneck at higher levels. Tree with dynamic network 2012-08-30. 22. Fat tree Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(23) 3. Dynamic Networks. Properties Summary table Crossbar. Omega Network. Dynamic Tree. Diameter. 1. log p . 2⋅log p . Arc connectivity. 1. 2. 2. Bisection bandwidth. p. Cost (# links). p. 2012-08-30. 23. 2. p 2 p 2. 1. p−1. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(24) 4. Routing and Switching. Routing Routing algorithm defines a path to send messages between nodes Requirements ● ● ● ●. Deadlock-free Consideration of the topology Avoid Contention Avoid Congestion. Types of algorithms ● ●. 2012-08-30. minimal, non-minimal deterministic, adaptive 24. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(25) 4. Routing and Switching. Switching Switching strategy defines how a message travels along a routing path ● ● ●. Segmentation Allocation type of the path How messages are processed in switching nodes. Strong influence on the transfer time of messages between nodes. 2012-08-30. 25. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(26) 5. Communication Operations. Message Passing Costs Message size. m [ B]. Bandwidth Transfer time. Latency = Overhead + Transfer time [ B⋅s−1 ] 1 t B= bandwidth t m=t s t B m m =t m bandwidth B. Hop time. th. Byte transfer time. Signal Delay Transport Latency Sender overhead Receiver overhead 2012-08-30. 26. Latency. Sender Receiver. Sender overhead. Transmission time Signal delay. Transmission time. Receiver overhead. Transport latency Total latency [after Hennessy, Patterson] Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(27) 4. Routing and Switching. Store-and-Forward Routing Message is transferred completely between nodes on the path Communication time for path of length l: t comm=t s m t B t h l. Time P1 P2 P3 P4. simplified for modern equipment:. Store-and-Forward Routing with 4 nodes. t comm=t s m l t B. 2012-08-30. 27. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(28) 4. Routing and Switching. Packet Switching Time. Divide mesage in packages to reduces transfer time. P1. Other advantages. P3. ● ● ●. Packet losses cheaper Packages can use different pathes Better error correction possibilities. Communication time (static route). P2 P4 P1 P2 P3 P4. t comm=t s t h lt B ' m s t B ' =t p t B 1 Packet Routing with 4 nodes r for a message divided in packages t p effort to packetize message, r message length in packet, sheader size of packet. . 2012-08-30. 28. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(29) 4. Routing and Switching. Circuit Switching Path established through the sending of a control message, exists until the the communication ends All nodes active at the same time to transfer the message Communication time for path of length l: t comm=t s t B mc l m. Time P1 P2 P3 P4. Circuit switching with 4 nodes. mc length of control message 2012-08-30. 29. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(30) 4. Routing and Switching. Cut-Through Routing Virtual Cut-Through Routing ●. Packages are subdivided and transported in a pipelined manner after evaluation of the header ●. ●. Message is divided into „flow control units“ (flits) – smaller than packets. Collection at nodes where route is blocked. Variant Wormhole Routing ●. Flits are blocked at their current position. Advantages: ● ●. Safe of intermediate stores and sends Reduced buffer size. Communication Time:. 2012-08-30. 30. t comm=t s l t h t B m. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(31) 5. Communication Operations Communication influences the efficiency of a prallel program essentially Examples presented here should give an impression how important good implementations are (what will be done for most of us by the developers of libraries like MPI). Assumptions for the following ● ● ●. 2012-08-30. Cut-through routing Bidirectional links Single-port communication model 31. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(32) 5. Communication Operations. Linear Array, Ring (I) 3. 3. 1. 1. 2. 2. 7. 6. 5. 4. 7. 6. 5. 4. 0. 1. 2. 3. 0. 1. 2. 3. 1 3. 3. 2. 1. 3. One-to-All Broadcast on a ring with eight nodes (MPI_Bcast). 2 1. All-to-one Reduction on a ring with eight nodes (MPI_Reduce). Dual operations [after Grama et al.] 2012-08-30. 32. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(33) 5. Communication Operations. Linear Array, Ring (II) 1 (4). 6. 5. (7). (6). (5). (4). (0). (1). (2). (3). 1. 2. 1 (0). 1 (1). .... 7(0). 3. 7. 1 (2). (7,6,5,4,3,2,1)) th. 7 Step 2(5). 6. 5. (7,6). (6,5). (5,4). (4,3). (0,7). (1,0). (2,1). (3,2). 1. 2. 0. 4. 7(6). 6. 5. 4. (6,5,4,3,2,1,0). (5,4,3,2,1,0,7). (4,3,2,1,0,7,6). (1,0,7,6,5,4,3). (2,1,0,7,6,5,4). (3,2,1,0,7,6,5). 1. 2. 7(2). 7(3). 3 7(4). 2(2). 2nd Step. (0,7,6,5,4,3,2). 2(3). 2(6). 7. 2(4). 7(7). 7(5). 0. 4. 7(1). 1st Step. 1 (7). 7. 1 (5). 1 (3). 1 (6). 0 2 (7). 2 (0). 3 2 (1). All-to-All Broadcast on a ring with eight nodes (MPI_Allgather). [after Grama et al.] 2012-08-30. 33. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(34) 5. Communication Operations. Mesh 15 4. 4. 11. 4. 5. 6. 7. 0. 1. 2. 4. 4. 4 2. 3. 10. 3. 9. 3. 8. 4. 3. 14 4. 13. 4. 12. 2. 3. 1 One-to-All Broadcast on a mesh (MPI_Bcast) [after Grama et al.] 2012-08-30. 34. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(35) 6. Practical Aspects. MPI Benchmarks Quick evaluation of a system ● ●. Self-written or some general available tests IMB: http://software.intel.com/en-us/articles/intel-mpi-benchmarks/. Example: Ping-Pong Process 0. Process 1. MPI_Send(). t. MPI_Recv() MPI_Send() N1/2. MPI_Recv() 2012-08-30. 35. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(36) 6. Practical Aspects. Simplified Cost Model Cut-Through-Routing ● ● ●. t comm=t s l t h t B m. Prefer communication in bulk Minimization of the transfer distance Minimization of the data volume. Reality allows simplification ● ● ●. t comm=t s t B m. Limited influence on process mapping Often randomized routing Per-Hop time can be ignored often. Consequences for programmer ●. ●. Assumes the same time between arbitrary nodes (= assume completely connected network) Accuracy loss: Only valid in networks without congestion ● ●. 2012-08-30. Topologies are sensible to congestion in different grade Communication patterns produce different congestion 36. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(37) 6. Practical Aspects. Hybrid parallelization Hybrid parallelization – f.ex. combined use of MPI and OpenMP or MPI and Pthreads or... Promising approach due to increasing number of cores sharing memory and decreasing available network bandwidth per core Potential advantages ● ● ●. Better scaling (avoiding data decomposition) Reducing load imbalances Reducing resource requirements (communication volume and bandwidth, memory). Problems ● ●. 2012-08-30. Increase of the complexity in the development Success not guaranteed, time effort 37. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(38) 6. Practical Aspects. Process mapping Process mapping. Example for the Influence of the process mapping on the communication in an application with cartesian neighbourhood MPI communication. Blue double lines - Inter-socket Red single lines - Inter-node Left: Ranks sequentially placed onto nodes Middle: Ranks placed in roundrobin-order onto nodes Right: Two-level decomposition. Taken from Georg Hager, Gabriele Jost, and Rolf Rabenseifner: Communication Characteristics and Hybrid MPI/OpenMP Parallel Programming on Clusters of Multi-core SMP Nodes. Proceedings of the Cray Users Group Conference 2009 (CUG 2009), Atlanta, GA, USA, May 4-7, 2009. 2012-08-30. 38. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(39) 6. Practical Aspects. Nonblocking and asynchronous communication Do not confuse nonblocking and asynchronous communication Further advantages of nonblocking communication ● ●. ●. Avoid deadlocks Reduced effort for process synchronisation (f.ex. eager protocol) Handling of multiple requests (f.ex. duplex communication). 1 double precision :: delay 2 integer :: count, req 3 count = 80000000 4 delay = 0.d0 5 6 do 7 call MPI_Barrier(MPI_COMM_WORLD, ierr) 8 if(rank.eq.0) then 9 t = MPI_Wtime() 10 call MPI_Irecv(buf, count, MPI_BYTE, 1, 0, & 11 MPI_COMM_WORLD, req, ierr) 12 call do_work(delay) 13 call MPI_Wait(req, status, ierr) 14 t = MPI_Wtime() - t 15 else 16 call MPI_Send(buf, count, MPI_BYTE, 0, 0, & 17 MPI_COMM_WORLD, ierr) 18 endif 19 write(*,*) ’Overall: ’,t,’ Delay: ’,delay 20 delay = delay + 1.d-2 21 if(delay.ge.2.d0) exit 22 enddo. Simple Benchmark to test if asynchronous MPI communication is possible (taken from [4] Hager, Wellein: Introduction to...) 2012-08-30. 39. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(40) 6. Practical Aspects. Optimization of I/O Example of a cluster file system (Lustre). Compute Nodes Switches. Disk servers MDS. 2012-08-30. 40. OSS. OSS. OSS. OSS. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(41) 6. Practical Aspects. Optimization of I/O (contd.) Program initialization, results, checkpoints Every user's activities influence all other users! How to do inefficient I/O? ●. ●. Read all data on one processor and distribute it with broadcast, vice versa collect all results on one node and save it there to a file. (produces a lot of idle time) Distribute all data in a huge number of small files and let every processor read one file (makes MDS the bottleneck and slows down all users). How to improve the I/O? ●. ●. Define an I/O processor for a group of processors ● Reduces load in the filesystem as well as the size of communication collectives Example: 20.000 processes where each of 200 I/O processes distribute the data to 99 other processes. 2012-08-30. 41. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(42) 7. Cray's High Speed Network. Cabling Images. PDC's Cray test system 2012-08-30. 42. PDC's Cray Lindgren system Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(43) 7. Cray's High Speed Network. Topology Types Class 0 ●. 1 row with up to 3 cabinets. Class 1 ●. 1 row, between 4 and 16 cabinets. Class 2 ●. 2 equal rows, between 16 and 48 cabinets. Class 3 ●. 2012-08-30. >2 equal rows, between 48 and 320 cabinets. 43. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(44) 7. Cray's High Speed Network. XE6 Compute Node. NIC. DIMM. CPU. Netlink Block. 48-port Router. DIMM. Gemini 1. CPU. Gemini 0. Network Mezzanine Card. NIC. CPU. CPU. DIMM. DIMM. Top: Gemini block diagram Bottom: Gemini external links (4.68/9.375 GB/sec) Right: XE6 Compute node. Z Node 3. Node 2. Node 1. Node 0. X Y. 2012-08-30. 44. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(45) 7. Cray's High Speed Network. Cable Connectors of a Cage Connections of a cage X3. Y: cable + mezzanine (blade). ●. Y1. X2. X1. Y0. X0. Z: cable + backplane (chassis) X: cable. ● ●. Y. Y. Z0. Z1. 31. 27. 23. 19. 15. 11. 07. 03. 30. 26. 22. 18. 14. 10. 06. 02. Z. Z 29. 25. 21. 17. 13. 09. 05. 01. 28. 24. 20. 16. 12. 09. 04. 00. Y. X3 Y. X. Y1. X2. X1. Y0. X0. Y. Z 2012-08-30. 45. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(46) 7. Cray's High Speed Network. Cabling - Class 0 (I). 2012-08-30. 46. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(47) 7. Cray's High Speed Network. Cabling - Class 0 (II). 2012-08-30. 47. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(48) 7. Cray's High Speed Network. Cabling - Class 1. 2012-08-30. 48. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(49) 7. Cray's High Speed Network. Cabling - Class 2. 2012-08-30. 49. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(50) 7. Cray's High Speed Network. Parallel MPI Pingpong 2. 4. 8. group size. Execution on Lindgren: 171 nodes 1024 processes (6 ppn). 2012-08-30. 50. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(51) 7. Cray's High Speed Network. Parallel MPI Pingpong. Execution on Lindgren: 171 nodes 4096 processes (24 ppn). 2012-08-30. 51. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(52) 7. Cray's High Speed Network. Parallel MPI Pingpong. For comparison execution on Ekman: 128 nodes 1024 processes (8 ppn). 2012-08-30. 52. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(53) Literature (I) 1). 2). Introduction to parallel computing / Ananth Grama ... [et al.]. Harlow, England ; New York : Addison-Wesley, 2003. ISBN: 0201648652 Parallel programming : for multicore and cluster Systems / Thomas Rauber and Gudula Rünger Berlin, Heidelberg: Springer, 2010. ISBN: 978-3-642-04817-3. 3) (Predecessor of 2) in German language and used for the lecture). Parallele Programmierung / Thomas Rauber ; Gudula Rünger Berlin ; Heidelberg ; New York : Springer, 2007. ISBN: 978-3-540-46549-2 2012-08-30. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(54) Literature (II) 4). Introduction to High Performance Computing for Scientists and Engineers / Georg Hager; Gerhard Wellein Boca Baton, London, New York : CRC Press, 2011. ISBN: 978-1-4398-1192-4. 5). High Performance Computing : Programming and Applications / John Levesque; Gene Wagenbreth Boca Baton, London, New York : CRC Press, 2011. ISBN: 978-1-4200-7705-6. 2012-08-30. Introd. to High Performance Computing - Interconnection Networks - M. Schliephake/PDC.
(55)
Related documents
Regressions include time-varying controls (income level, education dummies, percentage of new construction, increase in population density and average apartment
Capital flow volatility, financial deepening and capital market performance in low-income countries.
Based on the technique employed in the first model where the PVECM estimation was applied to establish the short- and long-run relationships among foreign remittance flow
Our results suggest that if effective control requires high ownership concentration and this is achieved at the cost of under-diversification of controlling shareholders then
The DRIVE consortium has enlisted top European autophagy experts (Figure 2), who have significantly contributed to our fundamental and clinical understanding of autophagy, and
Descriptively, recourse for wrongs does considerably better than cor- rective justice as a theory of tort law: corrective justice fails to ex- plain recovery by claimants who
For the DNA extraction of Tuber sp., from fruiting bodies or mycorrhized roots, all the evaluated protocols demonstrated effectiveness, nevertheless the protocol based
Virtual commercial environments operating in cyberspace, where information is the 'life blood' of business processes, and through which virtual value chains have emerged, can
The green and climate bonds market was launched in 2007 by the European Investment Bank (EIB) and the World Bank to demonstrate to private investors that funds were being applied
