Parallel application development

Jan
Ciesko



HPC
Consultant


Microsoft






 

Microsoft
HPC
landscape




 

Introduction




 

Microsoft
HPC
Server
2008




 

Demo




 

Parallel
application
development




 

Microsoft
Parallel
Programming
initiative




 

MPI
/
OMP
/
TPL




 

Demos




 

Microsoft
HPC
landscape




 

Introduction




 

Microsoft
HPC
Server
2008




 

Demo




 

Parallel
application
development




 

Microsoft
Parallel
Programming
initiative




 

MPI
/
OMP
/
TPL




 

Demo


Riggid
body
–
fluid
interaction.
FEM.


Financial
services
and
computing,



parametric
sweep


Trend:


Observation:
HPC
is
becoming
a
decisive
factor
for
successful


businesses.




 

Reducing

application
development
complexity





 

Providing
development
tools




 

Supporting
parallelization
paradigms
and
applications
scenarios




 

Offering
libraries
(abstraction)




 

Providing
an
efficient
HPC
platform
usage
and
administration




 

Deployment





 

Integration
with
infrastructure
and
policies




 

Job
submission
and
scheduling




 

Diagnostics
and
fail‐over
redundancy




 

Providing
an
high
performance
computing
platform




 

Supporting
shared
/
distributed
memory
systems




 

Supporting
fast
interconnects





 

Enhancing
the
operating
system
for
high
performance





 

OS
+
HPC
Tools
+
Front
End
+
Back
End
+
Dev




 

Microsoft
HPC
landscape




 

Introduction




 

Microsoft
HPC
Server
2008




 

Demo




 

Parallel
application
development




 

Microsoft
Parallel
Programming
initiative




 

MPI
/
OMP
/
TPL




 

Demos


• Server
Cluster
in
WS2003
EE
(up
to
8
nodes)
 • Provides
availability
for
mission
critical
apps
(DB,
ERP,
CRM)
 High‐availability
or
fail‐over
clusters
 • Network
Load
Balancing
in
every
version
of
WS2003
(up
to
32
nodes)
 • Provides
high
availability
for
scale‐out
apps
(Web
servers)
 Load
balancing
clusters
 • Improve
performance
by
splitting
computational
task
across
many
different
nodes
 • No
imposed
node
limit
 High‐performance
computing
(HPC)
clusters
 • Closely
related
to
cluster
computing
but
operates
more
like
a

computing
utility
than
like
 a
single
computer
 • Support
heterogeneous
collections
of
computers
that
do
not
always
trust
each
other
 Grid
computing


 
New
System
Center
UI
  
PowerShell
for
CLI
Management
  High
Availability
for
Head
Nodes
  Windows
Deployment
Services
  Diagnostics/Reporting
  Support
for
Operations
Manager
  
Support
for
SOA
and
WCF
  
Granular
resource
scheduling
  
Improved
scalability
for
larger
 clusters
  
New
Job
scheduling
policies
  Interoperability
via
HPC
Profile
  
NetworkDirect
(RDMA)
for
MPI
  
Improved
Network
 Configuration
Wizard
  
Shared
Memory
MS‐MPI
for
 multi‐core
  
MS‐MPI
integrated
with
 Windows
Event
Tracing
  
Improved
iSCI
SAN
Support
in
 Win2008
  
Improved
Server
Message
 Block
(
SMB
v2)
  
New
3rd_{party
parallel
system} file
support
for
Windows
  
New
Memory
Cache
Vendors
 16


•

 

Support
for
larger
clusters


–  Create
new
designs
for
clusters
of
size,
 including

 “heterogeneous”
clusters
 –  Scale
deployment
and
administration
 technologies
 –  Provide
interfaces
for
those
accustomed
 to
*nix


•

 

Improve
interoperability
with
existing


IT
infrastructure


–  Interoperability
with
existing
job
 schedulers
 –  High
speed
file
I/O
through
native
 support
for
parallel
and
clustered
file
 systems


•

 

Broader
application
support


–  Simplify
the
integration
of
new
 applications
with
the
job
scheduler
 –  Addressing
needs
of
in‐house
and
open
 source
developers


•

 

Platform
Support


–  Built
for
Windows
Server
2008
 –  Cluster
nodes
with
different

hardware
/
 software
  

Priorities


  Comparable
with
hardware‐optimized
MPI
 stacks
 ▪  Focus
on
MPI‐Only
Solution
for
version
2
   Verbs‐based
design
for
close
fit
with
 native,
high‐perf
networking
interfaces
   Coordinated
w/
Win
Networking
team’s
 long‐term
plans
  

Implementation


  MS‐MPIv2
capable
of
4
networking
paths:

 ▪  Shared
Memory

 

between
processors
on
a
motherboard
 ▪  TCP/IP
Stack
(“normal”
Ethernet)
 ▪  Winsock
Direct
(and
SDP)

 

for
sockets‐based
RDMA
 ▪  New
RDMA
networking
interface
   HPC
team
partners
with
networking
IHVs
 to
develop/distribute
drivers
for
this
new
 interface
 User
 Mode
 Kernel
 Mode
 TCP/Ethernet

 Networking
 Kernel
By‐Pass
 MPI
App
 Socket‐Based
 App
 MS‐MPI
 Windows
Sockets

 

(Winsock
+
WSD)
 Networking
Hardware
 Networking
Hardware
 Networking
Hardware
 Networking
Hardware
 Networking
Hardware
_{Hardware
Driver} Networking
 Hardware
 Networking
 Hardware
 Mini‐port
 Driver
 TCP
 NDIS
 IP
 Networking
Hardware
 Networking
Hardware
 User
Mode
Access
Layer
 Networking
 Hardware
 Networking
 Hardware
 WinSock
 Direct
Provider
 Networking
 Hardware
 Networking
 Hardware
 NetworkDirect
 Provider
 RDMA
 Networking
 OS
 Component
 CCP
 Component
 IHV
 Component
 (ISV)
App


Private
Network


Public
Network


Highly
Available


Head
Node



WCF


Brokers


Head
node
 Failover
 Head
node


[…]


1.
User
submits
job.
 2.
Session
Manager
 assigns
WCF
Broker
 node
for
client
job
 3.
HN
Provides
WCF
 Broker
node
 5.
Requests
 4.
Client
connects
to
Broker
 and
submits
requests
 7.
Responses
return
to
client
 Compute
Nodes
 Workstation
 Workstation
 Workstation
 6.
Responses
 20
 Track
service
resource
usage
 Run
service
as
the
user
 Restart
upon
failure
 Write
Tracing
 Balance
the
requests
 Grow
&
shrink
service
pool
 Provide
WS
Interoperability
 Node
heatmaps
 Perfmon
 Event
logs
 Job
status
 Service
usage
 report
 Tracing
logs
 What
 Backend
 does
 What
admin
 sees
 What
user

 runs
 Control
 Path
 Data

 path


What
is
it?

 • A
draft
OGSA
(Open
Grid
Services
 Architectures)
interoperability
standard
 for
batch
job
scheduler
task
submission
 and
management
 • Based
on
web
services
standards
(HTTP,
 XML,
SOAP)

 What
is
its
value?
 • Enables
integration
of
HPC
applications
 executing
on
different
platforms
and
 schedulers
via
web
services
standards
 What’s
the
Status?
 • Passed
the
public
comment
period
 • Working
on
new
extensions
 Windows
Cluster
 Window
Center
 Windows
Center
 LSF

/
PBS
/
SGE
/
Condor
 Linux,
AIX,
Solaris

 HPUX,
Windows


demo


Loca%on
 Tukwila,
WA
 Hardware
–
Machines
 256
Dual‐CPU,
quad‐core
Intel
5320
 Clovertown
1.86GHz
processors
and
 8GB
RAM
 Hardware
–
Networking
 Private
&
Public:

Broadcom
GigE
 MPI:

Cisco
Infiniband
SDR
incl/
34
IB
 switches
in
leaf&node
configura%on
 Number
of
Compute
Nodes
 256
 Total
Number
of
Cores
 2048
 Total
Memory
 2+
TB
of
RAM
 Par%culars
of
for
current
Linpack
 Runs
 


Best
Linpack
ra%ng
so
far
 11.75
TeraFLOPS



 


Best
cluster
efficiency
so
far
 77.1%
 For
Comparison…
 Linpack
ra%ng
from
June2007
 Top500
run
(#106)
on
the
same
 hardware
 8.99
TeraFLOPS
 Cluster
efficiency
from
June2007
 Top500
run
(#106)
on
the
same
 hardware
 59%
 Typical
Top500
efficiency
for
 Clovertown
motherboards
w/
IB
 regardless
of
Opera%ng
System
 65‐77%


 (2
instances
of
79%)


30%
improvement
in


efficiency
on
the
same


hardware.


Release:



Nov
07
Beta
1


Summer
08
RTM


Less
than
2
hours
to
deploy




 

Microsoft
HPC
landscape




 

Introduction




 

Microsoft
HPC
Server
2008




 

Demo




 

Parallel
application
development




 

Microsoft
Parallel
Programming
initiative




 

MPI
/
OMP
/
TPL




 

Demos




 

Microsoft
HPC
landscape




 

Introduction




 

Microsoft
HPC
Server
2008




 

Demo




 

Parallel
application
development




 

Microsoft
Parallel
Programming
initiative




 

MPI
/
OMP
/
TPL




 

Demos


  Development
and
Parallel
debugging
in
Visual
Studio
   3rd_{party
Compilers,
Debuggers,
Runtimes
etc..
available}  

Emerging
Technologies
–
Parallel
Framework


   LINQ/PLINQ
–
natural
OO
language
for
SQL
queries
in
.NET
   C#
Futures
–
way
to
explicitly
make
loops
parallel
  

For
the
future:
Parallel
Computing
Initiative
(PCI)


  Triple
investment
with
a
new
engineering
team
  Compilers
Focused
on
common
tools
for
developing
multi‐core
codes
from
desktops
to
clusters




• Visual
Studio
 • Intel
C++
 • Gcc

 • PGI
Fortran
 • Intel
Fortran
 • Absoft
Fortran
 • Fujitsu

 Profilers
and
Tracers
 • PerfMon
 • ETW
(for
MS‐MPI)
 • VSPerf
/VSCover
 • CLRProfiler
 • Vampir
(Being
 ported
to
 Windows)
 • Intel
Collector/ Analyzer(Runs
on
 CCS
w
Intel
MPI)
 • Vtune
&
 CodeAnalyst
 • Marmot
(Being
 ported
to
 Debuggers
 • Visual
Studio
 • WinDbg
 • DDT
 Runtimes
and
 Libraries
 • MPI
 • OpenMP
 • C#
Futures
 • MPI.C++
and
 MPI.Net
 • PLINQ


• Standard
MPICH
+
Windows
related
enhancements
+
RDMA
(v2)
 • MPICH2
based
with
Windows
enhancements
(w
RDMA
in
CCSV2)
 MPI
 • Available
for
C++
and
C++/CLI
(OpenMP
for
managed/.Net
code!)
 OpenMP
 • Library
for
writing
concurrent
C#
programs:
Parallel.For,
etc,
using
generics
&
anonymous
delegates
(MORE)
 C#
Futures
 • Based
on
Boost.MPI
library:
Optimized,
natural
integration
of
MPI
w
C++
and
STL
 MPI.C++
and
 MPI.Net
 • (Parallel)
Language
Integrated
Query:
SQL
syntax
built
into
C#
 • Joins,
maps,
reductions,
sorts,
…;

being
extended
to
clusters
(MORE)
 Plinq
 • Employs
SOA
for
load
balancing
a
web‐service
like
interactive
app
on
a
cluster
 App
Integration
 • Parametric
Sweeps:
embarassingly
parallel
jobs
 • Updated
interface:
ease
of
use


Job
Scheduler



• Integrated
support
for
Data
Parallel
programming
in
C#;
rich
syntax
(research
compiler)
 • Offload
to
generic
GPGPU’s
 • Various
3rd_{party
solutions
that
integrate
into
VS}

GPGPU






 

Microsoft
HPC
landscape




 

Introduction




 

Microsoft
HPC
Server
2008




 

Demo




 

Parallel
application
development




 

Microsoft
Parallel
Programming
initiative




 

MPI
/
OMP
/
TPL




 

Demos


using
System;



using
MPI;



class
MPIHello



{




 static
void
Main(string[]
args)




 {




 
 using
(new
MPI.Environment(ref
args))




 
 {




 
 


//
MPI
program
goes
here!




 
 }



 
}



}


using
System;
 using
MPI;
 class
Ring
 {
 



static
void
Main(string[]
args)
 



{
 







using
(new
MPI.Environment(ref
args))
 







{
 





Communicator
comm
=
Communicator.world;
 





if
(comm.Rank
==
0)
 





{
 





//
program
for
rank
0
 





}
 





else
//
not
rank
0
 





{
 





//
program
for
all
other
ranks
 





}
 







}
 



}
 }




 

Microsoft
HPC
landscape




 

Introduction




 

Microsoft
HPC
Server
2008




 

Demo




 

Parallel
application
development




 

Microsoft
Parallel
Programming
initiative




 

MPI
/
OMP
/
TPL




 

Demos




 

Technical
Preview
–
Private
Release




 

Beta
1
–
Publicly
Available
now!




 

Beta
2
–
Coincides
with
Windows
Server
2008


RTM




 

RTM
–
90
to
120
days
after
Windows
RTM


Nov
2007


Tech


Preview


Beta
1


Beta
2


RTM


Aug
2007


Early



2008


Summer


2008


Technical
Communities,
Webcasts,
Blogs,
Chats
&
User
Groups
 http://windowshpc.net/
 http://blogs.msdn.com/excel/

 http://blogs.msdn.com/cumgranosalis/

 Microsoft
Developer
Network
(MSDN)
&
TechNet
 http://msdn2.microsoft.com/en‐us/office/aa905419.aspx


 Trial
Software
and
Virtual
Labs
 http://www.microsoft.com/technet/ccs/evaluation/trial/default.mspx
 http://office.microsoft.com/en‐us/sharepointserver/FX100492001033.aspx
 Product
Information
 http://office.microsoft.com/excell

 http://www.microsoft.com/office/sharepointserver

 http://www.microsoft.com/windowsserver2003/ccs

 http://msdn.microsoft.com/office/server/moss
 http://msdn2.microsoft.com/en‐us/office/bb203828.aspx