• No results found

UFTP High-performance data transfer for UNICORE

N/A
N/A
Protected

Academic year: 2021

Share "UFTP High-performance data transfer for UNICORE"

Copied!
16
0
0

Loading.... (view fulltext now)

Full text

(1)

Dr. Bernd Schuller, Tim Pohlmann

Federated Systems and Data division Jülich Supercomputer Centre

Forschungszentrum Jülich GmbH July 8, 2011

UNICORE Summit, Toruń

M it g lie d d e r H e lm ho lt z-G e m e in sc h a ft

UFTP

(2)

July 8, 2011, UNICORE Summit, Toruń Slide 2

Outline

 Filetransfer in UNICORE  UFTP ■ Principles ■ Deployment ■ Examples  Outlook

(3)

July 8, 2011, UNICORE Summit, Toruń Slide 3

Storages, Jobs and Data

Local Filespace Client HOME, TMP, ... Job directories UNICORE Server Client Import & Export Server-to-server data movement UNICORE Server HOME, TMP, ... HTTP GridFTP FTP ... Job data staging UNICORE Server

(4)

July 8, 2011, UNICORE Summit, Toruń

Data flows using the BFT

data transfer

scientific clients and applications command-line client Eclipse-based client Other clients

access to data and management of computational jobs

Local RMS (e.g. Torque, LL, LSF, etc.) Local RMS (e.g. Torque, LL, LSF, etc.)

authentication, firewall transversal

Gateway

UNICORE services

UNICORE basic services UNICORE basic services

Target System Interface Target System Interface

BFT import/export requires three socket connections BFT staging requires four socket connections

(5)

July 8, 2011, UNICORE Summit, Toruń

Ideal data flow

scientific clients and applications command-line client Eclipse-based client Other clients

access to data and management of computational jobs

Local RMS (e.g. Torque, LL, LSF, etc.) Local RMS (e.g. Torque, LL, LSF, etc.)

Target System Interface Target System Interface

Import/export without extra socket connections staging without extra socket connections

(6)

July 8, 2011, UNICORE Summit, Toruń Slide 6

 Firewall!

■ Direct connections from the outside to the TSI login node are

usually not allowed

■ Statically opening ports (or worse, port ranges) is a security risk

 Port opening technique is required

■ UDP based hole punching (like Skype), but UDP is not directly

suited for file transfer

TCP based: passive FTP is widely understood, but considered

insecure

(7)

July 8, 2011, UNICORE Summit, Toruń Slide 7

Basic idea: use passive FTP to open ports

Client Server FTP port 1. „PASV“ data port 5432 Firewall „5432“ 2. open 5432 for Client 3. connect to port 5432 4. close control connection

5. close 5432

(8)

July 8, 2011, UNICORE Summit, Toruń Slide 8

 FTP by itself is insecure:

■ Users log in using username/password

 UNICORE provides a highly secure channel from client to server which

is used for additional security measures:

■ File transfers are always initiated via UNICORE

■ Client must authenticate using a „secret“ that is exchanged via

UNICORE

 Requires an secure „command port“ in addition to the FTP port

(9)

July 8, 2011, UNICORE Summit, Toruń

UFTP deployment

UFTP support since version 6.4.1 command-line client Eclipse-based client

Local RMS (e.g. Torque, LL, LSF, etc.)

UFTPD

UNICORE/X

server Cluster login node

TSI UNICORE/X

UFTP support since version 6.4.1

commands other UNICORE

servers

pseudo FTP

UFTP support since version 6.4.1

listen

(10)

Slide 10

File transfer using UFTP

Client 4. Upload/download File system uftpd uftp-client 1. Create transfer UNICORE/X Firewall 2. init

Init information sent via command channel includes:

- client IP - file name - authn secret

- optional encryption key

3. get filetransfer properties (uftp host and port, # of streams)

(11)

July 8, 2011, UNICORE Summit, Toruń Slide 11

 uftpd server runs with root privileges (because it needs to access files from all

users)

■ Switch effective user/group ID before file access

 Sending commands via the Command channel allows local users to

read/write files under any user ID

■ Command port not accessible outside the firewall ■ Protect it using client authenticated SSL and ACL file

 Attacker might connect to the newly opened sockets on the uftpd server ■ Client IP is checked, and a secret key is required for authentication  Data channels might be sniffed

■ Optional symmetric encryption (64 bit key, blowfish algorithm)

(12)

July 8, 2011, UNICORE Summit, Toruń Slide 12

Using UFTP from UCC

uftp.client.host=localhost uftp.streams=2

uftp.encryption=false

optionally add UFTP related preferences …

>ucc put­file ­s /home/... ­t https://... ­P UFTP Specify the protocol in file operations

{

 Imports: [

   { From: “u6://...?protocol=UFTP“, To: … },  ],

}

(13)

July 8, 2011, UNICORE Summit, Toruń Slide 13 0 50 100 150 200 250 300 350 400 0 20 40 60 80 100 120 140

UFTP performance: example

UFTP UFTP encrypted BFT File size (MB) T ra ns fe r ra te ( M B /s ec )

Localhost system, Core 2, 4GB memory UNICORE 6.4.1 with Perl TSI

UFTP 1.0.0, 2 streams

(14)

July 8, 2011, UNICORE Summit, Toruń Slide 14

 UNICORE FTP (yes, think of "Grid- (rather Globus-)FTP“)

Multiple parallel TCP connections per data transfer

Client-Server, Server-Server

Secure, simple, easy to deploy and operate

Dynamically open ports in the firewall using the „FTP“

protocol

Platform independent (Java)

 Widely available with UNICORE version 6.4.1

as rpm, deb, tar.gz thanks to pac(k)man

(15)

July 8, 2011, UNICORE Summit, Toruń Slide 15

 Deployment in realistic environments (DEISA/PRACE and others)

 URC support (coming soon!)

 Workflow system support (current version is still 6.3.x)

 Performance testing

 Enhancements

■ Support multi-homed servers

■ Experiment with buffer sizes

 Other uses of the „FTP trick“ not related to data transfer? E.g. when

setting up (cloudy, virtual, …) systems dynamically?

(16)

July 8, 2011, UNICORE Summit, Toruń Slide 16

 Thanks for early testing and feedback on UFTP

Krzysztof Benedyczak

Michael Rambadt, Michael Stephan, Björn Hagemeier

References

Related documents

A diferencia de la mayoría de los ejecutivos (que establecen redes de contacto para tener acceso a recursos, para promocionarse a sí mismos o a sus compañías, o para impulsar

Aradial Proprietary and Confidential 34 IP Network PSTN PSTN Media Gateway Media Gateway SoftSwitch SIP Application Server (w/Internal IVR) Aradial Prepaid /

Sophomores Welcome Week Begins JR/SR Welcome Week Begins Welcome Week

If two countries with identical technologies but di ff erent labor supply were left in portfolio autarky, the representative worker in every country would attain the same

While the ability to scale capacity on-demand and provide broadly available elasticity is a benefit of cloud computing and the future direction of the public cloud, Oracle believes

A comparison made of body mass and body measurements of European bison living nowadays in Bia ³ owie ¿ a Forest with earlier data form the same locality suggest that high inbreeding

If the law was changed to allow the local intermediate districts (Boards of Cooperative Education Services) or BOCES to hire and assign teachers to different districts as needed,