Software Entwicklungen für das LSDF Datenmanagement

(1)

Institute for Data Processing and Electronics Steinbuch Centre for Computing

Software Entwicklungen für das LSDF Datenmanagement

Rainer Stotzka,

V. Hartmann, T. Jejkal,, P. Neuberger,

S. Ochsenreither, F. Rindone, T. Schmidt, H. Pasic

J. van Wezel, A. Garcia,

(2)

Institute for Data Processing and Electronics

2

Access to Data Infrastructures

KIT

LSDF

Virtual Research Communities share resources across borders (computing centers, countries): • Computing • Storage • Networking • Facilities • Services, etc. NEW: • Data as a Service

(3)

Requirements

Storage

 Availability and reliability (24/7)

 Scalability (5 - 10 PB/a)

 Sustainability (>> 10 a)

 Performance and throughput (> 1 TB/h per application)

Collaborative data networks

 Distribution (worldwide)

 Accessibility (multiple protocols: Grid, Cloud, Web, …)

 Security (X.509 certificates)

Tools and applications (software)

 Flexibility (multiple communities with a huge variety

of requirements)

 Programming interfaces (API)

(4)

4

LSDF objectives

Dedicated for science data ExaByte scale data

To archive data, long term sustainability (10 yrs. – ?)

To enable scientists to gain better scientific results by providing Data intensive analysis

Added value services for data intensive processing To provide high performance access, high throughput “Barrier free” access (easy-to-use)

Sustainability and interoperability

“Guidelines”:

• PARADE White Paper (2009): Strategy for a European Data Infrastructure

• ESFRI Data Management Task Force (2009): e-IRG Report on Data Management

• OAIS (2002): Reference Model for an Open Archival Information System

• High Level Experts Group (2010): Riding the Wave – European Commission Report on Scientific Data

• HLEG-SD (2010?): Note on Data Services infrastructure

(5)

Software Development

Development of software, technologies and algorithms

ADALAPI DataBrowser Meta data

Data intensive applications

Development of services to support scientific communities Scientific experiments, applications, communities Infrastructure LSDF LSDF Software and Service Development ADALAPI DataBrowser Workflow Meta data

(6)

6

ADALAPI

Abstract Data Access Layer

Application Programming Interface • Java class library

• Seamless application access to LSDF • Independent of transfer

protocol and location

• Protocols and filesystems

• local files, • gsiftp • sftp • http(s) • hdfs • Authentifikation: X.509 certificates, user/passwd • Performance

up to 85 MB/s, 1 GE, gsiftp LSDF Storage Infrastructure

Applications Tools DataBrowser Scientific exp. DAQ … Grid Workstations Client software Visualization … Cloud ADALAPI

(7)

DataBrowser

API: Data and meta data organization

GUI: File, data and project explorer

Functions:

• Data management • Queries in meta data cataloges

• Up-/Download • Control of

data analysis + vis. workflows • Easy-to-use • Extensible • World-wide access • Stable DataBrowser

(8)

8

(9)

Why is meta data necessary?

Meta data describe the contents of data

Everybody uses meta data:

File name and extension

(e.g.

rainer.jpg, budget.xls, Readme.doc

)

Location

(e.g.

/…/EU-projects/2010/Fishy/budget.xls

)

Personal know-how



Sufficient for small file systems

Have you ever tried to locate a file or info-somewhere-in-a-file-system

15 years old ?

in the file system of a colleague ?

in a 100 PetaByte file system ?

(10)

10

Model of the LSDF meta data management

Idea

:

Clear separation between

Data (files),

Logical File Catalog DB Logical File Catalog DB Logical File Catalog DB Logical File Catalog DB Logical File Catalogs DB Storage Meta data

(11)

Model of the LSDF meta data management

Idea

:

Clear separation between

Data (files),

Data organization (directory structure)

Logical File Catalog DB Logical File Catalog DB Logical File Catalog DB Logical File Catalog DB Logical File Catalogs Logical Directory Catalog DB DB My project

dir dir dir

dir dir

(12)

12

Model of the LSDF meta data management

Idea

:

Clear separation between

Data (files),

Data organization (directory structure)

and

Associated meta data

Logical File Catalog DB Logical File Catalog DB Logical File Catalog DB Logical File Catalog DB Logical File Catalogs Logical Directory Catalog Logical Project Catalog DB DB DB • name • owners • access rights • date • community • (sub)subcommunity • measurement type • device, instrument • …

Meta data structure depends on project, instruments, time, …

(13)

Hierarchical Catalog System (Repository)

LFN  Physical File Name

Logical File

Catalog DB

Logical File

Catalog DB

Logical File

Catalog DB

Logical File

Catalog DB

Logical File Catalogs Computing Storage LDN  LDN, LFN Logical Directory Catalog LPN  LDN, meta data Logical Project Catalog DB DB DB Meta data scheme repository Zebrafish II ANKA BL1 Material research Zebrafish I Digital objects in Arts and Humanities

APIs and Tools

Catalogs LSDF Systems Sustainable Easily extensible Independent of data formats Enhanced performance: distribution of access Safety by redundancy Use of open

standards Generic file tree

(14)

14

Additional Data Services

How do I insert a new scientific project ?

 Data and meta data organization experts

for projects with specific needs

 Generic meta data format for simple file trees

How do I transfer my data to a different location? Do I loose my meta data?

 Import-export to standard data and meta data formats

 Archive-in-a-box

(Web installer or DVD, zip-archive, etc.)

(15)

Community Services

Complex image analysis chain:

3D image stack, time series,

Leica Image Format data set size:

100 GB Transfer to LSDF Automatic data conversion to RAW Image processing and analysis Re sul ts LSDF storage and online processing Offline processing Computing Storage Computing DataBrowser Workflow Meta data

(16)

16

Data Intensive Computing

1 projection

36 projections

2.8 TB read, 1.7 TB write, rotations and projections, 2 h

Workflow

Visualization of huge data sets:

Maximum projection, arbitrary viewpoint

HeadNode

- Job preparation and distribution Computing Nodes

- Load data, compute rotation and projection, write results

(17)

Scientific communities

Systems biology (ITG, BioQuant, Immunogenetics) Vertebrate development studies and

Deconvolution (5000 data sets  <180 min.) Synchroton facilities and beamlines

ANKA data storage

HGF Programme Photon-Neutron-Ion “High Data Rate Initiative” Climate research

Material research Arts and humanities

(18)

18

Conclusions

LSDF is a powerful structure

 more than data storage and cluster computing

Design for future requirements  ExaByte storage + interactivity • R&D in progress

LSDF offers

Sustainability and safety

Flexibility for future requirements Support

Interactivity

Software and tools

Community-specific services