Co-ordinated by
Storage Solutions
Jeffrey van der Hoeven (National Library of the Netherlands)
Outline
• Introduction
• Two cases: a library and a research institute
• Cloud storage: hot or not…
• Costs
Partners involved
• Libraries:
– British Library, Deutsche Nationalbibliothek, National Library
of the Netherlands (KB)
• Data centres:
– CINES, CSC – IT Center for Science
• Research organisations:
– European Space Agency, Fondazione Rinascimento Digitale,
CINI
• ICT companies:
Two storage cases
• Case 1: National Library of the Netherlands (KB)
– Deposit library for all published content in and about the
Netherlands.
– Large-scale digitization of books, newspapers, magazines.
– Safe place for national & international scientific publications.
Two storage cases
• Case 2: CERN
– European Organization for Nuclear Research, is one of the
world’s largest and most respected centres for scientific research.
– Known for its Large Hadron Collider (LHC) recently
discovering the Higgs-particle.
– Distributed storage solution for LHC data
Case 1: storage solution at KB
• In-house data centre for primary storage and computing
• Off-site location for back-up
• Hosting 230 servers (80 physical / 150 virtual)
• Managing 550 TB of storage space
Storage tiers
Books
Newspapers
Hierarchical Storage Management
Bronze Steel Silver Gold Magazines Webarchive Journals Parliament
Very fast, very expensive
Used: indexing, databases SAN with HiPerf SAS disks
Fast, expensive
Used: web hosting, processing SAN with HiCap SAS disks
Slow (45 sec), sustainable
Used: long-term archiving Disk-based NAS with WORM
Very slow (> 45 sec)
Used: back-up & restore LTO tape
Off-site
Back-up
Storage strategy
Storage on-site
Bronze Steel Silver Gold
Storage management
Platinum Bronze
Pre-ingest Digital processing Access
Shared file system File system
Case 2: storage solution at CERN
• Distributed data centres for primary storage and computing
• CERN Advanced STORage manager (CASTOR) is a
hierarchical storage management system developed at CERN for physics data files.
• Files can be: stored, listed, retrieved, accessed remotely
• Combination of disk and tape
• Targeted to large files (> 1 GB)
• Used for raw data and simulation files
• Retrieval time: around 4 hours
Important factors in long-term storage
• Volume (size and number of files)
• Type of data (structured / unstructured)
• Growth rate
• Retention (long term or temporary)
• Availability
Storage
in the cloud
Cloud basics:
•
IaaS: Infrastructure as a Service
storage/compu:ng op afstand
•
PaaS: Pla<orm as a Service
computer pla<orm op afstand
•
SaaS: So=ware as a Service
Benefits of cloud storage
•
Scalable
•
Availability
•
Pay per TB per month
•
No need for own ICT infrastructure
However…
•
Is it sustainable?
•
Who is responsible for the data?
•
Which jurisdic:on is applied?
•
What if I want to migrate to other cloud?
•
No money? No data!
•
Conclusion: be aware of this if you consider cloud for
long-‐term storage.
Read on:
Many ways of computing TCO for storage
A personal favorite (KB):
• Four Principles for Reducing Total Cost of Ownership (2011
Hitachi) *
• Describes 34 storage cost components
• Agnostic for type of storage system
* Available at:
Cloud storage costs vs in-house
Can we still afford it?
• Recent insights *:
– Disk storage is becoming more popular in archiving.
– Kryder’s law seems to fail, as disk storage density seems not
to keep up the pace of a yearly 30-40% increase of storage density.
– Physical limits of hard disk drive seems reached.
– Monopoly of hard disk producers Seagate and Western Digital
is risky as prices might go up, especially in case of shortage.
– Conclusion: storage costs are becoming a bottleneck for
long-term preservation.
* David Rosenthal blog post, available at:
We value your input!
• Short APARSEN survey on storage solutions
• Topics addressed such as:
– Mission, preservation policy, legal status
– Type of data, volume, number of items, ownership
– Access to data
– Storage solution, in-house/outsourced
– Storage storage management (tiering)
– Back-up or replication (or both)
– Costs
Survey: how to participate?
• Survey will be out in February / March 2013
• Go to http://www.alliancepermanentaccess.org/
• And browse to WP23 Storage Solutions
• Start the online survey
Statements for discussion…
• As storage will become the largest cost component in the
long run, we need to be picky about what to preserve.
