Decentralized Deduplication in SAN Cluster File Systems

(1)

Decentralized Deduplication

in SAN Cluster File Systems

Austin T. Clements∗

Irfan Ahmad Murali Vilayannur Jinyuan Li

VMware, Inc. ∗_{MIT CSAIL}

(2)

Storage Area Networks

1.3 TB

(3)

Storage Area Networks

1.3 TB

(4)

Storage Area Networks

1.3 TB

(5)

Storage Area Networks

1.3 TB

(6)

Storage Area Networks

1.3 TB

(7)

Storage Area Networks

1.3 TB 237 GB

(8)

Storage Area Networks

1.3 TB 237 GB

5X reduction

(9)

Deduplication

(10)

Deduplication

(11)

Deduplication

(12)

Deduplication

(13)

Deduplication

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9..

(14)

Deduplication

(15)

Deduplication

(16)

Deduplication

(17)

The Problem

Slow— Lots of IO in the write path. Can’t cache the index.

Very Slow— Writes require allocation and thus coordination. No hope of disk locality.

Hopelessly Slow— Multi-host lock contention on shared index.

(18)

The Problem

(19)

The Problem

(20)

Three-Stage Deduplication

Decentralized Deduplication

Out-of-band deduplication of live, primary storage Process duplicates efficiently, in large batches Minimize contention on the index

Resilient to stale index information

Unique blocks remain mutable and sequential

=⇒ No overhead for blocks that don’t benefit from deduplication

(21)

Three-Stage Deduplication

DeDe

(22)

Three-Stage Deduplication

DeDe

(23)

Three-Stage Deduplication

DeDe

(24)

Three-Stage Deduplication

DeDe

(25)

Three-Stage Deduplication

DeDe

(26)

Three-Stage Deduplication

DeDe

(27)

Three-Stage Deduplication

DeDe

(28)

Three-Stage Deduplication

DeDe f4bea9.. f2a4d2.. 0e7a26.. 15ba2b.. d5e341.. bc6887.. ab7373.. 4ee207.. 9b1f28.. 9b575b.. 94c9c9.. 288bc7..

(29)

Three-Stage Deduplication

DeDe f4bea9.. f2a4d2.. 0e7a26.. 15ba2b.. d5e341.. bc6887.. ab7373.. 4ee207.. 9b1f28.. 9b575b.. 94c9c9.. 288bc7..

(30)

Three-Stage Deduplication

DeDe 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. f4bea9.. f2a4d2.. 0e7a26.. 15ba2b.. d5e341.. bc6887.. ab7373.. 4ee207.. 9b1f28.. 9b575b.. 94c9c9.. 288bc7..

(31)

Three-Stage Deduplication

DeDe 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. f4bea9.. f2a4d2.. 0e7a26.. 15ba2b.. d5e341.. bc6887.. ab7373.. 4ee207.. 9b1f28.. 9b575b.. 94c9c9.. 288bc7..

(32)

Three-Stage Deduplication

DeDe 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. ab7373.. 4ee207.. 9b1f28.. 9b575b.. 94c9c9.. 288bc7..

(33)

Three-Stage Deduplication

(34)

Three-Stage Deduplication

(35)

Three-Stage Deduplication

DeDe 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9..

(36)

Three-Stage Deduplication

DeDe 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9..

(37)

Three-Stage Deduplication

DeDe 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. ? ? ? ?

(38)

Three-Stage Deduplication

(39)

Three-Stage Deduplication

(40)

Implementation

(41)

Implementation

(42)

Implementation

(43)

Implementation

(44)

Implementation

(45)

VMFS

Compare-and-share

(46)

VMFS

Compare-and-share

?

(47)

VMFS

Compare-and-share

(48)

VMFS

Compare-and-share

DeDe finds duplicates. VMFS eliminates them.

(49)

Write Monitor

A lightweight kernel module monitors writes, computes hashes It buffers the write log in userspace before writing it to disk Safe to buffer the log because index is resilient

150 MB of regular writes → 1 MB sequential log write

(50)

Write Monitor

A lightweight kernel module monitors writes, computes hashes

It buffers the write log in userspace before writing it to disk Safe to buffer the log because index is resilient

(51)

Write Monitor

A lightweight kernel module monitors writes, computes hashes It buffers the write log in userspace before writing it to disk

Safe to buffer the log because index is resilient

(52)

Write Monitor

A lightweight kernel module monitors writes, computes hashes It buffers the write log in userspace before writing it to disk

Safe to buffer the log because index is resilient

(53)

Write Monitor

(54)

Write Monitor

(55)

The Index

Map from hashes to block locators, list sorted by hash

Unique blocks are located in files and remain mutable A virtual arena stores COW references to all shared blocks

(56)

The Index

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared

Map from hashes to block locators, list sorted by hash

Unique blocks are located in files and remain mutable A virtual arena stores COW references to all shared blocks

(57)

The Index

Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable

A virtual arena stores COW references to all shared blocks

(58)

The Index

(59)

The Index

(60)

The Index

Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable A virtual arena stores COW references to all shared blocks

(61)

The Index

(62)

The Index

(63)

Indexing and Duplicate Elimination

(64)

Indexing and Duplicate Elimination

(65)

Indexing and Duplicate Elimination

(66)

Indexing and Duplicate Elimination

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared 12067c.. 6cd412.. c277d6..

(67)

Indexing and Duplicate Elimination

(68)

Indexing and Duplicate Elimination

(69)

Indexing and Duplicate Elimination

(70)

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared 6cd412.. c277d6..

(71)

Indexing and Duplicate Elimination

(72)

Indexing and Duplicate Elimination

(73)

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared c277d6..

(74)

Indexing and Duplicate Elimination

(75)

Indexing and Duplicate Elimination

(76)

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared

?

c277d6..

(77)

Indexing and Duplicate Elimination

(78)

Indexing and Duplicate Elimination

(79)

Indexing and Duplicate Elimination

(80)

Indexing and Duplicate Elimination

(81)

Indexing and Duplicate Elimination

(82)

Indexing and Duplicate Elimination

(83)

Phew.

(84)

Evaluation

How much space does DeDe save? How much overhead does DeDe introduce?

How fast can DeDe deduplicate?

(85)

Evaluation

How much space does DeDe save?

How much overhead does DeDe introduce? How fast can DeDe deduplicate?

(86)

Evaluation

(87)

Evaluation

(88)

Space Savings: VDI Cluster

Corporate Virtual Desktop Infrastructure cluster Desktop XP VM’s

6–12 months of active use Originally cloned from small number of base images

...                            113 VM’s    1.9 TB 1.3 TB

(89)

Space Savings: VDI Cluster

...                            113 VM’s    1.9 TB 1.3 TB

(90)

Space Savings: VDI Cluster

...                            113 VM’s    1.9 TB 1.3 TB

(91)

Space Savings: VDI Cluster

1.3 TB

(92)

Space Savings: VDI Cluster

1.3 TB 237 GB

(93)

Space Savings: VDI Cluster

1.3 TB 237 GB 173 GB unique 61 GB shared

(94)

Space Savings: VDI Cluster

1.3 TB 237 GB 173 GB unique 61 GB shared 2.7 GB

(95)

Space Savings: VDI Cluster

1.3 TB 237 GB 173 GB unique 61 GB shared 2.7 GB 1.3 GB index file 194 MB v. arena 1.1 GB FS metadata

(96)

Runtime Effects

Write monitoring Disk array caching

EMC CLARiiON CX3-40

(97)

Runtime Effects

EMC CLARiiON CX3-40

(98)

Runtime Effects

EMC CLARiiON CX3-40

(99)

Runtime Overhead: Write Monitoring

Worst-case benchmark: 100% sequential write IO, No computation

Baseline Write Monitor

CPU 33% 220% Bandwidth (MB/s) 233 233 Latency (ms) 8.6 8.6 (See paper for database application benchmark)

(100)

Runtime Overhead: Write Monitoring

(101)

Runtime Overhead: Write Monitoring

CPU 33% 220% Bandwidth (MB/s) 233 233 Latency (ms) 8.6 8.6

(See paper for database application benchmark)

(102)

Runtime Overhead: Write Monitoring

(103)

Runtime Gains: Disk Array Caching

Reduced storage footprint Better caching Less IO

(104)

Runtime Gains: Disk Array Caching

0 10 20 30 40 50 60 70 80 90 2 4 6 8 10 12 14 16 18 20

Average boot time (secs)

# VMs booting concurrently VDI "boot storm"

(105)

Runtime Gains: Disk Array Caching

0 10 20 30 40 50 60 70 80 90 2 4 6 8 10 12 14 16 18 20

# VMs booting concurrently VDI "boot storm" Full copies

(106)

Runtime Gains: Disk Array Caching

0 10 20 30 40 50 60 70 80 90 2 4 6 8 10 12 14 16 18 20

# VMs booting concurrently VDI "boot storm" Full copies

Deduplicated

(107)

Out-of-band Deduplication Rate

Index scan 6.6 GB/sec COW sharing 2.6 MB/sec (It’s a prototype!)

Virtually no cost for unique blocks

9 GB ofnew shared blocks per hour

(And provisioning can be special-cased)

(108)

Out-of-band Deduplication Rate

Index scan 6.6 GB/sec

COW sharing

2.6 MB/sec (It’s a prototype!)

(109)

Out-of-band Deduplication Rate

COW sharing 2.6 MB/sec

(It’s a prototype!)

(110)

Out-of-band Deduplication Rate

(111)

Out-of-band Deduplication Rate

(112)

Related Work

Centralized archival Venti

Data Domain Foundation

Centralized primary storage NetApp ASIS

Microsoft Single Instance Store Distributed

Farsite

SAN with Coordinator DDE

(113)

Conclusion

Decentralized, out-of-band, live file system deduplication.

Deduplication is effective.

Deduplication is hard.

Three-stage deduplication has only modest performance overhead. Thank you.

(114)

Conclusion

(115)

Conclusion

(116)

Conclusion

Three-stage deduplication has only modest performance overhead.

Thank you.

(117)