Implementing the Hadoop Distributed File System Protocol on OneFS Jeff Hughes EMC Isilon

(1)

Implementing the Hadoop Distributed

File System Protocol on OneFS

Jeff Hughes EMC Isilon

(2)

Outline

 Hadoop Overview

 _{OneFS Overview}

 MapReduce + OneFS

 _{Details of isi_hdfs_d}

(3)

(4)

Apache Hadoop Project

 Two main components

 MapReduce

 Distributed File System (DFS)

The Apache Hadoop software library is a framework that allows for the distributed

processing of large data sets across clusters of computers using a simple programming model.

(5)

Hadoop: MapReduce

 MapReduce

 _{Distributed computation framework}

 Optimized for batch processing

 _{Typical I/O profile:}

5

DFS

(6)

(7)

Hadoop: HDFS Semantics

 Metadata/data server cluster architecture

 _{Cluster coherent namespace and data}

 Write-once-read-many access

 _{Single writer only}

 Can append to existing files

 _{Data mirrored 3x for resiliency}  Client exposed to data topology

 _{Block locations as part of file metadata}

7

http://www.snia.org/sites/default/files2/sdc_archives/2010_presentations/wednesday/DhrubaBorth akur-Hadoop_File_Systems.pdf

(8)

Hadoop: Why HDFS?

 Portability – All user space and OS independent

 _{Purpose-built – Primary workflow is MapReduce}

 Limited set of operations to implement

 _{Single software package}

 Fluid client/server protocol development

 _{Exposure of data topology}

(9)

(10)

OneFS: Architecture

Isilon IQ Storage

Layer Communication Infiniband Intracluster

Servers

Client/Application Layer Ethernet Layer

Servers

(11)

OneFS: OS

 Built from the ground up on FreeBSD

 _{File system is a loadable kernel module}

with VFS interface

 _{Supports POSIX syscalls locally}

 Protocol servers access /ifs paths

 FS Built for mixed namespace access

 Supports SMB, NFS, HTTP, etc.

(12)

OneFS: Semantics

 Symmetric cluster architecture

 _{Metadata distributed across all nodes}

 Tightly coupled group semantics

 _{Globally coherent file system access}

 Distributed lock manager

 _{Two-phase commit for all write operations}

(13)

(14)

MapReduce + OneFS: Architecture

OneFS Node NameNode DataNode OneFS Node NameNode DataNode OneFS Node NameNode DataNode OneFS Node NameNode DataNode Hadoop Node DFSClient 1) Request(“/file”) 2) Response

(block locations) 3) GetBlock(block)

OneFS runs a daemon that speaks NameNode and DataNode natively

(15)

MapReduce + OneFS: Benefits

 Easier integration to existing workflows

 _{First class multi-protocol access}

 Reduce ETL stages

 _{Increased disk efficiency}

 HDFS: 30% usable, OneFS: 80% usable

 _{Reduced data center footprint}

 More data management options

 _{Snapshots, site replication, etc.}

(16)

MapReduce + OneFS: Challenges

 Typical data path locality changes

 MapReduce ➝ HDFS acts like DAS

 MapReduce ➝ OneFS goes over the network

 _{Client/server compatibility and maintenance}

 OneFS and MapReduce clusters run different

software versions

(17)

MapReduce + OneFS: Mitigations

 1GbE < SATA controller < 10GbE

 _{Hadoop designed for 1GbE}

 10GbE prices dropping

 _{Denser storage == less nodes/networking}

 “Rack” locality limits cross-switch contention

17 Rack A Rack B Rack C

(18)

MapReduce + OneFS: Performance

DFS

Read Map Task Map Output Shuffle Reduce Task Write DFS

Typically ~100Mbit per task from HDFS

I/Os against temp vary considerable per job

More variable, still ~100Mbit per task to HDFS

 Performance bottleneck likely to be temp space

 _{Terasort example: 75% of I/Os against temp}

 Latency not much impact over HDFS large

(19)

(20)

HDFS Protocols

 Two TCP based protocols

 _{NameNode – metadata operations}

 DataNode – data transfers

 _{About 26 NameNode RPCs}

 Mostly use fully qualified paths

 _{POSIX-like file attrs (modebits, user/group)}

 Only 2 DataNode client operations

(21)

NameNode Request Example

Example getFileInfo(“/testfile”) request (think stat):

21 “/testfile”

Parameter type (string) Method name

(22)

NameNode Response Example

(23)

isi_hdfs_d

 Multi-threaded daemon runs on all nodes

 _{Services both NN and DN protocols}

 Translates RPCs to POSIX system calls

 _{Stateless, underlying FS handles coherency}

23 OneFS Node isi_hdfs_d Thread Request VFS

OneFS

Syscall Response

(24)

Example NameNode RPCs

 Most NameNode RPCs are straightforward

 _{setPermission()}→ chmod(…)

 setTimes() → utimes(…)

 _create()→ open(…, O_CREAT, …)

 Other RPCs need some creative interpretation

 _{recoverLease()/renewLease()}

 abandonBlock()

(25)

HDFS Data Path

 NN RPC: getBlockLocations(file)/addBlock(file)

 _{Returns list of LocatedBlocks}

25 LocatedBlock long offset Block long blkid long genStamp long numBytes DatanodeInfo[] ...  DFSClient connects to DN

 _{Chooses which DatanodeInfo}

based on locality

 _{Only Block structure passed to}

(26)

LocatedBlocks Translation

LocatedBlock long offset Block long blkid long genStamp long numBytes DatanodeInfo[] ...

Logical byte offset into the file Opaque to client, used by DN

Inode number Size of extent

Absolute byte offset

<IP:port> and rack info for

Specific to OneFS and isi_hdfs_d

(27)

Read Path Example

27 getBlockLocations()

LocatedBlocks

DN_OP_READ(Block)

Data stream

(28)

NameNode Connection Routing

 NameNode is configured as single URL

 _{Easy configuration:}

hdfs://log-server.isilon.com:8020/

 _{DNS round-robin to distribute across nodes}

 Metadata IOPs get spread out

 _{OneFS maintains cross-node consistency}

 IP Failover plus client retries for resiliency

(29)

DataNode References

 DataNode references returned by NameNode

 _{All OneFS DataNodes can access same data}

 Each LocatedBlocks for reads has 3 DN refs

 _{Round-robin across available nodes}

 Multiple refs lets client try other nodes before

coming back to the NameNode again

 Write path only 1 reference per logical block

 No need for client to replicate writes

(30)

Few Quirks

 User/group identities are strings

 _{OneFS natively stores UIDs or SIDs only}

 Requires name resolution on access

 _Locking!

 HDFS uses “leases” to restrict to single writer

 _{Implemented but no cross-protocol contention}

 Hadoop apps don’t expect files to move

(31)

It Works!

31

You see this across NFS:

And the same directory on HDFS:

(32)

Conclusions

 HDFS protocol can map to POSIX pretty easily

 _{Not all traditional shared storage is bad for}

Hadoop workflows

 _{Locality features worth preserving even without}

node locality as a possibility

(33)

33

Questions?

[email protected]