MapReduce

MapReduce [162] is a programming model for distributed data processing, which can be used for writing in any language, it can be writing in any computer lan- guage, including Ruby, Java, Python and C. It works by distributing tasks across multiple machines through the use of a job scheduler, which is performed by the master machine. Each slave machine then processes the data stored on it. MapRe- duce consists of two components: Job Tracker, which resides in the master controls and monitors the distribution of task. Task Tracker, which resides in the slave and processes each assigned task, sending its status to Job Tracker. For the par- allelisation process, MapReduce breaks the processing into 2 phases that can be executed in parallel on multiple machines[150]:

• Map function: This applies to every input record producing intermediate key-value pair, which are then stored on a local disk ready to be transferred to machines where a reducer is assigned to process the intermediate output.

• Reduce function: This merges the intermediate results from the Map phase and produces a final output result, which is stored in HDFS.

Each phase has key-value pairs as an input and output that a programmer specifies by map and reduce tasks. All jobs are executed on slave nodes as a map task or reduce task.

2.3.2.1 MapReduce Workflow

MapReduce applications that need to be run in Hadoop are called MapReduce jobs. In order to begin data processing, a client application submits a MapRe- duce job to Job Tracker(master) as a java code. Job Tracker communicates with Namenode (master) to find which Datanodes (slaves) contains blocks of input data.

The Job Tracker divides each MapReduce job into a set of tasks called map or reduce. Task Tracker running on those machines is then scheduled with the java code required to execute map function on local data. Several map and reduce tasks are running concurrently on each slave. The number of map slots and reduce slots are configured, which are dependent on the number of processors available in nodes to overlap computation and I/O. If all available slots are occupied, pending tasks must wait until some slots are freed up.

When the map task is complete, each machine stores the output result call intermediate data in its local temporary storage. It then sends the data over the network to a machine running reduce task for final computation. The communic- ation between reducer and mapper happens through a TCP/IP protocol. There

are cases when data is not stored locally, such as when new nodes are added or when the node fails and the task is assigned to other node. In both cases, the new data node communicates with name node to be directed to nodes that have copies of the data; it then copies the data to local storage.

The following diagram explains the MapReduce programing model for the se- quential phases (map and reduce) that the MapReduce data framework follows when executing a job, see figure 2.13 for illustration:

Figure 2.13: MapReduce data flow framework when executing a job

1. Map Phase:

• RecordReader: This reads files from HDFS or any storage specified by the programmer; all data is then transformed into key-value pairs where the key is a unique id and the value is the corresponding data in bytes. It is then submitted to InputFormat in the form shown below: map(K1, V1)→list(K2,V2)

• One InputFormat: Multiple types of key-value pairs provided by RecordReaders are accepted, all key-value pairs are combined and sub- mitted to Mappers in Inputsplit form.

• Mapper: Key-value pairs are generated through Inputsplit, with each node running one map task and run it in parallel. One map task takes a key-value pair, processes it and generates another key-value pair for reduced phase input. Mappers group key-value pairs according to re- quirements of algorithms and dispatch them to Reducers.

Mappers group key-value pairs according to requirements of algorithms and dispatch them to Reducers.

• Shuffle phase: When the nodes complete their map task they are ready for sort phase (copy), where nodes communicate with each other to pass key-value pairs to be sorted. This is the only phase where node communicates with each other.

• Sort phase: keys are sorted according to the key ID, presented as: Shuffle( list(K2,V2))→(K2, list(V2) )

• Reducer phase: This makes each reducer take all key-value pairs with the same key and merges them. It then performs computations on the values according to the instruction from java code. Reducer can take a subset of all the key-value pairs, but will always have all the values to one key. The result will be submitted to OutputFormat. Each reducer generates one output to storage (HDFS). This can be controlled through an implementation of Outputformat. Reducer phase takes the form shown below:

Reduce(K2, list(V2)) →list(V3)

• OutputFormat: This deploys RecordWriter to write results back to HDFS ready for the client to read. The network is used when the blocks of the result have to be replicated by HDFS for redundancy.