Apache Hadoop NextGen MapReduce (YARN)

MapReduce has undergone a complete overhaul in hadoop-0.23 and we now have, what we call,
MapReduce 2.0 (MRv2) or YARN.
The fundamental idea of MRv2 is to split up the two major functionalities of the JobTracker,
resource management and job scheduling/monitoring, into separate daemons. The idea is to have a
global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either
a single job in the classical sense of Map-Reduce jobs or a DAG of jobs.
The ResourceManager and per-node slave, the NodeManager (NM), form the data-computation
framework. The ResourceManager is the ultimate authority that arbitrates resources among all the
applications in the system.
The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with
negotiating resources from the ResourceManager and working with the NodeManager(s) to execute
and monitor the tasks.

The ResourceManager has two main components: Scheduler and ApplicationsManager.

The Scheduler is responsible for allocating resources to the various running applications subject to
familiar constraints of capacities, queues etc. The Scheduler is pure scheduler in the sense that it
performs no monitoring or tracking of status for the application. Also, it offers no guarantees about
restarting failed tasks either due to application failure or hardware failures. The Scheduler performs
its scheduling function based the resource requirements of the applications; it does so based on the
abstract notion of a resource Container which incorporates elements such as memory, cpu, disk,
network etc. In the first version, only memory is supported. The Scheduler has a pluggable policy
plug-in, which is responsible for partitioning the cluster resources among the various queues,
applications etc. The current Map-Reduce schedulers such as the CapacityScheduler and the
FairScheduler would be some examples of the plug-in.
The CapacityScheduler supports hierarchical queues to allow for more predictable sharing of cluster
resources
The ApplicationsManager is responsible for accepting job-submissions, negotiating the first
container for executing the application specific ApplicationMaster and provides the service for
restarting the ApplicationMaster container on failure.
The NodeManager is the per-machine framework agent who is responsible for containers,
monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the
ResourceManager/Scheduler.
The per-application ApplicationMaster has the responsibility of negotiating appropriate resource
containers from the Scheduler, tracking their status and monitoring for progress.
MRV2 maintains API compatibility with previous stable release (hadoop-1.x). This means that all
Map-Reduce jobs should still run unchanged on top of MRv2 with just a recompile.

Your cluster has slave nodes in three different racks, and you have written a rack topology script
identifying each machine as being in hadooprack1, hadooprack2, or hadooprack3. A client machine
outside of the cluster writes a small (one-block) file to HDF S. The first replica of the block is written
to a node on hadooprack2. How is block placement determined for the other two replicas?
1. One will be written to another node on hadooprack2, and the other to a node on a different rack.
2. Either both will be written to nodes on hadooprack1, or both will be written to nodes on
hadooprack3.
3. They will be written to two random nodes, regardless of rack location. 4. One will be written to
hadooprack1. and one will be written to hadooprack3.
Correct Answer : 2
Explanation : HEWS is designed to reliably store very large files across machines in a large cluster.
It stores each file as a sequence of blocks; all blocks in a file except the last block are the same
size. The blocks of a file are replicated for fault tolerance. The block size and replication factor are
configurable per file. An application can specify the number of replicas of a file. The replication
factor can be specified at file creation time and can be changed later. Files in HDFS are write-once
and have strictly one writer at any time.
The NameNode makes all decisions regarding replication of blocks. It periodically receives a
Heartbeat and a Blockreport from each of the Datallodes in the cluster. Receipt of a Heartbeat
implies that the Datallode is functioning properly. A Blockreport contains a list of all blocks on a
Datallode.
For the default threefold replication, Hadoop's rack placement policy is to write the first copy of a
block on a node in one rack, then the other two copies on two nodes in a different rack. Since the
first copy is written to hadooprack2, the other two will either be written to two nodes on hadoprack1,
or two nodes on hadooprack3.
Replica Placement The First Baby Steps
The placement of replicas is critical to HDFS reliability and performance. Optimizing replica
placement distinguishes HDFS from most other distributed file systems. This is a feature that needs
lots of tuning and experience. The purpose of a rack-aware replica placement policy is to improve
data reliability, availability, and network bandwidth utilization. The current implementation for the
replica placement policy is a first effort in this direction. The short-term goals of implementing this
policy are to validate it on production systems, learn more about its behavior, and build a foundation
to test and research more sophisticated policies.
Large HDFS instances run on a cluster of computers that commonly spread across many racks.
Communication between two nodes in different racks has to go through switches. In most cases,
network bandwidth between machines in the same rack is greater than network bandwidth between
machines in different racks.
The NameNode determines the rack id each Datallode belongs to via the process outlined in
Hadoop Rack Awareness. A simple but non-optimal policy is to place replicas on unique racks. This
prevents losing data when an entire rack fails and allows use of bandwidth from multiple racks when
reading data. This policy evenly distributes replicas in the cluster which makes it easy to balance
load on component failure. However, this policy increases the cost of writes because a write needs
to transfer blocks to multiple racks.
For the common case, when the replication factor is three. HDFSs placement policy is to put one
replica on one node in the local rack, another on a node in a different (remote) rack, and the last on
a different node in the same remote rack. This policy cuts the inter-rack write traffic which generally
improves write performance. The chance of rack failure is far less than that of node failure; this
policy does not impact data reliability and availability guarantees. However, it does reduce the
aggregate network bandwidth used when reading data since a block is placed in only two unique
racks rather than three. With this policy, the replicas of a file do not evenly distribute across the
racks. One third of replicas are on one node. two thirds of replicas are on one rack, and the other
third are evenly distributed across the remaining racks. This policy improves write performance
without compromising data reliability or read performance.
The current, default replica placement policy described here is a work in progress.

Five (sometimes six) ways to set configuration values (from highest

priority to least):

1. In Java code with custom setter (which only some properties

have)
job.setNumReduceTasks(25)
Getters also available
System.out.println("numReducers: "+job.getNumReduceTasks());
2. In Java code with conf class before job creation
conf.set("mapred.userlog.retain.hours", "23");
conf.set("mapred.reduce.tasks", "100");
Job job = new Job(conf, "movie count");
3. Set property on command line
hadoop jar hadoop-student.jar actorlist -D
mapred.userlog.retain.hours=22
/user/hadoop/veggie/input /user/hadoop/veggie/output
=-D mapred.reduce.tasks=6'
Repeat entry for multiple properties
=-D mapred.heartbeats.in.second=80 -D
mapred.map.max.attempts=2'
Put value in single quotes if multiple words
-D mapred.job.name='My Job'
4. Load custom configuration file on command line
hadoop jar hadoop-student-homework-1.0.0-SNAPSHOT.jar
actorlist
=-conf /Users/john/custom.xml /user/hadoop/veggie/input'
/user/hadoop/veggie/output
5. Set environment variable to a dir which contains the three sitespecific configuration files
export HADOOP_CONF_DIR=/Users/john/psuedo-conf
6. Change the "read-only" default configuration files under src/
Bad idea because it affects all users, but possible (may get you
walked out the door)