Hadoop in a Slice

Users who are interested in experimenting with Hadoop workloads in a slice can use ExoGENI tools to request, provision and start virtual Hadoop clusters on-demand. This post describes how to use Flukes and postboot scripts to build Hadoop clusters. As described in a previous post, it is possible to use postboot scripts to assign hostnames to VMs and add entries to /etc/hosts so that VMs can address each other. The postboot scripts are also useful to start different kinds of daemons after the VM boots, and hence enable automatic setup of virtual clusters, eg. Condor, Hadoop etc. In this post, we will walk through the following steps to create a Hadoop cluster on the ExoGENI racks.

  1. Setting up a request in Flukes
  2. Constructing postboot scripts for Hadoop master and workers
  3. Submitting request to ExoGENI and verifying when slice is ready
  4. Verifying that Hadoop cluster is ready to accept MapReduce jobs

Flukes: Request for Hadoop cluster

Users can create a request in Flukes with one Node and one Nodegroup connected by a link. To edit the properties of the Node, which is going to be the Hadoop master, right click on the Node and select “Edit Properties”. Change the name of the Node to “HadoopMaster”. Select an image type, c1.xlarge in the example below, and you can optionally bind the slice to a particular rack, FIU XO rack in the example. ExoGENI already provides a VM image with Hadoop tools already built in. This image should show up in the list of available images as “Debian6 (squeeze) + Hadoop”. Select that image. Assign a dataplane IP address for the Hadoop master, 172.16.1.10/24 in the example. Click OK.

The Nodegroup represents the Hadoop workers. Right click on the Nodegroup and select “Edit Properties”. Change the name of the Nodegroup to “HadoopWorkers”. Select image, image type and site binding as above. Provide the number of servers as the number of Hadoop worker nodes. Assign the dataplane IP of the first Hadoop worker, 172.16.1.20/24 in the example. The IP address of the other workers would be automatically set, 172.16.1.21/24 an so on.

Constructing postboot scripts

Postboot scripts are needed to edit /etc/hosts and to start the Hadoop daemons.  Right click on the “HadoopMaster” Node and select “Edit Properties”. Click on “PostBoot Script”. Provide a postboot script for the Hadoop master, as below. It sets up /etc/hosts, hostname etc. It waits until it can be pinged using the dataplane IP address, since that IP address should exist before Hadoop master daemons start. It then invokes the hadoop-euca-init.sh script to start the Hadoop master daemons.

Hadoop master postboot script

#!/bin/bash
echo "Hello from neuca script" > /home/hadoop/log
echo $HadoopMaster.IP("Link0") $HadoopMaster.IP("Link0") >> /etc/hosts
#set ( $sizeWorkerGroup = $HadoopWorkers.size() - 1 )
#foreach ( $j in [0..$sizeWorkerGroup] )
 echo $HadoopWorkers.get($j).IP("Link0") $HadoopWorkers.get($j).IP("Link0") >> /etc/hosts
#end
name=$self.Name()
if [ "$name" = "HadoopMaster" ];
then
 echo $HadoopMaster.IP("Link0") > /etc/hostname
 hostname -F /etc/hostname
 hostname >> /home/hadoop/log
 while true; do
     PING=`ping -c 1 $HadoopMaster.IP("Link0") > /dev/null 2>&1`
     if [ "$?" = "0" ]; then 
         break
     fi
     sleep 5
 done
 echo '/home/hadoop/hadoop-euca-init.sh $HadoopMaster.IP("Link0") -master' >> /home/hadoop/log
 /home/hadoop/hadoop-euca-init.sh $HadoopMaster.IP("Link0") -master
 echo "Done starting daemons" >> /home/hadoop/log
fi

Right click on the “HadoopWorkers” Nodegroup and select “Edit Properties”. Click on “PostBoot Script”. Provide a postboot script for the Hadoop workers, as below. It sets up /etc/hosts, hostname etc. It waits until the Hadoop master can be pinged, since the Hadoop master needs to be ready before the workers are up. It also waits until it can be pinged on it’s dataplane IP address, since that IP address should exist before Hadoop worker daemons start. It then invokes the hadoop-euca-init.sh script to start the Hadoop worker daemons.

Hadoop Workers’ postboot script

#!/bin/bash
echo "Hello from neuca script" > /home/hadoop/log
echo $HadoopMaster.IP("Link0") $HadoopMaster.IP("Link0") >> /etc/hosts
#set ( $sizeWorkerGroup = $HadoopWorkers.size() - 1 )
#foreach ( $j in [0..$sizeWorkerGroup] )
 echo $HadoopWorkers.get($j).IP("Link0") $HadoopWorkers.get($j).IP("Link0") >> /etc/hosts
#end
 echo $self.IP("Link0") > /etc/hostname
 hostname -F /etc/hostname
 hostname >> /home/hadoop/log
 while true; do
     PING=`ping -c 1 $HadoopMaster.IP("Link0") > /dev/null 2>&1`
     if [ "$?" = "0" ]; then 
         break
     fi
     sleep 5
 done
 while true; do
     PING=`ping -c 1 $self.IP("Link0") > /dev/null 2>&1`
     if [ "$?" = "0" ]; then 
         break
     fi
     sleep 5
 done 
 echo '/home/hadoop/hadoop-euca-init.sh $HadoopMaster.IP("Link0")' >> /home/hadoop/log
 /home/hadoop/hadoop-euca-init.sh $HadoopMaster.IP("Link0")
 echo "Done starting daemons" >> /home/hadoop/log

Submitting Hadoop slice request and verifying whether slice is ready

Set slice name, and submit the request to ExoGENI. You will see a response from the system showing the individual slivers that are being requested. Click OK. At this point the request has been sent and the slivers are being brought up.

Go to “Manifest View” in Flukes. Click on My Slices. Select your slice based on the slice name you set. Click on Query. When all slivers in your slice are ready, you will see an “Active” state beside all the slivers, and the slice is ready to use. A “Ticketed” state means that the sliver is not ready yet. Wait until all slivers are ready.

Verifying that Hadoop cluster is ready to accept MapReduce jobs

At this point the nodes in the slice are ready for user access. Users can right click on the HadoopMaster node in the manifest view to login to the Hadoop master VM. Note that the Hadoop daemons startup scripts might take a few more seconds to finish execution. Users can check logs on the Hadoop master by doing

$ tail -f /usr/local/hadoop-0.20.2/logs/hadoop-root-namenode-<master-dataplane-ip-address>.log

To check the number of datanodes (workers) that have joined the hadoop cluster, users can run the ‘dfsadmin’ command as follows.

$ /usr/local/hadoop-0.20.2/bin/hadoop dfsadmin -report

This command should output the number of Hadoop workers that are currently in the Hadoop cluster as “Datanodes available” (See below). Once the requested number of Hadoop workers are output, the Hadoop cluster setup is complete. At this point the Hadoop cluster is ready to accept MapReduce jobs.

......
-------------------------------------------------
Datanodes available: 4 (4 total, 0 dead)
Name: 172.16.1.21:50010
Rack: /default/rack0
Decommission Status : Normal
Configured Capacity: 68698120192 (63.98 GB)
DFS Used: 36864 (36 KB)
Non DFS Used: 5238116352 (4.88 GB)
DFS Remaining: 63459966976(59.1 GB)
DFS Used%: 0%
DFS Remaining%: 92.38%
Last contact: Tue Jul 02 03:05:36 UTC 2013
......
Note: The Hadoop cluster is configured with a base configuration. If the user wants to use advanced configuration properties, she would need to modify the /home/hadoop/hadoop-euca-init.sh script in the image. She can either build a new image with the modified configuration, or edit the file using postboot scripts. Hadoop is installed at /usr/local/hadoop-0.20.2 on the provided image.

Have something to add?

Loading Facebook Comments ...