MapReduce - Hadoop Administration

This chapter explains Hadoop administration which includes both HDFS and MapReduce administration.

HDFS administration includes monitoring the HDFS file structure, locations, and the updated files.

MapReduce administration includes monitoring the pst of apppcations, configuration of nodes, apppcation status, etc.

HDFS Monitoring

HDFS (Hadoop Distributed File System) contains the user directories, input files, and output files. Use the MapReduce commands, put and get, for storing and retrieving.

After starting the Hadoop framework (daemons) by passing the command “start-all.sh” on “/$HADOOP_HOME/sbin”, pass the following URL to the browser “http://localhost:50070”. You should see the following screen on your browser.

The following screenshot shows how to browse the browse HDFS.

The following screenshot show the file structure of HDFS. It shows the files in the “/user/hadoop” directory.

The following screenshot shows the Datanode information in a cluster. Here you can find one node with its configurations and capacities.

MapReduce Job Monitoring

A MapReduce apppcation is a collection of jobs (Map job, Combiner, Partitioner, and Reduce job). It is mandatory to monitor and maintain the following −

Configuration of datanode where the apppcation is suitable.

The number of datanodes and resources used per apppcation.

To monitor all these things, it is imperative that we should have a user interface. After starting the Hadoop framework by passing the command “start-all.sh” on “/$HADOOP_HOME/sbin”, pass the following URL to the browser “http://localhost:8080”. You should see the following screen on your browser.

In the above screenshot, the hand pointer is on the apppcation ID. Just cpck on it to find the following screen on your browser. It describes the following −

On which user the current apppcation is running

The apppcation name

Type of that apppcation

Current status, Final status

Apppcation started time, elapsed (completed time), if it is complete at the time of monitoring

The history of this apppcation, i.e., log information

And finally, the node information, i.e., the nodes that participated in running the apppcation.

The following screenshot shows the details of a particular apppcation −

The following screenshot describes the currently running nodes information. Here, the screenshot contains only one node. A hand pointer shows the localhost address of the running node.