some useful knowleges in Hadoop

1. Visual Report

Hadoop Core may be configured to report detailed information about the activities on the cluster. Hadoop Core may report to a file, via Ganglia (http://ganglia.info/), which provides a framework for displaying graphical reports summarizing the activities of large clusters of machines. The file hadoop-metrics.properties controls the reporting. The default is to not report.

2. JVM options for the task virtual machines

The mapred.child.java.opts parameter is commonly used to set a default maximum heap size for tasks. The default value is -Xmx200m. Most installation administrators immediately change this value to -Xmx500m. A significant and unexpected influence on this is the heap requirements (io.sort.mb), which by default will cause 100MB of space to be used for sorting.

During the run phase of a job, there may be up to mapred.tasktracker.map.tasks.maximum map tasks and mapred.tasktracker.reduce.tasks.maximum reduce tasks running simultaneously on each TaskTracker node, as well as the TaskTracker JVM. The node must have sufficient virtual memory to meet the memory requirements of all of the JVMs. JVMs have non-heap memory requirements; for simplicity, 20MB is assumed.

A cluster that sets the map task maximum to 1, the reduce task maximum to 8, and the JVM heap size to 500MB would need a minimum of (1 + 8 + 1) * (500+20) = 10 * 520 = 5200MB, or 5GB, of virtual memory available for the JVMs on each TaskTracker host. This 5GB value does not include memory for other processes or servers that may be running on the node.

The mapred.child.java.opts parameter is configurable by the job.

3. Enable Job Control Options on the Web Interfaces

Both the JobTracker and the NameNode provide a web interface for monitoring and control. By default, the JobTracker provides web service on http://JobtrackerHost:50030 and the NameNode provides web service on http://NamenodeHost:50070. If the webinterface.private.actions parameter is set to true, the JobTracker web interface will add Kill This Job and Change Job Priority options to the per-job detail page. The default location of these additional options is the bottom-left corner of the page (so you usually need to scroll down the page to see them).