Monitoring guide

All service components of Hadoop provide complete native monitoring functions. In addition, we have also integrated Ganglia monitoring tool in KMR. SSH Tunnel should be established before accessing to these resources.

Establish SSH Tunnel

SSH Tunnel should be established before accessing to services in the cluster, and the browser should be configured as follows:

If no EIP is configured for your KMR cluster or you do not want to use SSH Tunnel, you can activate a windows cloud host under the same account. You can skip this step to access to the cluster to monitor resources through internal network address of the master node.

Establish SSH Tunnel

After the cluster is created successfully, you can obtain public network IP, user name, password and SSH port number of the master node as specified in the message sent by the system, and establish SSH Tunnel to connect the cluster through the following configuration:

1. Refer toSSH Connection Guide to prepare for the master node connection.
   
2. Select SSH>Tunnels, enter a port not used by local computer, select “Dynamic” and “IPv4”, keep “Destination” empty, and click “Add” button.
   
   
   
3. Click “Open”, if the window pops up, click “Yes”.
   
   Notes: SSH tunnel BUG exists in early version of putty. Please download the latest version from official website, or select other SSH tool.
   
   http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

Configure the browser

Configure Firefox, and select “Remote DNS” option to parse the host correctly.

Now, you can access to native Hadoop services directly through your browser.

Monitor the cluster status with Ambari console

Ambari provides a powerful set of management and maintenance features for KMR services, including the cluster user, service installation, service monitoring, etc.

As shown in the figure below, after entering the cluster user page through Admin menu in upper right corner, we can see that Ambari has prebuilt a set of users for the Hadoop services installed. These users include two types: first, service users in Ambari system. These users are stored in the metadata repository of Ambari itself; and second, Linux users established on the target host through the Agent service. These Linux users will be used when Hadoop runs. Ambari offers highly automated cluster user function, which not only eliminates the worry of manual maintenance of the cluster users, but also provides the convenience for integrating the privilege later.

In KMR, we integrate Ambari service in the cluster, so you can access to Web console of Ambari to monitor KMR cluster. The service page is shown as follows:

View Hadoop service status

Hadoop and other applications installed on KMR cluster will expose the user interface on the website hosted on the master node. These pages show statistics and monitoring information of various cluster services. Through Establish SSH Tunnel step, you can access to these pages directly with the browser.

Service	Port
YARN ResourceManager	http://master-private-IP:8088
HDFS Namenode	http://master-private-IP:50070
OOZIE	http://master-private-IP:11000
YARN HistoryServer	http://master-private-IP:19888
Spark History Server	http://master-private-IP:18080

YARN ResourceManage

HDFS Namenode

OOZIE

YARN HistoryServer

Spark History Server