All Documents
Current Document

Content is empty

If you don't find the content you expect, please try another search term


Monitoring guide

Last updated:2020-05-08 23:04:52

All service components of Hadoop provide complete native monitoring functions. In addition, we have also integrated Ganglia monitoring tool in KMR. SSH Tunnel should be established before accessing to these resources.

Establish SSH Tunnel

SSH Tunnel should be established before accessing to services in the cluster, and the browser should be configured as follows:
If no EIP is configured for your KMR cluster or you do not want to use SSH Tunnel, you can activate a windows cloud host under the same account. You can skip this step to access to the cluster to monitor resources through internal network address of the master node.

Establish SSH Tunnel

After the cluster is created successfully, you can obtain public network IP, user name, password and SSH port number of the master node as specified in the message sent by the system, and establish SSH Tunnel to connect the cluster through the following configuration:

  1. Refer toSSH Connection Guide to prepare for the master node connection.
  2. Select SSH>Tunnels, enter a port not used by local computer, select “Dynamic” and “IPv4”, keep “Destination” empty, and click “Add” button.
  3. Click “Open”, if the window pops up, click “Yes”.
    Notes: SSH tunnel BUG exists in early version of putty. Please download the latest version from official website, or select other SSH tool.

    Configure the browser

    Configure Firefox, and select “Remote DNS” option to parse the host correctly.
    Now, you can access to native Hadoop services directly through your browser.

    Monitor the cluster status with Ambari console

    Ambari provides a powerful set of management and maintenance features for KMR services, including the cluster user, service installation, service monitoring, etc.
    As shown in the figure below, after entering the cluster user page through Admin menu in upper right corner, we can see that Ambari has prebuilt a set of users for the Hadoop services installed. These users include two types: first, service users in Ambari system. These users are stored in the metadata repository of Ambari itself; and second, Linux users established on the target host through the Agent service. These Linux users will be used when Hadoop runs. Ambari offers highly automated cluster user function, which not only eliminates the worry of manual maintenance of the cluster users, but also provides the convenience for integrating the privilege later.
    In KMR, we integrate Ambari service in the cluster, so you can access to Web console of Ambari to monitor KMR cluster. The service page is shown as follows:

    View Hadoop service status

    Hadoop and other applications installed on KMR cluster will expose the user interface on the website hosted on the master node. These pages show statistics and monitoring information of various cluster services. Through [Establish SSH Tunnel]() step, you can access to these pages directly with the browser. Service Port
    YARN ResourceManager http://master-private-IP:8088
    HDFS Namenode http://master-private-IP:50070
    OOZIE http://master-private-IP:11000
    YARN HistoryServer http://master-private-IP:19888
    Spark History Server http://master-private-IP:18080

    YARN ResourceManage


    HDFS Namenode




    YARN HistoryServer


    Spark History Server


On this page
Pure ModeNormal Mode

Pure Mode

Click to preview the document content in full screen