Hadoop Monitoring Technical Reference

Overview

Hadoop monitoring is a Gateway configuration file that enables monitoring of the Hadoop cluster, nodes, and daemons through the JMX and Toolkit plug-ins.

This Hadoop integration template consists of the following components:

  • Hadoop Distributed File System (HDFS) 
  • Yet Another Resource Negotiator (YARN)

The Hadoop Distributed File System or HDFS provides scalable data storage that can be deployed on hardware and optimised operations for large datasets.

The other component Yet Another Resource Negotiator or YARN assigns the computation resources for executing the application:

  • YARN ResourceManager - takes inventory of available and allocate resources to running applications.
  • YARN NodeManagers - monitors resource usage and communicates with the ResourceManager.

Intended audience

This technical reference is intended for users who will be using Active Console to monitor data from Hadoop. If you are setting up the integration for the first time, see Hadoop Monitoring User Guide.

Metrics and dataviews

Hadoop HDFS namenode info

Column Name Description
Name Name of the service.
SoftwareVersion Namenode service version.
SecurityEnabled Number security setup.
State Namenode service active state.
   

Hadoop HDFS namenode cluster

Column Name Description
Name Name of the service.
CapacityUsedGB Current used capacity across all datanodes.
CapacityRemainingGB Current remaining capacity.
CapacityTotalGB Current raw capacity.
FilesTotal Number of files and directories.
TotalLoad Number of connections.
NumLiveDataNodes Number of live datanodes.
NumStaleDataNodes Number of datanodes marked stale due to delayed hearbeat.
NumDeadDataNodes Number of dead datanodes.
BlocksTotal Number of allocated blocks in the system.
BlockCapacity Number of block capacity.
CorruptBlocks Number of blocks with corrupt replicas.
UnderReplicatedBlocks Number of blocks under replicated.
MissingBlocks Number of missing blocks.
   

Hadoop HDFS secondaryNamenode info

Column Name Description
Name Name of the service.
CheckpointDirectories Secondarynamenode checkpoint directories.
CheckpointEditlogDirectories Secondarynamenode checkpoint edit log directories.
SoftwareVersion Secondarynamenode service version.
   

Hadoop HDFS datanodes summary

Column Name Description
name Datanode name and (dfs) port address.
infoAddr Datanode Web UI address.
usedSpaceGB Datanode used capacity.
nonDfsUsedSpaceGB Datanode non-dfs used capacity.
capacityGB Datanode raw capacity.
remainingGB Datanode remaining capacity.
numBlocks Number of blocks in the datanode.
version Datanode service version.
volFails Number of failed volumes in the datanode.
   

The number of rows displayed is equal to the number of datanodes set in the whole cluster.

Hadoop YARN resource manager

Column Name Description
Name Name of the service.
NumActiveNMs Number of active nodemanagers.
NumDecommissionedNMs Number of decomissioned nodemanagers.
NumLostNMs Number of lost nodemanagers for not sending hearbeats.
NumUnhealthyNMs Number of unhealthy nodemanagers.
AppsRunning Number of running applications.
AppsFailed Total number of failed application.
AllocatedMB Current allocated memory in MB.
AvailableMB Available memory in MB.
   

Hadoop YARN nodeManagers summary

Column Name Description
Hostname Hostname where the nodemanager service is running.
State Current nodemanager state.
NodeID nodemanager Node ID.
NodeHTTPAdress nodemanager WEB UI address.
NodeManagerVersion nodemanager service version.
HealthReport nodemanager health report.
   

Note: The number of rows displayed is equal to the number of nodemangers running.

Hadoop node metrics dataview

Hadoop HDFS datanode volume info

Column Name Description
dir Path of volume directory.
numBlocks Current number of blocks in the datanode volume.
usedSpace Used space in the datanode volume.
freeSpace Free space in the datanode volume.
reservedSpace Reserved space for datanode volume.
storageType Type of datanode volume storage.
reservedSpaceForReplicas Reserved space for replicas.
   

Note: The number of rows displayed is equal to the number of volumes in a single datanode.