Essential Hadoop Ecosystem Commands: YARN, HDFS, HBase, Spark & Flink Reference

Introduction

In today's data-driven era, the Hadoop ecosystem serves as a core big data technology stack widely used for massive data storage, processing, and analysis. Its components collaborate to build a complete data processing pipeline. Mastering common commands for core components is key to efficient cluster management and data operations.

1. YARN (Resource Scheduling & Management)

YARN is the Hadoop cluster's resource management core, allocating CPU, memory, and other resources to upper-layer applications such as MapReduce and Spark.

1.1 Resource Management Commands (Cluster Health Monitoring)

Node Status Viewing

Command: yarn node -list Description: Lists all active NodeManager nodes, showing node ID, hostname, health status, and running container count.

Scenario: Daily cluster node health checks to quickly locate offline nodes.

Example:

yarn node -list
# Output: Displays list of nodes with RUNNING status

Command: yarn node -list -all Description: Includes inactive nodes (e.g., LOST, SHUTDOWN) for troubleshooting node failures or maintenance states.

Scenario: After a node goes down abnormally, confirm whether it is correctly marked as inactive.

Command: yarn node -status <nodeId> Description: Retrieves detailed resource usage for a specific node (CPU/memory utilization, disk space, container status).

Scenario: Analyzing single-node load spikes or resource shortages.

Queue Management

Command: yarn queue -status <queueName> Description: Shows queue resource usage (capacity, used resources, waiting task count, queue priority).

Scenario: Monitoring critical queues (e.g., default, high_priority) for resource contention and adjusting quotas.

Example:

yarn queue -status root.default
# Output: Current queue capacity, used memory/CPU, waiting application count, etc.

Command: yarn queue -list Description: Lists all configured queue names in the cluster.

Scenario: Verifying queue creation or modification, often used for permission configuration validation.

1.2 Application Management Commands (Job Full Lifecycle)

Submit Application

Command: yarn jar <jarPath> <mainClass> [args] Description: Submits a Java application (e.g., MapReduce job) to the YARN cluster.

Scenario: Running traditional MapReduce jobs such as log cleansing or data aggregation.

Example:

yarn jar hadoop-mapreduce-examples.jar wordcount /input /output
# Submits WordCount job to count words in HDFS files

Application Status Viewing

Command: yarn application -list Description: Lists all running applications, displaying application ID, name, user, queue, and status (RUNNING/FAILED/FINISHED).

Scenario: Real-time monitoring of running jobs to detect anomalous tasks.

Command: yarn application -status <appId> Description: Retrieves detailed application information (progress, resource consumption, failure cause).

Scenario: Troubleshooting stuck or failed jobs, e.g., Container OOM errors.

Log Retrieval & Job Termination

Command: yarn logs -applicationId <appId> Description: Downloads application logs (including Driver and Executor logs).

Scenario: Analyzing job failure causes such as code exceptions or missing dependencies.

Command: yarn application -kill <appId> Description: Forcefully terminates an abnormal application to release occupied resources.

Scenario: Quick damage control when a job shows no progress for a long time or leaks resources.

1.3 Dynamic Configuration Updates (Cluster Hot Reload)

Command: yarn rmadmin -refreshQueues Description: Reloads queue configuration (e.g., capacity-scheduler.xml) so that dynamically adjusted queue capacities/priorities take effect.

Scenario: Online adjustment of queue resource allocation policies without cluster restart.

Command: yarn rmadmin -refreshNodes Description: Updates the node list, adding new nodes or removing faulty ones.

Scenario: After cluster scale-out/scale-in, ensure ResourceManager perceives node changes.

2. HDFS (Distributed File System)

HDFS stores large-scale data with high-throughput access, serving as the foundational storage layer of the Hadoop ecosystem.

2.1 File & Directory Operations (Data Access)

Data Upload & Download

Command: hdfs dfs -put <localPath> <HDFSPath> Description: Uploads local files to HDFS, supporting single files or directories (use -R for recursion).

Scenario: Uploading raw data (e.g., log files, business data) to the cluster for processing.

Example:

hdfs dfs -put /data/access.log /user/hadoop/logs/
# Uploads local log file to HDFS logs directory

Command: hdfs dfs -get <HDFSPath> <localPath> Description: Downloads HDFS files to local storage, supporting target directory specification.

Scenario: Downloading processed result files (e.g., report data) for local analysis.

Directory & File Viewing

Command: hdfs dfs -ls <HDFSPath> Description: Lists files/directories under the path with size, permissions, and modification time.

Scenario: Confirming file existence or directory structure correctness.

Example:

hdfs dfs -ls /user/hadoop/output
# Views file list under output directory

Command: hdfs dfs -cat <HDFSFilePath> Description: Displays text file content, supports large file chunked display.

Scenario: Quick preview of configuration files or small datasets in HDFS.

Creation & Deletion

Command: hdfs dfs -mkdir -p <HDFSDirPath> Description: Recursively creates directories ( -p auto-creates parent directories).

Scenario: Initializing data storage paths, e.g., creating dedicated directories for new business lines.

Command: hdfs dfs -rm -r <HDFSPath> Description: Recursively deletes directory and contents ( Caution! deleted items move to trash, requires fs.trash.interval configuration).

Scenario: Cleaning up expired data or erroneous output directories.

2.2 Cluster Management Commands (Storage Health Checks)

Cluster Status Report

Command: hdfs dfsadmin -report Description: Shows overall cluster status (node count, total storage, used/remaining space, replica status).

Scenario: Daily inspection of cluster storage usage to determine if expansion is needed.

Key Metrics:

DFS Used%: Storage utilization (recommended ≤ 80%).

Under replicated blocks: Count of blocks with insufficient replicas (ideal value is 0).

Filesystem Check

Command: hdfs fsck <HDFSPath> -files -blocks -locations Description: Checks filesystem integrity, reporting corrupted blocks, replica distribution, and node locations.

Scenario: Repairing data corruption issues (e.g., after disk failure, verify blocks are correctly replicated).

Safe Mode Operations

Command: hdfs dfsadmin -safemode enter/leave/get Description:

enter: Enters safe mode (read-only, for metadata repair).

leave: Exits safe mode (waits for replicas to meet threshold).

get: Views current safe mode status.

Scenario: When NameNode metadata is corrupted, enter safe mode for repair (e.g., manually replicate missing blocks).

2.3 Block & Replica Management (Data Reliability)

Command:

hdfs dfs -setrep -w <replicationFactor> <HDFSFilePath>

Description: Sets file replication factor ( -w waits for replication to complete).

Scenario: Increasing replica count for critical data (e.g., from default 3 to 5 replicas).

Command: hdfs fsck <HDFSFilePath> | grep "Under replicated" Description: Filters under-replicated blocks to pinpoint data reliability issues.

Scenario: After disk failure, quickly identify files needing repair.

3. HBase (Distributed Columnar Database)

Built on HDFS, HBase provides random read/write access to massive structured data, suitable for high-concurrency, low-latency scenarios (e.g., real-time counters, user profiles).

3.1 HBase Shell Core Commands (Table & Data Operations)

Table Definition (DDL)

Command: create 'tableName', 'cf1', 'cf2' Description: Creates a table with specified column families (HBase stores data by column family; recommended ≤ 3 column families).

Scenario: Initializing business tables, e.g., a users table with info (basic info) and logs (operation logs) column families.

Example:

hbase shell
create 'users', 'info', 'logs'

Command: describe 'tableName' Description: Views table schema (column family configs, version count, compression algorithm).

Scenario: Verifying table storage configuration meets business requirements (e.g., TTL settings).

Data Operations (DML)

Command: put 'tableName', 'rowKey', 'cf:qualifier', 'value' Description: Inserts or updates a row (rowKey is the unique identifier).

Scenario: Writing user registration info, e.g., rowKey user_1001, inserting name into info:name column.

Example: put 'users', 'user_1001', 'info:name', 'Alice' Command: get 'tableName', 'rowKey' Description: Retrieves a single row, supports specifying column family or qualifier.

Scenario: Querying user details by user ID, e.g., get 'users', 'user_1001'.

Command: scan 'tableName', {FILTER => "condition"} Description: Scans table data with filtering (e.g., rowKey prefix, column value range).

Scenario: Counting user operation logs within a time range, e.g., scan 'logs', {FILTER => "RowFilter(=, 'binary:202401')"}.

Table Maintenance

Command: disable 'tableName' + drop 'tableName' Description: Must disable table before deletion to avoid metadata inconsistency.

Scenario: Dropping deprecated tables, e.g., removing historical data tables no longer in use.

3.2 Cluster Management Commands (Metadata & Failure Recovery)

Command: start-hbase.sh / stop-hbase.sh Description: Starts/stops HBase cluster (includes HMaster and RegionServer).

Scenario: Routine cluster start/stop, or preparation/cleanup before maintenance.

Command: hbase hbck Description: Checks table metadata consistency, repairs orphan Regions, inconsistent block references, etc.

Scenario: After abnormal cluster restart, repair table metadata corruption (use -repair for auto-repair with caution).

4. Spark (Distributed Computing Engine)

Spark supports batch, streaming, and machine learning workloads. Jobs are submitted via spark-submit and run on YARN or standalone mode.

4.1 Job Submission Commands (Job Submit & Resource Config)

Command: spark-submit Syntax:

spark-submit \
--master <mode> \
# e.g., yarn, local[4], spark://master:7077
--deploy-mode <mode> \
# cluster (cluster mode) or client (client mode)
--executor-memory <size> \
# e.g., 4g
--num-executors <count> \
# e.g., 20
--class <mainClass> \
# application entry class
<jarPath> [args]

Scenarios:

YARN Cluster Mode: --master yarn --deploy-mode cluster for production large-scale jobs.

Local Debug: --master local[2] uses 2 local threads to simulate distributed execution.

Example:

spark-submit \
--master yarn \
--executor-memory 8g \
--num-executors 50 \
--class com.spark.Job \
spark-job.jar /input/data /output/result

4.2 Monitoring & Debugging Commands (Job Diagnostics)

Command: yarn application -list | grep spark Description: Filters all Spark applications to quickly locate job IDs.

Scenario: Finding target Spark job status when multiple jobs run in parallel.

Web UI Access:

Local Mode: http://localhost:4040 (during task run) to view DAG, Stage timing, GC.

YARN Mode: Via YARN Web UI find Spark job link to access Executor details.

Scenario: Analyzing Shuffle bottlenecks (e.g., data skew causing a Task to run too long).

Command: yarn logs -applicationId <appId> Description: Retrieves Spark job logs including Driver and Executor stderr/stdout.

Scenario: Troubleshooting code errors (e.g., NullPointerException) or dependency conflicts.

5. Flink (Stream Processing Framework)

Flink focuses on low-latency, high-throughput stream processing with event-time handling and state management.

5.1 Cluster Management Commands (Standalone Mode)

Command: start-cluster.sh / stop-cluster.sh Description: Starts/stops Flink standalone cluster (includes JobManager and TaskManager).

Scenario: Start/stop operations for self-hosted Flink clusters, suitable for small-to-medium deployments.

Command: flink list -m <JobManagerAddress> Description: Lists running jobs in the cluster, showing job ID, name, status, submission time.

Scenario: Confirming successful job submission or viewing historical job list.

5.2 Job Submission & Control (Cross-Mode Common)

Command: flink run Syntax:

flink run \
-m <JobManagerAddress> \
# e.g., yarn-cluster, localhost:8081
-p <parallelism> \
# global parallelism
-c <mainClass> \
# job entry class
<jarPath> [args]

Scenarios:

YARN Mode: -m yarn-cluster submits to YARN cluster for large-scale stream processing.

Local Mode: -m localhost:8081 for local job logic debugging.

Example:

flink run \
-m yarn-cluster \
-p 10 \
-c com.flink.StreamJob \
flink-job.jar --input topic --output hdfs://path

Command: flink cancel <jobId> Description: Stops a running job, supports generating Savepoint ( -s <path>) for fault recovery.

Scenario: After fixing job logic errors, restart job and recover state from Savepoint.

5.3 Monitoring & Logs (Visualization & Diagnostics)

Web UI Access: http://<JobManagerIP>:8081 Features: View job topology, Task backpressure status, Checkpoint duration, network transmission latency.

Scenario: Locating stream processing bottlenecks (e.g., Source read slowness causing backpressure, or Sink write timeouts).

Command: flink logs <jobId> Description: Retrieves job logs including detailed TaskManager and JobManager logs.

Scenario: Analyzing Checkpoint failure causes (e.g., state size exceeding memory limits).

6. Summary: Command Categories & Core Scenarios

By mastering the above commands, one can essentially complete core tasks such as cluster operations, job scheduling, data access, and fault troubleshooting. Combined with each component's Web UI and monitoring tools, this enables end-to-end management from command line to visualization.