How Hadoop 2.0 Collects and Manages Job Logs with YARN

Hadoop 2.0 provides a job log collection component similar to Hadoop 1.0. Since YARN is now a general resource manager, a universal log collection module is needed; while a generic module is under development, this article explains the built‑in MRv2 (MapReduce on YARN) log collection, its principles and configuration.

In Hadoop 2.0 each job produces two logs: the job‑run log generated by MRAppMaster and the task‑run logs. The job‑run log records start time, finish time, counters, etc., and is analogous to the JobHistory log in Hadoop 1.0. The ApplicationMaster itself runs in a container (e.g., container_1385051297072_0001_01_000001) and also writes its own log.

Example of an ApplicationMaster job‑run log (stored in Avro and saved as JSON):

{
  "type":"JOB_SUBMITTED","event":{"org.apache.hadoop.mapreduce.jobhistory.JobSubmitted":
  {"jobid":"job_1385051297072_0002","jobName":"QuasiMonteCarlo","userName":"yarn","submitTime":1385393834983,"jobConfPath":"hdfs://hadoop-test/tmp/hadoop-yarn/staging/yarn/.staging/job_1385051297072_0002/job.xml","acls":{},"jobQueueName":"default","workflowId":"","workflowName":"","workflowNodeName":"","workflowAdjacencies":"","workflowTags":""}
},
  "type":"JOB_INITED","event":{"org.apache.hadoop.mapreduce.jobhistory.JobInited":
  {"jobid":"job_1385051297072_0002","launchTime":1385393974505,"totalMaps":8,"totalReduces":1,"jobStatus":"INITED","uberized":false}
},
  "type":"JOB_INFO_CHANGED","event":{"org.apache.hadoop.mapreduce.jobhistory.JobInfoChange":
  {"jobid":"job_1385051297072_0002","submitTime":1385393834983,"launchTime":1385393974505}}
}

The job‑run log generation process:

ResourceManager launches the ApplicationMaster. The AM writes logs to

${yarn.app.mapreduce.am.staging-dir}/yarn/.staging/job_XXXXX_XXX/

, producing three files with extensions .jhist, .summary, and .xml.

After all tasks finish, the AM copies these three files to ${mapreduce.jobhistory.intermediate-done-dir}/${username}, appending “_tmp” to the filenames.

The AM renames the copied files, removing the “_tmp” suffix, so they end with .jhist, .summary, and .xml.

A periodic scanner moves the files from the intermediate “done_intermediate” directory to the final done directory (configured by mapreduce.jobhistory.done-dir) and deletes the .summary file.

The AM cleans up the staging directory.

Task logs are stored on each NodeManager’s local disks under ${yarn.log.dir}/userlogs. Enabling log aggregation pushes these logs to HDFS, allowing centralized analysis. Example directory listing:

yarn@YARN-001:/opt/yarn/yarn-client$ ls ../yarn/logs/userlogs/*

The JobHistory Server is a separate service that reads the .jhist files and provides a Web UI showing job start/end times, task durations, counters, and links to the logs. It is configured in mapred-site.xml and started with sbin/mr-jobhistory-daemon.sh start jobhistoryserver.

Configuration files: yarn-site.xml – YARN‑wide settings used by ResourceManager, NodeManager, and clients. mapred-site.xml – MapReduce‑specific settings required by clients and the JobHistory Server.