Enabling CGroup in Hadoop Yarn NodeManager to Limit Container CPU Resources

Linux Control Groups (CGroup) allow fine‑grained resource limiting for processes, and Yarn NodeManager can use CGroup to restrict CPU usage of containers.

To enable CGroup in Yarn, set the container‑executor class to

org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor

and configure the Unix group in yarn.nodemanager.linux-container-executor.group as shown:

<property>
<name>yarn.nodemanager.container-executor.class</name>
<value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.group</name>
<value>hadoop</value>
</property>

Enable the CGroup handler by setting

yarn.nodemanager.linux-container-executor.resources-handler.class

to

org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler

:

<property>
<name>yarn.nodemanager.linux-container-executor.resources-handler.class</name>
<value>org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler</value>
</property>

Configure the CGroup hierarchy and mounting options, for example:

<property>
<name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
<value>/hadoop-yarn</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name>
<value>/sys/fs/cgroup</value>
</property>

CPU limits are controlled via yarn.nodemanager.resource.percentage-physical-cpu-limit (e.g., 80 %) and

yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage

(false for non‑strict sharing). Set yarn.nodemanager.resource.count-logical-processors-as-cores to true to count logical CPUs.

<property>
<name>yarn.nodemanager.resource.percentage-physical-cpu-limit</name>
<value>80</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.resource.count-logical-processors-as-cores</name>
<value>true</value>
</property>

Note: On Linux kernel 3.10.0‑327.el7.x86_64 enabling CGroup in Yarn triggers a kernel bug that hangs the system; upgrade the kernel to avoid this issue.