How to Securely Deploy Kata Containers with Custom Kernels and VirtioFS Limits

Background

As cloud‑native technologies are adopted, container isolation and performance requirements become stricter. Traditional runtimes such as Docker and containerd have limitations in security isolation and resource overhead, prompting the adoption of Kata Containers as a secure container solution.

Kata Containers creates a lightweight MicroVM for each container, giving each container its own isolated hardware abstraction layer (CPU, memory, I/O, kernel). This combines hardware virtualization with container agility, meeting high‑security standards for workloads like finance while avoiding the performance penalty of full VMs.

MTU Configuration

Docker’s default MTU is 1500, which may exceed the effective MTU in virtualized environments (e.g., 1450 after accounting for overlay overhead). Two solutions are offered:

Use --network host to share the host network stack (no MTU matching, suitable only for testing).

Set "mtu": 1450 in /etc/docker/daemon.json to force containers to use the adjusted MTU.

$ systemctl status docker
$ systemctl start docker
$ mkdir -p /etc/docker   # create config directory
$ touch /etc/docker/daemon.json   # create empty config file
$ chmod 644 /etc/docker/daemon.json
$ systemctl restart docker

The daemon.json should contain:

{
  "insecure-registries": ["r.addops.soft.360.cn"],
  "mtu": 1450,
  "iptables": true,
  "live-restore": true,
  "init": true,
  "data-root": "/data/docker",
  "selinux-enabled": false
}

VPC networks also require MTU ≤ 1450 because overlay tunnels consume additional header space.

Build Steps

1. Build Root Filesystem Image

$ ./kata-containers/tools/osbuilder/rootfs-builder/rootfs.sh -l
$ export distro="ubuntu"
$ export ROOTFS_DIR="$(realpath kata-containers/tools/osbuilder/rootfs-builder/rootfs)"
$ sudo rm -rf "${ROOTFS_DIR}"
$ pushd kata-containers/tools/osbuilder/rootfs-builder
$ script -fec 'sudo -E USE_DOCKER=true ./rootfs.sh "${distro}"'
$ popd
$ pushd kata-containers/tools/osbuilder/image-builder
$ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh "${ROOTFS_DIR}"'
$ popd

The command generates a *.img file under kata-containers/tools/osbuilder/.

2. Build Custom Kernel

$ ./build-kernel.sh -h
$ cd kata-containers/tools/packaging/kernel
$ ./build-kernel.sh setup -v 5.19 -a x86_64 -g nvidia
$ ./build-kernel.sh -v 5.19 -a x86_64 -g nvidia build

Key options: -v 5.19: kernel version. -a x86_64: architecture. -g nvidia: enable NVIDIA GPU support.

The resulting bzImage is placed in

kata-containers/packaging/kernel/kata-linux-5.19-114/arch/x86/boot/

.

3. Deploy to Kubernetes

# Apply deployment manifests on the host
kubectl apply -f kata-rbac.yaml
kubectl apply -f kata-deploy-stable.yaml
kubectl apply -f kata-runtimeClasses.yaml
systemctl restart containerd
# Verify kata fields in /etc/containerd/config.toml
cat /etc/containerd/config.toml

Copy the built image and kernel to the host configuration directory:

cp kata-containers-anolis.img /opt/kata/share/kata-containers/
cp bzImage /opt/kata/share/kata-containers/vmlinuz-5.19

Update /opt/kata/share/defaults/kata-containers/configuration.toml with the correct paths for hypervisor.qemu.path, kernel, and image.

4. VirtioFS Rate‑Limiting

git clone -b devel https://code.geelib.qihoo.net:11443/g-virt-dev/virtiofsd.git
dnf install libcap-ng-devel libseccomp-devel
cargo build --release
cd ./virtiofsd/target/release

Replace the host’s /opt/kata/libexec/virtiofsd with the built binary and set the following fields in configuration.toml:

shared_fs = "virtio-fs"
virtio_fs_daemon = "/opt/kata/libexec/virtiofsd"
valid_virtio_fs_daemon_paths = ["/opt/kata/libexec/virtiofsd"]
virtio_fs_extra_args = ["--fs-bw-burst=20000000","--fs-bw-size=10000000","--thread-pool-size=1","--announce-submounts"]

Ensure the container image used supports virtio-fs; otherwise the Kata container will fail to start.

Summary

Version mismatches are a common source of errors; keep all components aligned.

Install the latest Go version to build yq.

Use Docker‑CE; avoid the default Podman installation from yum install docker.

Kubernetes uses containerd via CRI, so containerd must be properly configured for Kata.

Deploying Kata Containers provides VM‑level isolation with near‑container performance, and virtioFS rate‑limiting protects shared kernel resources under heavy workloads.