Why Does Containerization Slow Down My App? A Deep Dive into VM vs Container Performance

Background

When migrating a monolithic application to a cloud‑native stack, the workload is often moved from virtual machines (VMs) to containers orchestrated by Kubernetes. An initial observation showed that the same service exhibited higher latency and lower throughput after containerization.

Benchmark Results

Load testing was performed with wrk under identical hardware and CPU‑saturated conditions.

VM deployment : average response time (RT) = 1.68 ms, throughput = 716 queries per second (QPS).

Container deployment (K8s + Calico IPIP overlay, NodePort service) : average RT = 2.11 ms, throughput = 554 QPS.

The containerized deployment is ~25 % slower in latency and ~29 % lower in QPS.

Root‑Cause Analysis

Network architecture differences

In the container scenario traffic follows the path: NodePort → iptables → Calico’s virtual interface → pod. This introduces a veth pair, adding extra hops and increasing the number of soft‑interrupts (softirqs) processed by the kernel.

Soft‑interrupt overhead

Perf measurements revealed a ~14 % increase in softirq handling for the container case. The kernel code path responsible for packet transmission is:

static netdev_tx_t veth_xmit(struct sk_buff *skb, struct net_device *dev)
{
    ...
    if (likely(veth_forward_skb(rcv, skb, rq, rcv_xdp)))
        ...
}

static int veth_forward_skb(struct net_device *dev, struct sk_buff *skb,
       struct veth_rq *rq, bool xdp)
{
    return __dev_forward_skb(dev, skb) ?: xdp ?
        veth_xdp_rx(rq, skb) :
        netif_rx(skb); // soft‑interrupt handling
}

static inline void ____napi_schedule(struct softnet_data *sd,
         struct napi_struct *napi)
{
    list_add_tail(&napi->poll_list, &sd->poll_list);
    __raise_softirq_irqoff(NET_RX_SOFTIRQ); // raise softirq
}

This full kernel stack traversal for each packet explains the higher softirq count and the observed latency degradation.

Optimization Strategies

Underlay networking (macvlan / ipvlan)

Replacing the Calico IPIP overlay with an underlay solution removes the veth pair. In ipvlan L2 mode containers transmit directly through the host’s eth0, eliminating the softirq path. In ipvlan L3 mode the host acts as a router, enabling cross‑subnet communication while still avoiding the overlay overhead.

eBPF‑based CNI (Cilium)

Cilium implements a high‑performance data path using eBPF, bypassing iptables and reducing kernel processing. Benchmarks show that Cilium achieves higher QPS and lower CPU utilization compared with Calico, making it suitable for latency‑sensitive workloads.

Conclusion

Containerization introduces additional network hops and soft‑interrupt processing that can noticeably degrade performance. Selecting a CNI that avoids overlay networking (e.g., macvlan or ipvlan) or that leverages eBPF (e.g., Cilium) mitigates these effects and restores latency and throughput comparable to VM deployments.