Why Does Containerization Slow Down Your App? A Deep Dive into Performance Loss and Network Optimizations

Background

As more companies adopt cloud‑native architectures, monolithic applications are being refactored into micro‑services and deployed with containers orchestrated by Kubernetes. However, after containerization the same workload often shows worse performance than when running on a virtual machine, prompting an investigation.

Benchmark Results

Using wrk to load‑test the application on a VM yields an average response time (RT) of 1.68 ms and a throughput of 716 requests/s , while the CPU is nearly saturated.

After moving the application into containers, the same test reports an average RT of 2.11 ms and a throughput of 554 requests/s , again with the CPU fully utilized.

Performance comparison: VM RT 1.68 ms, container RT 2.11 ms; VM QPS 716/s, container QPS 554/s, indicating a 25 % increase in latency and a 29 % drop in QPS.

Root Cause Analysis

Architecture Differences

The containerized deployment uses Kubernetes with Calico in IPIP mode. Pods communicate via a Service NodePort, which routes through iptables to a Calico‑created virtual interface (calixxx) before reaching the pod. This additional hop lengthens the data path.

Soft‑Interrupt Overhead

Perf profiling shows that after containerization the soft‑interrupt (softirq) usage on the CPU rises significantly compared to the VM baseline. The veth pair used for container‑host communication triggers additional soft interrupts during packet transmission.

Further perf runs confirm a 14 % increase in soft‑interrupt count after containerization.

Code path for veth transmission:

static netdev_tx_t veth_xmit(struct sk_buff *skb, struct net_device *dev) {
    ...
    if (likely(veth_forward_skb(rcv, skb, rq, rcv_xdp)))
        ...
}

static int veth_forward_skb(struct net_device *dev, struct sk_buff *skb,
                            struct veth_rq *rq, bool xdp) {
    return __dev_forward_skb(dev, skb) ?:
           xdp ? veth_xdp_rx(rq, skb) : netif_rx(skb); // soft‑interrupt
}

/* Called with irq disabled */
static inline void ____napi_schedule(struct softnet_data *sd,
                                    struct napi_struct *napi) {
    list_add_tail(&napi->poll_list, &sd->poll_list);
    __raise_softirq_irqoff(NET_RX_SOFTIRQ); // raise soft‑interrupt
}

The veth transmission chain ends with a soft‑interrupt, explaining the higher soft‑interrupt count and the observed performance degradation.

Optimization Strategies

ipvlan L2 Mode

Switching to ipvlan in L2 mode eliminates the overlay network and allows containers to send packets directly through the host’s eth0, removing the veth‑induced soft‑interrupt overhead.

ipvlan L3 Mode

In L3 mode the host acts as a router, enabling containers to communicate across subnets without the extra overlay, further reducing latency.

Cilium CNI

Cilium, an eBPF‑based CNI, optimizes the data path by bypassing iptables and reducing kernel processing overhead. Benchmarks show Cilium achieving higher QPS and lower CPU usage than Calico.

Benchmark excerpt:

The data confirms that Cilium consistently outperforms Calico in both throughput and CPU efficiency.

Conclusion

Containerization brings many benefits—agility, resource utilization, and environment consistency—but it also introduces additional network complexity that can degrade performance, especially for latency‑sensitive workloads. The primary culprit is the extra soft‑interrupt overhead caused by the veth‑based overlay network. By adopting underlay networking solutions such as ipvlan/macvlan or switching to a high‑performance CNI like Cilium, teams can mitigate these issues and fully reap the advantages of cloud‑native deployments.