How Linux’s ksoftirqd Handles Network Packets: Inside Soft Interrupts

1. Introduction

Previously we discussed the Linux kernel network packet reception process. When a NIC raises a hardware interrupt (IRQ) to notify the CPU of incoming data, the CPU looks up the interrupt table and calls the registered handler, which jumps into the NIC driver. The driver disables further NIC interrupts, indicating it knows data is in memory and telling the NIC to write subsequent packets directly to memory without further CPU notifications, improving efficiency.

Because a hard‑interrupt handler cannot be pre‑empted, a long‑running handler would block other hardware interrupts. Therefore the kernel introduces soft interrupts to move time‑consuming work out of the hard‑interrupt context. The ksoftirqd process is dedicated to handling soft interrupts; when it receives a soft interrupt, it calls the corresponding handler, such as the network stack’s net_rx_action function.

Next we will briefly review the macro‑level packet‑reception flow to locate where soft interrupts fit, then introduce the kernel’s soft‑interrupt mechanism.

2. Macro Process of Packet Reception

Load NIC driver and initialize.

Packet arrives from the external network into the NIC.

NIC uses DMA to copy the packet into a ring buffer in kernel memory.

Hardware interrupt notifies the system that a packet has been received.

Driver invokes NAPI; if polling has not started, it begins polling.

ksoftirqd’s soft interrupt calls NAPI’s poll function to fetch packets from the ring buffer (each CPU runs a ksoftirqd thread registered at boot).

Memory region of the ring buffer is unmapped.

If packet steering or multi‑queue NIC is enabled, packets are distributed across multiple CPUs.

Packet moves from the queue into the protocol layer.

Protocol layer processes the packet.

Packet is placed into the appropriate socket’s receive queue.

3. Soft Interrupts

The kernel’s soft‑interrupt system provides a mechanism to execute code outside the hard‑interrupt (driver) context. While a hard‑interrupt handler runs, it masks part or all new hardware interrupts; the longer the mask, the higher the chance of lost events. Therefore time‑consuming operations should be offloaded from the hard‑interrupt path so that the hard interrupt can finish quickly and re‑enable interrupts.

For the network stack we focus on the soft‑interrupt thread ksoftirqd. Think of the soft‑interrupt system as a set of kernel threads (one per CPU) that run handlers registered via open_softirq(). The ksoftirqd/0 thread shown below runs on CPU 0.

4. ksoftirqd

The soft‑interrupt thread is spawned early in kernel boot. Initialization occurs in kernel/softirq.c:

static struct smp_hotplug_thread softirq_threads = {
      .store              = &ksoftirqd,
      .thread_should_run  = ksoftirqd_should_run,
      .thread_fn          = run_ksoftirqd,
      .thread_comm        = "ksoftirqd/%u",
};

static __init int spawn_ksoftirqd(void)
{
      register_cpu_notifier(&cpu_nfb);

      BUG_ON(smpboot_register_percpu_thread(&softirq_threads));

      return 0;
}
early_initcall(spawn_ksoftirqd);

Two callbacks are registered: ksoftirqd_should_run and run_ksoftirqd. Both are invoked from kernel/smpboot.c as part of the event‑handling loop.

The core soft‑interrupt processing function __do_softirq performs the following steps:

asmlinkage __visible void __do_softirq(void)
{
    unsigned long end = jiffies + MAX_SOFTIRQ_TIME;
    unsigned long old_flags = current->flags;
    int max_restart = MAX_SOFTIRQ_RESTART;
    struct softirq_action *h;
    bool in_hardirq;
    __u32 pending;
    int softirq_bit;

    /*
     * Mask out PF_MEMALLOC so the current task context is borrowed for the
     * softirq. A softirq such as network RX might set PF_MEMALLOC again if the
     * socket is related to swap
     */
    current->flags &= ~PF_MEMALLOC;

    pending = local_softirq_pending(); // get current CPU's softirq pending bits
    account_irq_enter_time(current);

    __local_bh_disable_ip(_RET_IP_, SOFTIRQ_OFFSET); // mark softirq context
    in_hardirq = lockdep_softirq_start();

restart:
    /* Reset the pending bitmask before enabling irqs */
    set_softirq_pending(0); // clear pending bits

    local_irq_enable(); // enable local interrupts

    h = softirq_vec; // point to first softirq handler (HI_SOFTIRQ)

    while ((softirq_bit = ffs(pending))) { // process each pending softirq
        unsigned int vec_nr;
        int prev_count;

        h += softirq_bit - 1; // locate the handler for this softirq
        vec_nr = h - softirq_vec; // softirq number
        prev_count = preempt_count();

        kstat_incr_softirqs_this_cpu(vec_nr);
        trace_softirq_entry(vec_nr);
        h->action(h); // execute the handler
        trace_softirq_exit(vec_nr);
        if (unlikely(prev_count != preempt_count())) {
            pr_err("huh, entered softirq %u %s %p with preempt_count %08x, exited with %08x?
",
                   vec_nr, softirq_to_name[vec_nr], h->action,
                   prev_count, preempt_count());
            preempt_count_set(prev_count);
        }
        h++; // move to next handler
        pending >>= softirq_bit; // shift pending bits
    }

    rcu_bh_qs();
    local_irq_disable(); // disable local interrupts

    pending = local_softirq_pending(); // check for newly raised softirqs
    if (pending) {
        if (time_before(jiffies, end) && !need_resched() && max_restart)
            goto restart;
        wakeup_softirqd(); // wake ksoftirqd thread if needed
    }

    lockdep_softirq_end(in_hardirq);
    account_irq_exit_time(current);
    __local_bh_enable(SOFTIRQ_OFFSET); // exit softirq context
    WARN_ON_ONCE(in_interrupt());
    tsk_restore_flags(current, old_flags, PF_MEMALLOC);
}

When viewing CPU utilization, the si field corresponds to softirqs, measuring the CPU time transferred from hard interrupts.

5. Monitoring

Soft‑interrupt statistics can be read from /proc/softirqs:

6. Summary

Interrupts are an asynchronous event‑handling mechanism that improves system concurrency. An interrupt triggers a handler split into an upper half (hard interrupt) for quick processing and a lower half (soft interrupt) for deferred work. Linux soft interrupts handle networking, timers, scheduling, RCU locks, etc. The /proc/softirqs file lets you observe their activity. Each CPU runs a ksoftirqd thread; when soft‑interrupt frequency is high, the thread’s CPU usage can rise, leading to delayed packet processing, scheduling slowdown, and performance problems.

Reference links:

https://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/

https://www.cnblogs.com/luoahong/p/10815283.html

http://kerneltravel.net/blog/2020/ksoftirqd_ljr/