Multi‑Kernel OS Research Overview (Part 2): Strengths and Weaknesses

Introduction

Traditional operating systems such as Windows and Linux were originally designed for single‑processor machines. As core counts grew, the monolithic design faced scalability limits due to lock‑protected shared kernel data accessed concurrently by multiple CPUs. Multi‑kernel operating systems treat a machine as a network of independent cores, each running its own kernel instance and coordinating via explicit message passing rather than implicit shared‑memory synchronization. This design aims to reduce costly cache‑coherence traffic and lock contention, improving scalability and performance on many‑core and heterogeneous processors. The following sections analyze several notable research projects and their respective advantages and disadvantages.

Notable Multi‑Kernel Operating Systems

Barrelfish (ETH Zurich / Microsoft Research)

First released in 2009, Barrelfish was one of the earliest multi‑kernel systems. It replicates kernel data structures (e.g., scheduler queues, page tables) on each core instead of sharing them, and cores communicate via asynchronous messages over an interconnect treated as a network link. The designers articulated three principles: (1) make all inter‑core communication explicit, (2) keep the OS structure independent of specific CPU architectures, and (3) treat OS state as replicated rather than globally shared. Experiments on Intel’s 48‑core SCC and heterogeneous CPUs showed competitive performance and better scalability as core counts increase. Although never deployed in production, Barrelfish demonstrated the feasibility of the multi‑kernel approach and inspired many later projects.

fos – Decomposed Operating System (MIT CSAIL)

Developed around the same time as Barrelfish, fos decomposes the OS into fine‑grained services (e.g., memory allocation, scheduling, file system) that run as server processes distributed across cores. Each core may host a server instance for a given service, eliminating time‑multiplexing of multiple OS tasks on a single core and improving locality. The model resembles cloud‑style distributed services: requests such as memory allocation are sent to the local core’s server, which may synchronize with other servers if needed. Built on a tiny microkernel that provides basic messaging and protection, fos runs most OS functionality in user‑space servers. The design scales to over 1 000 cores because each service can increase its server thread count with core count, and it isolates OS services on separate cores to avoid competition with application threads. Although a research prototype, fos introduced the message‑driven, decomposed OS service concept that has influenced subsequent multi‑kernel research.

Popcorn Linux (Virginia Tech)

Popcorn, introduced in the mid‑2010s, implements a “replicated kernel” approach on top of Linux. Multiple Linux kernel instances are launched—typically one per core or core group—and coordinated to appear as a single system. The runtime and compiler infrastructure allow processes and threads to migrate across heterogeneous ISA cores (e.g., Intel Xeon + Xeon Phi) without rewriting applications. Experiments showed up to a 52 % performance improvement over native Linux on heterogeneous cores, attributed to better load balancing and the elimination of slower off‑load frameworks. Popcorn also uses explicit inter‑core communication to maintain consistency, similar to Barrelfish, and leverages Linux’s kexec mechanism to start multiple kernels.

Twin‑Linux (LinSysSoft Technologies)

Twin‑Linux, released in 2010, is an early multi‑kernel experiment for Intel Core 2 Duo platforms. It assigns an independent Linux kernel instance to each core by modifying the GRUB bootloader to create an SMP‑compatible multi‑kernel boot sequence. The BSP and AP are treated as completely separate compute units, each with its own OS instance. Resource partitioning (memory, PCI devices, network adapters) ensures isolation; for example, on a 2 GB Core 2 Duo system each kernel receives 1 GB and a shared memory region is used for inter‑processor interrupts (IPI). By dedicating one core to network I/O and another to storage I/O, Twin‑Linux achieves effective load separation and cache‑efficiency gains, demonstrating that multi‑kernel concepts can be realized by extending existing boot infrastructure rather than redesigning the OS from scratch.

HPC Multi‑Kernel Co‑Design (IHK/McKernel)

In high‑performance computing, hybrid multi‑kernel designs have emerged to meet scalability and jitter requirements. IHK/McKernel, developed for the Fugaku supercomputer, runs a lightweight kernel (McKernel) on a subset of cores for compute‑intensive threads while Linux runs on the remaining cores for I/O and system services. The two kernels cooperate via shared memory or messaging to keep application state consistent. Benchmarks on Intel Xeon Phi‑based systems showed significant reductions in OS jitter and overhead, and on Fugaku McKernel achieved average speedups of ~4 % (up to 25 % in some cases) compared with finely tuned Linux. This approach balances raw performance with full Linux compatibility.

mOS (Intel Corporation)

Proposed in 2014, mOS targets extreme‑scale HPC by running a full‑weight kernel (FWK, typically Linux) alongside a lightweight kernel (LWK) on the same node. A hierarchical system‑call routing mechanism directs performance‑critical calls to the LWK, latency‑sensitive calls to Linux, and high‑latency or resource‑intensive calls to a remote OS node. Resource partitioning (static/dynamic memory split, ACPI‑based CPU limits, device specialization) isolates the two kernels. Techniques such as the “Cooperative Agent Kernel Extension” (CAKE) provide clean resource partitioning and safe sharing. Experiments show modest performance gains while preserving Linux’s full functionality.

RainForest (Institute of Computing Technology, Chinese Academy of Sciences)

RainForest, introduced in 2016, implements a horizontal OS model that separates the system into a supervisor (resource provider) and multiple sub‑OS instances (resource users). Each sub‑OS is an independent OS instance with its own physical resources, capable of rapid creation, destruction, and resizing. The design enforces three principles: role separation, elastic physical resource allocation, and limited state sharing. Implementation modifies Linux’s pa and va macros to remap sub‑OS memory, and uses a two‑stage trampoline for booting. Components such as FICM (fast inter‑core communication), supcon, subOScon, and RFcom provide communication and virtualization support. In tail‑latency benchmarks (200 ms limit), RainForest outperformed Linux, LXC, and Xen by 25.0 %–42.8 % throughput and improved CPU utilization by 16.6 %–25.1 %. On Spark workloads it achieved speedups of 1.43×, 1.16×, and 2.60× over Linux, LXC, and Xen respectively, using roughly 15 000 lines of code added to the Linux kernel.

Other Related Research

K2 (2014) targets heterogeneous coherence domains on mobile SoCs (e.g., ARM big.LITTLE) by running independent Linux kernels per domain and using a “shared‑most” model to keep most services synchronized, improving energy efficiency while preserving the existing mobile application model. Security‑focused work such as Composite demonstrates that isolated kernels on different cores can provide strong security separation, even if not labeled as multi‑kernel. Earlier distributed and cluster OS projects (Amoeba, MOSIX, Hive OS) share similar ideas of running independent kernels for fault isolation.

Conclusion

Over the past decade, multi‑kernel research has progressed from feasibility demonstrations (Barrelfish, fos) to heterogeneous performance gains (Popcorn, Twin‑Linux) and practical HPC deployments (IHK/McKernel, mOS, RainForest). While academic results show clear scalability, reliability, and security benefits, industrial adoption remains limited. The authors call for collaboration among hardware vendors, software companies, and the research community to bridge this gap and bring multi‑kernel Linux systems into production.