Skip to content
OS Interview Mastery: 7 Core Concepts from Processes to Zero-Copy with Diagrams & Java Practice
Fundamentals 45 min read

OS Interview Mastery: 7 Core Concepts from Processes to Zero-Copy with Diagrams & Java Practice

This comprehensive guide covers seven high-frequency OS interview topics — processes vs threads, IPC mechanisms, thread synchronization, deadlock conditions, virtual memory paging, CPU scheduling algorithms, and zero-copy techniques — with visual diagrams, comparison tables, Java code examples, and production pitfalls to move beyond rote memorization.

Rain's Path to Java Mastery
Rain's Path to Java Mastery
Rain's Path to Java Mastery

Process vs Thread

Process is the basic unit of resource allocation; thread is the basic unit of CPU scheduling. In Linux, both are represented by task_struct; threads are simply processes that share more resources (memory, file descriptors, signal handlers) via clone(CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND). Thread switching is cheaper because it avoids page-table switches and TLB flushes.

Key Differences

Core Role : Process = resource allocation unit; Thread = CPU scheduling unit.

Resource Ownership : Process has independent address space and resources; Thread shares process resources, owns only stack and registers.

Switch Overhead : Process high (full context, page tables); Thread low (stack pointer, program counter).

Communication : Process complex (pipes, queues, shared memory, sockets); Thread simple (shared variables, locks).

Robustness : Process high (isolation); Thread low (one crash kills process).

Concurrency Granularity : Process coarse; Thread fine.

Lifecycle : Process independent; Thread bound to process.

Java Mapping

HotSpot uses 1:1 kernel-thread model: each Thread maps to an OS thread. Private areas: VM stack, native stack, PC register. Shared areas: heap, method area, constant pool, file handles, sockets. One JVM = one OS process with main, GC, JIT compiler threads. Java 21 introduces virtual threads (user-mode scheduled) for I/O-bound high concurrency.

Inter-Process Communication (IPC)

Five core mechanisms compared:

Pipe

Core Principle : Kernel ring buffer.

Direction : Half-duplex.

Speed : Slow.

Pros : Simple, built-in sync.

Cons : Byte stream only, limited size.

Typical Use : Shell pipelines, parent-child.

Message Queue

Core Principle : Kernel message list.

Direction : Full-duplex.

Speed : Medium.

Pros : Async, multi-producer/consumer, typed.

Cons : Copy overhead, size limits.

Typical Use : Async decoupling, task dispatch.

Shared Memory

Core Principle : Physical memory mapped to multiple virtual spaces.

Direction : Full-duplex.

Speed : Fastest.

Pros : Zero-copy data transfer.

Cons : No built-in sync; needs semaphores.

Typical Use : High-volume data, video, DB cache.

Semaphore

Core Principle : Kernel counter.

Direction : No data.

Speed : Very fast.

Pros : Lightweight sync/mutex.

Cons : Cannot transfer data.

Typical Use : Critical section, shared-memory sync.

Socket

Core Principle : Network stack / Unix domain.

Direction : Full-duplex.

Speed : Slow.

Pros : Universal, cross-host, TCP/UDP.

Cons : High overhead, framing issues.

Typical Use : Microservices, cross-machine.

Selection Guide : Local high-throughput → shared memory + semaphore; local async decoupling → message queue; simple mutex → semaphore/mutex; cross-machine → socket; simple parent-child → anonymous pipe.

Java: PipedInputStream/OutputStream (thread-level, not OS pipe), MappedByteBuffer / FileChannel.map for shared memory, java.util.concurrent.Semaphore (backed by futex/pthread), Socket/ServerSocket/DatagramSocket and Netty for sockets.

Thread Synchronization

Mutex

Only one thread holds lock; operations lock() / unlock(). Properties: atomicity, mutual exclusion, non-preemptive. Java: synchronized, ReentrantLock. Modern mutexes block (yield CPU) rather than spin. Reentrant locks allow same thread to re-acquire. Priority inversion solvable via priority inheritance.

Condition Variable

Works with mutex; allows thread to release lock and wait for condition. Operations: await() / signal() / signalAll(). Must use while (!condition) await() to handle spurious wakeups. Java: java.util.concurrent.locks.Condition. Use case: producer-consumer, thread-pool queues.

Semaphore

Maintains permit counter; P(wait) decrements, V(signal) increments. Binary semaphore = mutex; counting semaphore = resource pool. No ownership — any thread can signal. Java: java.util.concurrent.Semaphore. Use case: DB connection pool, rate limiting.

Read-Write Lock

Multiple readers or single writer. Rules: read-read share, read-write exclusive, write-write exclusive. Java: ReentrantReadWriteLock. Write-preference prevents writer starvation. Read-lock upgrade to write-lock generally unsupported (deadlock risk). Overhead higher than mutex; only beneficial when reads dominate.

Comparison

Mutex : Concurrency = 1 thread; Scenario = general, short critical sections; Pros = simple, clear semantics; Cons = low concurrency, no condition wait.

Condition : Concurrency = low; Scenario = thread cooperation; Pros = precise wakeup, avoids busy-wait; Cons = must pair with mutex, complex.

Semaphore : Concurrency = N threads; Scenario = limit concurrent access; Pros = flexible count control; Cons = error-prone, can deadlock.

ReadWriteLock : Concurrency = high (read-heavy); Scenario = read-heavy workloads; Pros = high read throughput; Cons = writer blocks all readers.

Best practices: default to synchronized; read-heavy → ReentrantReadWriteLock; limit concurrency → Semaphore; coordination → Condition. await() always in while. StampedLock offers optimistic reads. CountDownLatch / CyclicBarrier built on AQS.

Deadlock

Four Necessary Conditions

Mutual Exclusion : Resource exclusive to one process (analogy: single toilet stall).

Hold and Wait : Hold one, request another (analogy: hold chopsticks, grab bowl).

No Preemption : Cannot forcibly take resource (analogy: can't unplug someone's PC).

Circular Wait : Cycle of waiting processes (analogy: A waits B, B waits C, C waits A).

Strategies

Prevention : Break one condition (e.g., ordered locking). Pros: simple, reliable. Cons: low utilization, poor concurrency. Use case: high-reliability systems.

Avoidance : Banker's algorithm: pre-check safety. Pros: higher utilization. Cons: needs max demand upfront, complex. Use case: banking, known workloads.

Detection & Recovery : Resource-allocation graph cycle detection; kill/preempt/rollback. Pros: highest utilization. Cons: detection cost, possible data loss. Use case: low deadlock probability.

Banker's algorithm data structures: Available[m], Max[n][m], Allocation[n][m], Need[n][m] = Max - Allocation. Safe state = exists sequence where all finish.

Java: nested locks common cause. Detect with jstack <pid> (prints "Found one Java-level deadlock"). Prevention: avoid nested locks, fixed lock ordering, tryLock(timeout).

Virtual Memory & Paging

Why Virtual Memory

Memory expansion: disk backs physical memory.

Isolation: each process gets private virtual address space.

Protection: per-page permissions (R/W/X).

Paging Mechanics

Virtual space → fixed-size pages (typically 4 KiB). Physical space → page frames of same size. Mapping via multi-level page tables (x86-64: 4 levels PGD→PUD→PMD→PTE). TLB (Translation Lookaside Buffer) caches recent translations; >95% hit rate due to locality.

Address Translation Example

Page size 4 KiB (2^12). Virtual address 0x12345678: offset = low 12 bits 0x678; page number = high bits 0x12345. Walk page tables (or TLB hit) to get frame number, combine with offset.

Page Fault Handling

OS pauses process.

Loads page from swap into free frame.

Updates page table.

Resumes process.

Segmentation vs Paging

Unit : Paging = fixed page; Segmentation = variable segment.

Goal : Paging = virtual memory, utilization; Segmentation = logical structure, sharing/protection.

Address Space : Paging = 1D; Segmentation = 2D (segment + offset).

Fragmentation : Paging = internal; Segmentation = external.

Transparent to Programmer : Paging = yes; Segmentation = no.

Java Relevance

JVM heap oversizing → swap thrashing → GC pauses spike. Fix: vm.swappiness=0/1, disable swap, or -XX:+AlwaysPreTouch to pre-fault all pages at startup.

Zero-copy via FileChannel.map (mmap) and transferTo (sendfile) leverages page-cache mapping.

Huge pages (2 MiB/1 GiB) reduce page-table entries, improve TLB coverage. Enable with -XX:+UseLargePages.

CPU Scheduling

Goals & Metrics

CPU Utilization : Busy time % (analogy: barista busy-ness).

Throughput : Processes/time (analogy: cups/hour).

Turnaround Time : Submit to finish (analogy: order to pickup).

Waiting Time : Time in ready queue (analogy: queue time).

Response Time : Submit to first response (analogy: order to start).

Algorithm Comparison

FCFS : Non-preemptive. Strength: simple, fair. Weakness: convoy effect, high turnaround. Best for: simple batch.

SJF : Non-preemptive. Strength: optimal avg turnaround. Weakness: starvation, need runtime prediction. Best for: batch with known lengths.

HRRN : Non-preemptive. Strength: balances long/short. Weakness: per-scheduling computation cost. Best for: fair batch.

Round Robin : Preemptive. Strength: fast response, fair. Weakness: quantum tuning trade-off. Best for: time-sharing.

Priority : Preemptive/Non. Strength: differentiates urgency. Weakness: starvation, priority inversion. Best for: real-time.

MLFQ : Preemptive. Strength: adaptive, balances all. Weakness: complex, possible starvation. Best for: modern general-purpose OS.

MLFQ: multiple queues with decreasing priority & increasing quantum. New jobs enter top queue (RR). If quantum expires → demote. If blocks for I/O → promote (rewards I/O-bound). Only run lower queues when higher empty.

Linux CFS

Completely Fair Scheduler uses red-black tree keyed by vruntime (virtual runtime). Task with smallest vruntime runs next. No fixed time slice; dynamically computes fair share. Analogy: slicing pizza by need.

Common Interview Questions

Why MLFQ? Solves short-job response, long-job throughput, I/O responsiveness simultaneously.

Quantum size? 10–100 ms; keep context-switch overhead <1% CPU.

Java scheduling? Preemptive priority (1–10) but final decision by OS kernel (Linux CFS, Windows similar).

Zero-Copy: mmap & sendfile

Traditional I/O Cost

Disk → kernel buffer (DMA) → user buffer (CPU copy, ctx switch) → socket buffer (CPU copy, ctx switch) → NIC (DMA). 4 copies, 2 context switches . CPU spends cycles copying unchanged data.

mmap

Maps kernel buffer into user virtual address space. User reads/writes directly. 3 copies (1 CPU + 2 DMA), 2 context switches ( mmap + write). Java: FileChannel.map() → MappedByteBuffer. Pros: random access, data modification. Cons: page faults, virtual address pressure, tricky cleanup ( Cleaner /GC).

sendfile

Kernel-internal transfer from file to socket. Linux 2.1: 3 copies (1 CPU + 2 DMA), 1 context switch . Linux 2.4+ with SG-DMA: 0 CPU copies, 2 DMA, 1 context switch — true zero-copy. Java: FileChannel.transferTo() / transferFrom(). Pros: maximum throughput, no user-space involvement. Cons: file→network only, no data modification, needs NIC SG-DMA support.

Comparison

Copies : mmap = 2 DMA + 1 CPU; sendfile = 2 DMA (Linux 2.4+).

Context Switches : mmap = 2; sendfile = 0.

Data Modification : mmap supported; sendfile not supported.

Random Access : mmap supported; sendfile not supported.

Best For : mmap = large files, need processing; sendfile = static file streaming.

Framework Usage

Netty: FileRegion (wraps transferTo).

Kafka: transferTo for log segments; mmap for index files (binary search).

RocketMQ: both mmap and sendfile.

Pitfalls

mmap: large files can exhaust virtual address space; MappedByteBuffer release non-deterministic.

sendfile: cannot encrypt/compress on the fly; requires SG-DMA for zero CPU copy.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

DeadlockZero-CopyOperating SystemsVirtual MemoryIPCInterview PreparationCPU SchedulingJava Concurrency
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.