How Go’s M‑P‑G Scheduler Optimizes System Calls and Reduces Blocking

Go's MPG scheduling model is a key feature that ensures the high efficiency of the Go language; this article provides a detailed introduction to its design.

Preface

All programs running on UNIX ultimately use C system calls to communicate with the kernel. In Go, these calls are wrapped in the syscall package and implemented in assembly.

Asynchronous system calls detach G from the netpoller, while synchronous calls detach G from M, illustrating the elegance of G‑P‑M compared to G‑M.

Blocking

In Go, blocking occurs in four main scenarios:

Goroutine blocks due to atomic, mutex, or channel operations; the scheduler switches it out and runs other Goroutines from the local run queue (LRQ).

Network or I/O operations block a Goroutine. Go's netpoller (using kqueue, epoll, or iocp) handles these without blocking M, allowing M to run other Goroutines from the LRQ.

Blocking system calls (e.g., file I/O) where the netpoller cannot help; the blocked Goroutine's M is also blocked, prompting the scheduler to introduce another M.

Sleep operations that block M; the sysmon thread monitors long‑running Goroutines and can preempt them.

System Calls

Go wraps all OS system calls using assembly in functions like Syscall and RawSyscall. Below is the Linux 386 implementation of Syscall:

TEXT ·Syscall(SB),NOSPLIT,$0-28
    CALL runtime·entersyscall(SB)
    MOVL trap+0(FP), AX // syscall entry
    MOVL a1+4(FP), BX
    MOVL a2+8(FP), CX
    MOVL a3+12(FP), DX
    MOVL $0, SI
    MOVL $0, DI
    INVOKE_SYSCALL
    CMPL AX, $0xfffff001
    JLS ok
    MOVL $-1, r1+16(FP)
    MOVL $0, r2+20(FP)
    NEGL AX
    MOVL AX, err+24(FP)
    CALL runtime·exitsyscall(SB)
    RET
ok:
    MOVL AX, r1+16(FP)
    MOVL DX, r2+20(FP)
    MOVL $0, err+24(FP)
    CALL runtime·exitsyscall(SB)
    RET

References: Golang – Scheduler analysis; Go: Goroutine, OS Thread and CPU Management.

Go optimizes system calls—blocking or not—by wrapping them in the runtime. This wrapper automatically dissociates the P from the thread M and allows another thread to run.

Asynchronous System Calls

Using the netpoller for network system calls prevents the Goroutine from blocking M, allowing M to continue executing other Goroutines from the LRQ without creating a new M, thus reducing OS scheduling load.

Illustration of G1 moving to the netpoller for an async network call, while M processes other Goroutines:

After the async call completes, G1 returns to the P’s LRQ and can resume execution.

Synchronous System Calls

When G1 performs a synchronous system call, it blocks M1. The scheduler separates M1 from its P, moves G1 away, and introduces a new M2 to serve the P.

After the blocking call finishes, G1 returns to the LRQ and can be executed again; M1 remains idle for future use.

sysmon Goroutine

Similar to kernel threads like pdflush or kswapd0, Go’s runtime includes a sysmon goroutine that runs on an M without needing a P. Every 10 ms it checks the runtime for abnormal states.

sysmon tasks include:

Deadlock detection (runtime.checkdead).

Timer management – fetching the next timer to fire.

Fetching ready Goroutines from netpoll.

Preemptive scheduling: if a Goroutine runs on the same M for over 10 ms, sysmon sets its preempt flag to true. On the next function call, the Goroutine checks this flag, detaches from M, and is placed on the global queue. Preemption does not occur in a tight for{} loop without function calls.

Triggering garbage collection when conditions are met.

Printing scheduling info, returning memory, and other periodic tasks.

When preemption is needed, sysmon signals the P with SIGURG, causing the signal handler to invoke the gsignal goroutine, which maps the signal to M and stops the running Goroutine.

Source: [email protected] – qiankunli.github.io/2020/11/21/goroutine_system_call.html