Module 03 // Deep Dive • Goroutines, Stacks & GMP Work-Stealing Scheduler • ← Back to All Modules
MODULE 03 // ARCHITECTURE & DEEP INTERNALS

Goroutine Internals: 2KB Stacks & The M:N Work-Stealing Scheduler (G, M, P)

In Go, concurrency is powered by cooperative, user-space green threads orchestrated by the runtime scheduler. We achieve millions of concurrent routines using 2KB initial stacks, contiguous stack copying, and lock-free work-stealing queues.

⏱ Reading Time: 18 min ✔ Go 1.26 Verified Runtime
STACK FOOTPRINT
2,048 B
vs 2-8 MB for standard OS pthreads
SCHEDULER MODEL
M:N ENGINE
M OS threads multiplex N Goroutines via P contexts
CONTEXT SWITCH COST
~10-20 ns
~2.2 µs kernel switch penalty
CONCURRENCY SCALE
NumCPU()
P instances dictate true parallel throughput
SEC 01 // ARCHITECTURAL FOUNDATION

The Fundamental Anatomy: 2KB Dynamic Stack vs OS Thread

Traditional operating system threads (pthreads) reserve a static, fixed guard-page memory space—typically 2MB to 4MB. If you spawn 100,000 OS threads, your program consumes up to 400GB of virtual memory before executing a single instruction, swiftly exhausting kernel thread limits and triggering OOM / ENOMEM panics.

POSIX THREAD (*PTHREAD) 2,048 KB FIXED

Allocated with kernel-managed page tables, hardware guard pages, and 16 register dumps.

HARDWARE GUARD PAGE (PROT_NONE)
UNCOMMITTED VIRTUAL MEMORY GAP (16 KB - 2048 KB)
16 KB STACK FRAME DUMP (ACTIVE)
CONTEXT SWITCH: Enters kernel transition, saves 16 GP registers, flushes TLB / L1 cache. Cost: ~2200 ns.
GOROUTINE (RUNTIME G) INITIAL 2,048 B

Allocated out of Go's heap memory cache [mcache]. Enlarged dynamically via contiguous stack copying.

EXEC'S HEAP: SP = runtime.gogo()
ACTIVE CALL FRAME: worker(id=1, wg=*sync.WaitGroup)
INITIAL 2KB BOUNDARY (DYNAMICALLY COPIED)
CONTEXT SWITCH: Saved entirely in user-space. Only 14 general-purpose registers [SP, PC, DX] swapped. Cost: ~10-20 ns.
⚡ Why Go 1.4 Abandoned Split Stacks for Contiguous Stacks

Historically, Go runtimes (<1.4) used segmented "split-stacks" linked via ring-pointers. This caused a catastrophic "hot split problem": calling a function repeatedly at the stack margin triggered continuous allocate and free loops, degrading throughput. Go solved this by adopting contiguous stack copying:

1. 2X BUFFER ALLOC
Compiler injects preamble checking SP vs stackguard0. Triggers runtime.morestack().
2. MEMCPY TO NEW
Runtime allocates brand new 2x contiguous space and copies active frames cleanly.
3. ADJUST PTRS
Pointers pointing into the old stack are safely adjusted via stackmap walks.
SEC 02 // INTERACTIVE RUNTIME SIMULATOR

The GMP Runtime Scheduler & Work-Stealing Engine

Visual telemetry of the triad: Goroutine (G), Machine OS Thread (M), and Logical Processor (P).

STATUS: RUNNING (NORMAL_LOAD)
GLOBAL RUN QUEUE (GRQ) — PROTECTED BY sched.lock 1/61 TICK DISPATCH
G10 (cron.Task) G12 (metrics.Flush) G14 (heartbeat) EMPTY (MUTEX HELD)
PROCESSOR P0 (ID: 0x1) RUNNING
MACHINE M0 (OS THREAD) TID: 4092
G1: HTTP CLIENT (RUNNING)
PC: 0x482991 • Stack: 2048 B
LOCAL RUN QUEUE (LRUQ) [3 OF 256]:
G2 (Database) G3 (Telemetry) G4 (Metrics)
PROCESSOR P1 (ID: 0x2) IDLE
MACHINE M1 (OS THREAD) TID: 4093
G0: RUNTIME SCHEDULER LOOP (IDLE)
Awaiting runnable work
LOCAL RUN QUEUE (LRUQ) [0 OF 256]:
EMPTY EMPTY
Step 1 State: Normal Local Execution. Machine M0 executes Goroutine G1 bound to Processor context P0. M0 fetches directly from P0's 256-slot lock-free local queue with 0 mutex overhead. Global run queue is bypassed.
SEC 03 // PRODUCTION SURVIVAL GUIDE

The 3 Fatal Production Concurrency Traps

These three insidious patterns cause the vast majority of silent memory exhaustion, CPU hangs, and p99 latency spikes in production Go microservices.

TRAP 01 // CRITICAL OOM INCIDENT
Permanent Goroutine Leaks
If a goroutine writes to an unbuffered channel without a listener, or reads from a channel that never receives or closes, it remains alive in _Gwaiting forever.
POST-MORTEM: 20k HTTP requests leaked 20k goroutines, consuming 40MB+ RAM until Linux OOM-killer killed the container.
PRODUCTION FIX: Always use buffered channel (cap 1) for single-result fire-and-forget, or pass context.WithTimeout.
TRAP 02 // SUBTLE BUG DATA RACE
Loop Variable Capture
Before Go 1.22, spawning go func() { print(v) }() inside a for _, v := range slice captured a single address across all iterations.
POST-MORTEM: All goroutines printed the final element value, resulting in duplicate user notifications and corruption.
PRODUCTION FIX: Explicitly pass go func(val string) {}(v) or rely on Go 1.22+ per-iteration variable semantics.
TRAP 03 // SCHEDULER HANG CPU STARVATION
Tight Compute Loops
Prior to Go 1.14, a tight for {} mathematical calculation without function calls could never be preempted by the cooperative scheduler.
POST-MORTEM: A single goroutine monopolized the OS core, preventing garbage collection stop-the-world STW and freezing the service.
PRODUCTION FIX: Go 1.14+ introduced non-cooperative async preemption via POSIX signals (SIGURG).
SEC 04 // RUNTIME DISASSEMBLY

Under The Hood: runtime.schedule()

Every OS thread managed by the Go runtime repeatedly executes the main scheduling loop. Below is the simplified architectural flow directly from src/runtime/proc.go:

go/src/runtime/proc.go // schedule() GO 1.26 RUNTIME KERNEL
func schedule() {
    _g_ := getg() // get current g (g0 stack)
    gp := _g_.m.curg

    // Phase 1: Periodic 1/61 check to prevent Global Run Queue starvation
    if _g_.m.p.ptr().schedtick%61 == 0 && sched.runqsize > 0 {
        lock(&sched.lock)
        gp = globrunqget(_g_.m.p.ptr(), 1)
        unlock(&sched.lock)
        if gp != nil { goto execute }
    }

    // Phase 2: Check Local Run Queue (LRUQ) — completely lock-free
    gp, inheritTime = runqget(_g_.m.p.ptr())
    if gp != nil { goto execute }

    // Phase 3: Work-stealing — steal 50% from random peer processor's queue
    gp, inheritTime = findrunnable() // Polls netpoll, steals 50% from other P

execute:
    // Swaps G & M registers in ~12ns via assembly (gogo)
    execute(gp, inheritTime)
}
Verified Benchmarks: Goroutine vs Operating System Thread (AMD64 Linux)
Operation Time Cost (ns) Memory Allocated Kernel Involvement
Goroutine Creation (go fn) ~42 ns 2,048 B / op None (Userspace)
OS Thread Creation (pthread_create) ~1,400 ns 2,048,000 B (2 MB) Clone Syscall
Goroutine Context Switch ~11 ns 0 B / op None (gogo asm)
OS Thread Context Switch ~1,450 ns 0 B / op Full TLB Flush
INCIDENT POST-MORTEM CHECKPOINT

Work-Stealing & Blocking Syscalls Under Heavy Load

Scenario: You launch 20,000 goroutines on an 8-core CPU (GOMAXPROCS=8). One goroutine initiates a synchronous blocking system call (e.g. waiting on a slow disk file I/O descriptor via an OS handle). What happens to the remaining runnable goroutines waiting in that Processor's local queue?

Interactive Goroutine Timeline Execution

See how Go schedules routines concurrently across multiple hardware lanes:

Sequential
task 1
done
task 2
done
task 3
done
Concurrent (goroutines)
task 1
done
task 2
done
task 3
done
Click “Run comparison” to see the difference.
package main

import (
	"fmt"
	"sync"
	"time"
)

func worker(id int, wg *sync.WaitGroup) {
	defer wg.Done()
	time.Sleep(time.Duration(4-id) * 50 * time.Millisecond)
	fmt.Printf("worker %d done\n", id)
}

func main() {
	var wg sync.WaitGroup
	for i := 1; i <= 3; i++ {
		wg.Add(1)
		go worker(i, &wg)
	}
	wg.Wait()
	fmt.Println("all workers done")
}
worker 3 done worker 2 done worker 1 done all workers done

Notice the order: worker 3 finishes first, then worker 2, then worker 1 — the reverse of the order they were started in, because each one sleeps for a deliberately different amount of time and Go makes no promise about the order independent goroutines finish in. sync.WaitGroup is the standard way to wait for a batch of them: call Add before starting each one, Done (usually deferred) when it finishes, and Wait blocks the caller until the count is back to zero.

Common mistake

Forgetting wg.Wait() (or not passing the WaitGroup by pointer) means main can exit before the goroutines finish. Go does not wait for stray goroutines when main returns — it just ends the program.

Goroutine leaks: forgotten work that never finishes

A goroutine that blocks forever is a leak — it never exits, never gets garbage collected, and holds onto whatever memory and channels it references. The most common cause: sending to a channel that nobody will ever receive from, or receiving from a channel that nobody will ever send to. You can detect leaks with runtime.NumGoroutine():

package main

import (
	"fmt"
	"runtime"
	"time"
)

func leak() {
	ch := make(chan int)
	go func() {
		ch <- 1 // blocks forever: no receiver
	}()
}

func main() {
	fmt.Println("goroutines before:", runtime.NumGoroutine())
	leak()
	time.Sleep(50 * time.Millisecond)
	fmt.Println("goroutines after:", runtime.NumGoroutine())
}
goroutines before: 1 goroutines after: 2
GOMAXPROCS: goroutines and OS threads

By default, GOMAXPROCS equals the number of CPU cores available, which means Go can run that many goroutines truly simultaneously on separate OS threads. You rarely need to change this — the Go scheduler handles multiplexing goroutines onto threads automatically. But knowing it exists helps when reasoning about performance: on a 4-core machine, up to 4 goroutines execute at the exact same instant, while the rest wait their turn. Setting runtime.GOMAXPROCS(1) forces all goroutines onto a single thread, which is sometimes useful for deterministic testing but not for production.

Channels: a synchronous handoff

A goroutine that computes something needs a way to hand the result back. A channel is a typed pipe: one goroutine sends a value in with ch <- value, another receives it with <-ch. Create one with make(chan T). With no extra argument, the channel has zero capacity — it's unbuffered.

From the Go language specification

“If the capacity is zero or absent, the channel is unbuffered and communication succeeds only when both a sender and receiver are ready.” An unbuffered channel holds nothing — it's a rendezvous point, not a mailbox. A send that has no receiver waiting simply blocks until one shows up.

S
sender goroutine
R
main goroutine
package main

import (
	"fmt"
	"time"
)

func main() {
	ch := make(chan string) // unbuffered

	go func() {
		fmt.Println("sender: waiting to send...")
		ch <- "hello"
		fmt.Println("sender: sent, handoff complete")
	}()

	time.Sleep(200 * time.Millisecond)
	fmt.Println("main: about to receive")
	msg := <-ch
	fmt.Println("main: received:", msg)
	time.Sleep(50 * time.Millisecond)
}
sender: waiting to send... main: about to receive main: received: hello sender: sent, handoff complete
Checkpoint Quiz Concurrency Intuition
What happens if goroutine A attempts to send to an unbuffered channel ch <- val while no other goroutine is currently executing a receive <-ch?
Correct! An unbuffered channel has a capacity of 0. It is a synchronous rendezvous point: sending blocks until a receiver is ready, and receiving blocks until a sender is ready. Neither can proceed alone.

Channel directions: restricting send and receive

When you pass a channel to a function, you can restrict what that function is allowed to do with it. A send-only channel (chan<- T) can only be sent to; a receive-only channel (<-chan T) can only be received from. This is a compile-time guarantee — the compiler catches accidental misuse:

package main

import "fmt"

// sendOnly can only send to the channel
func producer(out chan<- int) {
	for i := 0; i < 3; i++ {
		out <- i * 10
	}
	close(out)
}

// receiveOnly can only receive from the channel
func consumer(in <-chan int) {
	for v := range in {
		fmt.Println("received:", v)
	}
}

func main() {
	ch := make(chan int, 3)
	go producer(ch)
	consumer(ch)
}
received: 0 received: 10 received: 20
Direction restrictions encode intent

A function that accepts <-chan int tells callers "I only read from this channel" — it can't accidentally close it or send to it. Similarly, chan<- int says "I only write here." This is the same principle as unexported fields restricting access: use the type system to prevent bugs at compile time rather than at runtime. The range loop in consumer also automatically exits when the channel is closed, which is why the producer calls close(out).

Buffered channels: a queue with capacity

Add a number when creating the channel and you get a buffer: a fixed-size queue that can hold values even before anyone receives them. make(chan T, 3) creates a channel that can hold 3 values. A send only blocks once the buffer is full; a receive only blocks once it's empty.

S
sender
len(ch) = 0   cap(ch) = 3
Try sending a few values, then try sending past capacity 3.
package main

import "fmt"

func main() {
	ch := make(chan int, 3)

	ch <- 1
	fmt.Println("sent 1 -> len:", len(ch), "cap:", cap(ch))
	ch <- 2
	fmt.Println("sent 2 -> len:", len(ch), "cap:", cap(ch))
	ch <- 3
	fmt.Println("sent 3 -> len:", len(ch), "cap:", cap(ch))

	fmt.Println("received:", <-ch, "-> len:", len(ch))
	fmt.Println("received:", <-ch, "-> len:", len(ch))
	fmt.Println("received:", <-ch, "-> len:", len(ch))
}
sent 1 -> len: 1 cap: 3 sent 2 -> len: 2 cap: 3 sent 3 -> len: 3 cap: 3 received: 1 -> len: 2 received: 2 -> len: 1 received: 3 -> len: 0

len(ch) is how many values are sitting in the buffer right now; cap(ch) is the fixed size you gave make. None of these three sends blocked, because the buffer never filled up before the corresponding receive.

Common mistake: sending past capacity

A send to a full buffered channel doesn't error — it blocks until something is received. If nothing ever will be, that's a deadlock, and the Go runtime actually detects this specific case and crashes loudly instead of hanging forever silently:

package main

func main() {
	ch := make(chan int, 3)
	ch <- 1
	ch <- 2
	ch <- 3
	ch <- 4 // buffer full (len == cap == 3), no receiver anywhere -> deadlock
}
fatal error: all goroutines are asleep - deadlock! goroutine 1 [chan send]: main.main() /main.go:8 +0x67 exit status 2

The exact memory address and line offset will vary by machine and Go version — the message text itself, fatal error: all goroutines are asleep - deadlock!, is the stable part, and comes verbatim from the Go runtime source.

Closing channels

A channel can be closed to signal “nothing more is coming.” Closing is done by the sender with close(ch), never the receiver, and only ever once. Receiving from a closed channel never blocks: buffered values already inside come out first, in order, and after that every receive returns the element type's zero value immediately.

package main

import "fmt"

func main() {
	ch := make(chan int, 3)
	ch <- 1
	ch <- 2
	close(ch)

	v, ok := <-ch
	fmt.Println(v, ok)
	v, ok = <-ch
	fmt.Println(v, ok)
	v, ok = <-ch
	fmt.Println(v, ok)

	for v := range ch {
		fmt.Println("range:", v)
	}
	fmt.Println("range over closed+drained channel exited cleanly")
}
1 true 2 true 0 false range over closed+drained channel exited cleanly

The two-value receive form v, ok := <-ch is how you tell a real value from a zero-value-because-closed: ok is true for the first two receives (real sent values), then false once the channel is drained and closed — note the value is 0, the zero value for int, not some sentinel. for range over a channel receives until it's closed and drained, then exits the loop on its own — which is why the final line prints with no "range: ..." lines above it here: there was nothing left to range over.

Common mistake: sending on a closed channel

Unlike receiving, sending on a closed channel is not safe — it panics immediately:

package main

func main() {
	ch := make(chan int)
	close(ch)
	ch <- 1 // should panic: send on closed channel
}
panic: send on closed channel goroutine 1 [running]: main.main() /main.go:6 +0x37 exit status 2
Closed is not the same as nil

A channel variable that was only declared, never make'd, holds nil — and a nil channel behaves completely differently from a closed one. Sending or receiving on a nil channel never panics; it blocks forever. This is a common source of a silent, permanent hang, usually from forgetting to initialize a channel field on a struct.

⚡ Go Channel State Operations Cheat Sheet (The 3x3 Matrix)
Channel State Send (ch <- v) Receive (v, ok := <-ch) Close (close(ch))
nil
var ch chan T
Blocks forever Blocks forever Panics
close of nil channel
Open
make(chan T)
Transmits
or blocks if buffer full
Yields value (ok=true)
or blocks if empty
Closes channel
unblocks waiters
Closed
close(ch) called
Panics
send on closed channel
Drains buffered items,
then zero value (ok=false)
Panics
close of closed channel

for range over a channel: clean iteration

The for range loop you know from slices and maps also works on channels — it receives each value until the channel is closed and drained, then exits the loop automatically. This is the idiomatic way to consume all values from a producer:

package main

import "fmt"

func main() {
	ch := make(chan string, 3)
	ch <- "alpha"
	ch <- "beta"
	ch <- "gamma"
	close(ch)

	for msg := range ch {
		fmt.Println(msg)
	}
	fmt.Println("channel closed and drained")
}
alpha beta gamma channel closed and drained
range needs a closed channel

If you forget to call close(ch), the for range loop will block forever after draining the buffer — it doesn't know no more values are coming. The sender must close the channel to signal completion. To distinguish between receiving a genuine zero value versus receiving from a closed channel in non-loop code, use the two-value receive form: v, ok := <-ch (where ok is false once closed and drained).

Select: waiting on multiple channels

What if a goroutine needs to wait on more than one channel at once, and react to whichever one is ready first? That's what select is for — it looks like a switch, but each case is a channel operation.

1
ch1
?
2
ch2
ch1: 0 ch2: 0
Both channels below are ready at the same instant. Which one wins?

The most common use of select is a timeout, using time.After as one of the cases:

package main

import (
	"fmt"
	"time"
)

func main() {
	ch := make(chan string)

	go func() {
		time.Sleep(300 * time.Millisecond)
		ch <- "result"
	}()

	select {
	case res := <-ch:
		fmt.Println("got:", res)
	case <-time.After(100 * time.Millisecond):
		fmt.Println("timed out")
	}
}
timed out

The goroutine needed 300ms; the timeout case fires at 100ms, so it wins. Now the interesting case — what happens when two cases are ready at the exact same moment:

package main

import "fmt"

func main() {
	ch1 := make(chan int, 1)
	ch2 := make(chan int, 1)
	ch1 <- 1
	ch2 <- 2

	select {
	case v := <-ch1:
		fmt.Println("got from ch1:", v)
	case v := <-ch2:
		fmt.Println("got from ch2:", v)
	}
}
got from ch2: 2 got from ch1: 1 got from ch1: 1 got from ch1: 1 got from ch1: 1 got from ch2: 2 got from ch1: 1 got from ch2: 2 got from ch2: 2 got from ch2: 2
From the Go language specification

“If one or more of the communications can proceed, a single one that can proceed is chosen via a uniform pseudo-random selection.” That output above is the actual result of running that exact program 10 times in a row — not a simulation. Count them: 5 and 5. select does not favor the first case written, or either channel — which is exactly what the multiplexer visual above demonstrates every time you click it.

default: non-blocking channel operations

A select with a default case never blocks — if none of the channel cases are ready, the default runs immediately. This is useful for "try without waiting" patterns like polling or background checks:

package main

import "fmt"

func main() {
	ch := make(chan int)

	select {
	case ch <- 1:
		fmt.Println("sent")
	default:
		fmt.Println("channel not ready, skipping send")
	}

	fmt.Println("continues immediately without blocking")
}
channel not ready, skipping send continues immediately without blocking

The send to ch would block forever (no receiver), but the default case catches it and the program continues. This pattern is common in background goroutines that check for work without blocking on it.

The done channel pattern: broadcasting shutdown

A common need is to tell multiple goroutines to stop at once. Closing a channel broadcasts to every goroutine waiting on it — all of them receive the zero value simultaneously. A chan struct{} (zero-size, no data carried) is the idiomatic choice for a pure signal channel:

package main

import (
	"fmt"
	"sync"
)

func worker(id int, done <-chan struct{}, wg *sync.WaitGroup) {
	defer wg.Done()
	for {
		select {
		case <-done:
			fmt.Println("worker", id, "stopped")
			return
		default:
			// simulate work
		}
	}
}

func main() {
	done := make(chan struct{})
	var wg sync.WaitGroup

	for i := 1; i <= 3; i++ {
		wg.Add(1)
		go worker(i, done, &wg)
	}

	// Broadcast stop signal to all workers
	close(done)
	wg.Wait()
	fmt.Println("all workers stopped")
}
worker 3 stopped worker 1 stopped worker 2 stopped all workers stopped
Why chan struct{} and not chan bool?

A struct{} takes zero bytes of memory — it's the smallest possible type in Go. When you only need a signal (open/closed) and not a value, chan struct{} communicates intent clearly: "this channel carries no data, only a lifecycle signal." It also avoids any ambiguity about what true vs false might mean in the signal context.

Interview Questions

Common interview questions about goroutines, channels, and select — organized by experience level. Click a tab to filter.

Q1. What is a goroutine?

A goroutine is a lightweight function invocation managed by the Go runtime rather than the OS, started by prefixing a call with the go keyword. Many goroutines are multiplexed onto a small number of OS threads by Go's scheduler, so you can create thousands of them cheaply — each starts with only a few KB of stack that grows as needed.

Q2. How do you start a goroutine, and does the program wait for it to finish?

You write go f() to run f concurrently. The calling goroutine does not wait — execution continues immediately to the next statement. If main returns before the goroutine finishes, the goroutine is simply abandoned, so you need an explicit synchronization mechanism like sync.WaitGroup or a channel.

Q3. What is sync.WaitGroup used for?

sync.WaitGroup lets one goroutine wait for a collection of other goroutines to finish. You call Add(n) to set the count of goroutines to wait for, each goroutine calls Done() when finished, and the waiting goroutine calls Wait(), which blocks until the count reaches zero.

Q4. What is a channel in Go?

A channel is a typed conduit created with make(chan T) that goroutines use to send and receive values with the <- operator. Channels provide both data transfer and synchronization between goroutines, avoiding the need for goroutines to share memory directly.

Q5. What is the difference between an unbuffered and a buffered channel?

An unbuffered channel, made with make(chan T), has no capacity — a send blocks until another goroutine performs a matching receive, forming a synchronous handshake. A buffered channel, made with make(chan T, n), has room for n values, so a send only blocks once the buffer is full.

Q6. How do you close a channel, and why would you?

You call close(ch) to signal that no more values will be sent on it. It is typically used to tell one or more receiving goroutines that a stream of values is complete, most commonly so a range loop over the channel can terminate.

Q7. What does the "comma ok" idiom tell you when receiving from a channel?

Writing v, ok := <-ch receives a value and sets ok to true if the value came from an active send, or false if the channel is closed and drained, in which case v is the zero value of the channel's type. It's the only reliable way to distinguish a real zero value from a closed channel.

Q8. What happens when you range over a channel?

for v := range ch receives values from ch one at a time, blocking between each until a value arrives, and the loop exits automatically when the channel is closed and all buffered values have been received. It's a convenient shorthand for repeated receives with the comma-ok check.

Q9. What is a select statement used for?

select lets a goroutine wait on multiple channel operations at once, proceeding with whichever case is ready first. It's the primary tool for multiplexing several channels — for example waiting on a result channel and a timeout channel simultaneously.

← Back to the roadmap