Mastering Go Concurrency: Goroutines, Channels, and a Real‑World Web Crawler

Goroutines: Lightweight Threads

Goroutines are the core of Go's concurrency model. They are managed by the Go runtime with a small initial stack (typically a few kilobytes) and have very low creation and destruction overhead, allowing programs to run thousands of concurrent tasks.

Starting a Goroutine

package main

import (
    "fmt"
    "time"
)

func say(s string) {
    for i := 0; i < 5; i++ {
        time.Sleep(100 * time.Millisecond)
        fmt.Println(s)
    }
}

func main() {
    go say("world") // runs in a new goroutine
    say("hello")    // runs in the main goroutine
}

The go keyword before a function call launches that function in a separate goroutine, enabling concurrent execution with the caller.

Goroutine Scheduling

Go uses an M:N scheduler: many goroutines are multiplexed onto a smaller set of operating‑system threads. The scheduler maps goroutines to logical processors (GOMAXPROCS) and then binds those processors to OS threads, balancing work automatically.

Channels: Communication Between Goroutines

Channels provide a type‑safe way for goroutines to exchange data without explicit locks, reducing race conditions.

Creating and Using a Channel

ch := make(chan int) // unbuffered channel of ints

Data is sent and received with the arrow operator <-:

ch <- v        // send value v to channel ch
v := <-ch      // receive a value from ch and assign to v

Blocking Behavior of Channels

Unbuffered channels block the sending goroutine until a receiver is ready, and block the receiving goroutine until a sender provides a value. This synchronization guarantees that data is transferred safely.

Practical Example: Concurrent Web Crawler

The program below launches a goroutine for each URL, fetches the page, and reports the status and latency through a shared channel.

package main

import (
    "fmt"
    "net/http"
    "time"
)

func fetch(url string, ch chan<- string) {
    start := time.Now()
    resp, err := http.Get(url)
    if err != nil {
        ch <- fmt.Sprintf("error: %s", err)
        return
    }
    ch <- fmt.Sprintf("%s, %s, %dms", url, resp.Status, time.Since(start).Milliseconds())
}

func main() {
    urls := []string{
        "https://www.google.com",
        "https://www.baidu.com",
        "https://www.amazon.com",
    }
    ch := make(chan string)
    for _, url := range urls {
        go fetch(url, ch) // start a goroutine per URL
    }
    for range urls {
        fmt.Println(<-ch) // receive and print each result
    }
}

Conclusion

By leveraging goroutines and channels, Go offers concise yet powerful primitives for concurrent programming. They simplify parallel execution, synchronize data exchange, and enable developers to build efficient, reliable applications that fully utilize modern multi‑core hardware.