Worker Pools in Go

In the previous lessons, you learned about goroutines, channels, mutexes, and synchronization. In this lesson, you will learn how to combine these concepts to build a powerful and efficient Worker Pool in Go.

Worker pools are widely used in production systems to control concurrency, optimize resource usage, and process tasks efficiently.

What Is a Worker Pool?

A worker pool is a design pattern where:

• A fixed number of worker goroutines are created
• Tasks are sent to them through a channel
• Each worker processes tasks one by one

Instead of creating unlimited goroutines, worker pools provide controlled concurrency.

Why Worker Pools Are Important

Uncontrolled goroutines can lead to:

• High memory usage
• CPU exhaustion
• System instability

Worker pools help maintain predictable performance in systems like:

• API servers
• Background job processors
• File processing pipelines
• Task queues

Basic Worker Pool Structure

A typical worker pool consists of:

• A jobs channel
• A results channel
• Multiple worker goroutines

Simple Worker Function

Each worker listens for incoming jobs and processes them.

func worker(id int, jobs <-chan int, results chan<- int) {
    for job := range jobs {
        results <- job * 2
    }
}

Each job is doubled and sent to the results channel.

Creating a Worker Pool

Now let’s create a full worker pool with multiple workers.

package main

import "fmt"

func worker(id int, jobs <-chan int, results chan<- int) {
    for job := range jobs {
        fmt.Println("Worker", id, "processing job", job)
        results <- job * 2
    }
}

func main() {
    jobs := make(chan int, 5)
    results := make(chan int, 5)

    for w := 1; w <= 3; w++ {
        go worker(w, jobs, results)
    }

    for j := 1; j <= 5; j++ {
        jobs <- j
    }
    close(jobs)

    for r := 1; r <= 5; r++ {
        fmt.Println("Result:", <-results)
    }
}

How This Works

• Three workers wait for jobs
• Five jobs are sent to the jobs channel
• Workers pick jobs concurrently
• Results are collected safely

Real-World Example: Image Processing Queue

Imagine a system that processes uploaded images:

• Jobs = image IDs
• Workers = image processors
• Results = processed image IDs

Worker pools prevent server overload when many images are uploaded simultaneously.

Controlling the Number of Workers

The number of workers should be based on:

• CPU cores
• Memory limits
• Task complexity

For CPU-bound tasks, fewer workers are recommended. For I/O-bound tasks, more workers may be efficient.

Buffered vs Unbuffered Channels

Buffered channels allow sending jobs without blocking immediately.

jobs := make(chan int, 10)

This improves throughput when job production is faster than processing.

Common Mistakes

• Forgetting to close the jobs channel
• Using too many workers
• Not collecting results properly

Practice Exercises

Exercise 1

Create a worker pool that squares numbers instead of doubling them.

Exercise 2

Increase the number of workers and observe how processing speed changes.

Key Takeaways

• Worker pools manage concurrency efficiently
• They prevent resource exhaustion
• Used heavily in production Go systems
• Combine goroutines and channels effectively

What’s Next?

In the next lesson, you will learn about the Context Package and how it helps manage timeouts, cancellations, and request lifecycles in concurrent Go programs.