Thread Pools: Reuse Workers Instead of Spawning Threads
How a thread pool works: a fixed set of workers pulling tasks from a queue. Java tests show 12 tasks on 4 threads, what a bounded queue does when full, and the two common rejection policies.
A thread pool starts a fixed number of worker threads once and feeds them tasks through a queue, instead of creating a thread per task. That caps memory and avoids repeated start-up cost. Interviewers expect overload handling: with a bounded queue that is full, a rejection policy applies, such as AbortPolicy throwing an exception or CallerRunsPolicy creating back-pressure.
A thread pool starts a fixed number of worker threads once and feeds them tasks through a queue, instead of creating a new thread for every task. That caps resource use, avoids repeated start-up cost, and gives you one place to decide what happens when work arrives faster than it can be done.
- A thread pool has three parts: worker threads created once that loop taking and running tasks, a queue of waiting tasks, and a policy for a full queue or shutdown.
- In a Java test, 12 tasks submitted to Executors.newFixedThreadPool(4) ran on exactly 4 threads, with at most 4 running at once and the rest queued.
- A ThreadPoolExecutor with 2 workers, a queue of 2 and AbortPolicy accepted 4 of 6 blocking tasks and rejected 2 with RejectedExecutionException.
- CallerRunsPolicy runs an overflow task on the submitting thread, which slows the producer to match the workers and so creates back-pressure.
- A thread pool for CPU-bound work should be close to the number of CPU cores, while I/O-bound work benefits from more threads, with a common estimate of cores times (1 plus wait time divided by compute time).
- A pool-induced deadlock happens when tasks wait on other tasks queued in the same pool and every worker is waiting, so no task can run.
01The problem with one thread per task
The simplest way to run work concurrently is to start a new thread for each task. It works for a demo and fails under load. Each thread needs its own stack memory and an OS scheduling entry, and starting one takes time. When thousands of tasks arrive together, the process can run out of memory or spend more time switching between threads than doing work.
A thread pool fixes this by separating two things: how many threads exist, and how many tasks there are. The number of threads stays small and fixed. Tasks wait in a queue until a worker is free.
02The moving parts
A pool has three pieces. Worker threads, created once, each running a loop: take a task from the queue, run it, repeat. A queue that holds tasks waiting for a worker. And a policy for what to do when the queue is full or the pool is shut down.
In Java the standard class is ThreadPoolExecutor, and Executors.newFixedThreadPool(n) builds a simple one. Python has concurrent.futures.ThreadPoolExecutor, and most languages have an equivalent. Writing your own is a good exercise, but production code should use the library version, which handles shutdown, exceptions and edge cases already.
03Fixed workers, many tasks
ExecutorService pool = Executors.newFixedThreadPool(4);
for (int i = 0; i < 12; i++) {
pool.submit(() -> {
// record the thread name and how many tasks run at once
try { Thread.sleep(30); } catch (InterruptedException e) { }
});
}I ran this with counters for the thread names seen and the number of tasks active at any moment. The output was tasks=12 threads used=4 max at once=4. Twelve tasks were handled by the same four threads, and no more than four ever ran simultaneously. The other eight waited in the queue. That is the whole point: throughput is bounded by the pool size, and memory use stays flat however many tasks are submitted.
04Bounded queues and rejection
An unbounded queue hides overload. Tasks pile up in memory until the process runs out. A bounded queue forces a decision. The test used 2 workers and a queue of 2, and submitted 6 tasks that all block:
new ThreadPoolExecutor(2, 2, 0, TimeUnit.SECONDS,
new ArrayBlockingQueue<>(2),
new ThreadPoolExecutor.AbortPolicy());The result was accepted=4 rejected=2. Two tasks went to the workers, two sat in the queue, and the last two were rejected with a RejectedExecutionException. The caller finds out immediately and can retry, shed load or return an error. Failing fast is usually better than letting latency grow without limit.
05Back-pressure with CallerRuns
Java offers other policies. CallerRunsPolicy runs the overflow task on the thread that submitted it. In a test with 1 worker and a queue of 1, the third task printed overflow task ran on: main. Because the submitting thread is now busy running a task, it cannot submit more for a while. That slows the producer to match the consumers, which is called back-pressure.
DiscardPolicy silently drops the task and DiscardOldestPolicy drops the oldest queued one. Both lose work without telling you, so use them only when losing a task is acceptable, such as refreshing a non-critical cache.
06Sizing the pool
For CPU-bound tasks (computation with no waiting), a pool close to the number of CPU cores is right. More threads just add switching overhead. For tasks that spend most of their time waiting on a network or disk, more threads than cores helps, because waiting threads use no CPU. A common estimate is cores times (1 plus wait time divided by compute time).
Treat any formula as a starting point and measure. Also keep separate pools for different kinds of work. If slow database calls and quick in-memory work share one pool, the slow ones can use every worker and starve everything else.
07Pitfalls
If a task waits for another task that is queued in the same pool, and all workers are waiting like that, nothing can run. This is a pool-induced deadlock. Avoid tasks that block on results of tasks submitted to the same pool.
An exception inside a task submitted with submit is stored in the returned Future and surfaces only when you call get(). If you never do, the failure is invisible. Always check results or log inside the task.
And call shutdown() when finished. Non-daemon worker threads keep the JVM alive, so a forgotten pool can stop a program from exiting.
08Common interview questions
Q: Why use a thread pool?
A: It reuses threads, avoiding creation cost, and caps the number of threads so load cannot exhaust memory.
Q: What happens when all workers are busy?
A: New tasks wait in the queue. If the queue is bounded and full, the rejection policy applies.
Q: Name some rejection policies.
A: Abort (throw), CallerRuns (run on the submitter), Discard, DiscardOldest.
Q: How big should the pool be?
A: About the core count for CPU-bound work, larger for I/O-bound work, and then measure.
Q: What is a pool-induced deadlock?
A: Tasks wait on other tasks queued in the same full pool, so no worker is ever free.
Q: Why is an unbounded queue risky?
A: It hides overload and can grow until memory runs out.