Debugging a gRPC Deadlock at 2am

A postmortem on a connection pool starvation bug that took three days to find and about ten minutes to fix once I actually found it.
Debugging a gRPC Deadlock at 2am
FIG. 01 — DEBUGGING A GRPC DEADLOCK AT 2AMEST. 2026.01.22 | A.LABS

Under moderate concurrent load, requests between two internal services would just... hang. No error, no timeout firing, no logs indicating anything was wrong — connections just stopped completing. It only happened after the service had been running for a while, which made it miserable to reproduce.

"The bug was never in the logic. It was in the assumption that the pool would always have room."

Chasing the Wrong Lead

I spent the first day convinced it was a context cancellation issue, then a day convinced it was a proto serialization bug on large payloads. Neither was it. The actual clue was buried in a goroutine dump I almost didn't bother pulling — dozens of goroutines all blocked waiting on the same connection pool.

The Actual Fix

One service was making synchronous gRPC calls back to the caller from inside a handler — a circular call pattern that, under load, exhausted the client's connection pool waiting on responses that couldn't complete until the pool freed up. Classic deadlock, just disguised by gRPC's abstraction. Fixed by breaking the circular dependency and bumping pool size as a safety margin, not a solution.

Finis