Once a week, always sometime between 2 and 4am, one of our Node services would get OOM-killed and restart. The graphs made it obvious there was a leak — memory climbed steadily for days and never came back down — but "there's a leak somewhere" isn't a fix.
Reproducing It on Purpose
The first problem with memory leaks is that they're slow, so I needed a way to make one week of growth happen in ten minutes. I wrote a small load script that hit the same endpoints our real traffic hits, in the same rough proportions, and let it run against a local instance with --max-old-space-size set low enough to force the issue faster.
node --inspect --max-old-space-size=256 dist/server.js &
node scripts/load-test.js --duration=600 --rps=50
With --inspect running, I connected Chrome DevTools to the process and took heap snapshots every two minutes. The retained size kept climbing across snapshots even when the load test was steady, not bursty, which ruled out "it's just GC being lazy."
Reading the Heap Snapshot
Comparing two snapshots taken ten minutes apart, DevTools' comparison view showed one constructor with a growing count that had no business growing: EventEmitter listeners. Specifically, a growing number of closures retained by a single EventEmitter instance.
That pointed at the culprit: every incoming WebSocket connection registered a listener on a shared, module-level event bus, but disconnect handling never called .removeListener(). Connections came and went constantly, but the listener count only ever went up.
// before — leaks one listener per connection, forever
function onConnection(socket: WebSocket) {
bus.on('broadcast', (msg) => socket.send(msg));
}
// after — cleans up when the socket closes
function onConnection(socket: WebSocket) {
const handler = (msg: string) => socket.send(msg);
bus.on('broadcast', handler);
socket.on('close', () => bus.off('broadcast', handler));
}
It's an embarrassingly simple bug once you see it, and also exactly the kind of thing that's invisible in code review because both halves look correct in isolation.
Why It Took a Week to Notice
Each individual listener is tiny. The closure it captures — a reference to the socket and a bit of connection state — is a few hundred bytes. At our traffic level it took thousands of unclosed connections accumulating before the leak was big enough to matter, which is exactly why it survived staging and even a few days of production traffic before crossing the OOM threshold.
Guarding Against It Going Forward
Fixing the bug wasn't enough on its own; I wanted a way to catch the next one before it reaches production. I added a lightweight check to our health endpoint that reports bus.listenerCount('broadcast'), and a CI smoke test that opens and closes 500 sockets against a test server, asserting the listener count returns to zero afterward.
# excerpt from the smoke-test job
- name: websocket leak check
run: node scripts/websocket-leak-check.js --connections=500 --assert-zero
I also set EventEmitter.defaultMaxListeners down from the default of 10 to something closer to what we actually expect on any single bus, so a runaway listener count now throws a warning in logs immediately instead of silently accumulating for weeks. That single line has already caught one regression in review before it shipped.
The database side of the same service benefited from a similar "look at what's actually happening instead of guessing" approach — see the indexing work here — and the image this service ships in is the multi-stage Docker build I wrote up separately, which also made local reproduction of the memory limits much closer to production.