The connection pool that ran dry — and the 38 orphaned connections that made the hub render empty
The hub that responded but was empty
It was 12:00 on 08/07. The Bug Hunter (the automated auditor that renders routes and checks whether content actually mounted) ran as usual. Result: /hub, /playground, /knowledge, /tools/scrape with #app len=7 — the HTML only had an empty <div id="app"></div>, nothing inside.
Arachne responded HTTP 200. The deploy was green. But authenticated routes rendered empty.
And in the journal: 1,239 errors of QueuePool limit ... timeout 30.00 accumulated since 02:57.
The context: connection pools are finite
Arachne uses SQLAlchemy with PostgreSQL. The default pool is QueuePool with 10 active connections + 20 overflow = max 30 concurrent.
When the pool fills, new requests wait (timeout 30.00) and then fail. Classic bottleneck: if every request returns its connection, 30 is enough. If some hold their connection forever, the pool dries up.
The suspect: the X-API-Key authentication middleware.
The struggle: the generator that was never closed
The auth_middleware did a query to validate the API key:
# Before — the generator leaks the connection
_session = next(_get_session()) # opens the session... and NEVER closes
# uses _session to validate the key
# ... request ends, generator stays openThe problem: _get_session() is a generator. next() opens the session, but the connection only returns to the pool when the generator is closed (via finally or with). Without closing, each X-API-Key-authenticated request leaked 1 idle in transaction connection in PG.
With continuous MCP traffic (clients calling arachne_* constantly), orphaned connections accumulated: 38 stuck connections — some for 1.6h, others 8.6h.
# What the journal showed (1,239 times)
sqlalchemy.exc.TimeoutError: QueuePool limit of size 10 overflow 20 reached,
connection timed out, timeout 30.00
# What PG saw: 38 idle-in-transaction connections
SELECT state, COUNT(*) FROM pg_stat_activity
WHERE datname = 'arachne' GROUP BY state;
# 38 idle in transaction ← the orphansThe resolution: close the generator in finally
# After — keep the generator and close it in finally
_session_gen = _get_session()
try:
_session = next(_session_gen)
# validates the key
finally:
_session_gen.close() # runs __exit__ → session.close() → connection returns to poolA 22-line change. The generator is now closed in finally — the connection returns to the pool immediately, no matter what happens in between.
Immediate mitigation for the incident: pg_terminate_backend on the 38 orphaned connections + service restart.
Verification: Bug Hunter re-run → 11/11 checks OK, 0 failures, /api/auth/me 200.
Metrics
| Metric | Value |
|---|---|
| QueuePool errors in journal | 1,239 (since 02:57) |
| Stuck orphan connections | 38 (idle in transaction) |
| Oldest age | 8.6h |
| Pool config | 10 + 20 overflow = 30 max |
| Fix | _session_gen.close() in finally (22 lines) |
| Post-fix verification | Bug Hunter 11/11, /api/auth/me 200 |
Lessons learned
-
SQLAlchemy generator without close = connection leak —
next(_get_session())withoutfinallyis a trap. The session opens, but the connection only returns to the pool when the generator closes. -
HTTP 200 doesn’t mean “it worked” — authenticated routes responded 200 with empty HTML. Only the Bug Hunter (which renders and checks the DOM) caught it. HTTP status health checks aren’t enough for SPAs.
-
Pooling bugs are gradual, not binary — 1,239 errors accumulated since 02:57. It didn’t break suddenly; it bled connection by connection until exhaustion. Monitoring
pg_stat_activityis the safety net. -
Automated auditing catches what monitoring doesn’t — the Bug Hunter ran at the right time and found the symptom (empty routes) that led to the cause (exhausted pool).