The lock born in the wrong loop — and the browser pool that said 'not available in this worker'

“Browser agent not available in this worker”

Between 08 and 09/08, Arachne woke up intermittently. A scraping workflow would run, and out of nowhere: browser agent not available in this worker. Run it again — it worked. Again — it failed.

In the journal, the cryptic RuntimeError:

RuntimeError: <asyncio.locks.Lock object at 0x7f...> is bound to a different event loop

A lock bound to an event loop that wasn’t the worker’s. Like a padlock mounted on one door, then the lock moved houses — right key, wrong place.

The context: the browser pool shares a lock

Arachne keeps a browser pool (Camoufox + Playwright) — browsers reused between scraping and VRT so a new browser doesn’t spin up on every call. A pool like this needs a lock: two workers can’t use the same browser at the same time.

# The pool, simplified
_browsers = []  # list of alive browsers
_last_gc = 0.0  # last garbage collect

async def _get_browser(headless=True):
  async with _lock:  # protects pool access
  # ... finds or creates a browser

The lock was global, created at module import:

_lock = asyncio.Lock()  #  created at import time

And there lived the bug.

The struggle: the event loop that changes houses

In Python asyncio, an asyncio.Lock() created outside a loop gets bound to whichever loop is running at that moment — or to the current loop the first time it’s used.

Arachne has two execution paths:

• Normal requests (FastAPI) → run on the process’s main event loop
• Scheduled workflows (cron scheduler) → run via threadpool → each thread gets its own event loop

When the module was imported in the main process, the lock was born bound to the main loop. When the workflow ran in the threadpool and tried async with _lock, Python compared the lock’s loop with the current loop → RuntimeError: bound to a different event loop.

The symptom was double and confusing:

• The technical error (bound to a different event loop) showed up in internal logs
• The business message (“browser agent not available in this worker”) showed up for whoever consumed the workflow

Intermittent because it depended on when the import happened and which thread the workflow landed on.

The resolution: lazy init — born in the right loop

The fix (commit 457326e, 09/08 02:59): the lock is no longer created at import — it’s created inside the current loop, on first use.

#  Before — born bound to the import's loop
_lock = asyncio.Lock()

#  After — born in the loop that's running now
_lock = None  # type: Optional[asyncio.Lock]

async def _get_lock() -> asyncio.Lock:
  global _lock
  if _lock is None:
  _lock = asyncio.Lock()  # created INSIDE the current loop
  return _lock

And at both usage points:

# async with _lock:  →  async with await _get_lock():

The await _get_lock() runs inside the coroutine — the asyncio.Lock() is created in the right loop, the one executing that worker. 16 lines added, 3 removed. That’s it.

Yesterday’s story was a generator that never closed the Postgres session. Today it’s a lock born in the wrong loop. Same pattern: the resource must be born in the context where it will live.

Metrics

Takeaways

1. Global async resource = loop problem — asyncio.Lock(), asyncio.Queue() and friends created at import get bound to the process’s loop. If the code runs in a threadpool/other loop (cron, workers), it breaks with “bound to a different event loop”.

2. Lazy init fixes the lifecycle — creating the resource inside the coroutine (on first use) guarantees it’s born in the loop that will use it. That’s the pattern for any async singleton.

3. Intermittent error = context clue — failures that come and go are almost always timing-dependent: which loop, which thread, which import. The full RuntimeError in the journal is what gave away the cause.

4. Two symptoms, one cause — the business message (“not available in this worker”) hid the technical error. Always chase the full stack before “fixing” the symptom.

╔══════════════════════════════════════╗
║  Samuel Medeiros                    ║
║  Senior Software Engineer           ║
║  Stack: Python · TypeScript · Rust  ║
║  Projetos: Arachne, Dogwalk,        ║
║            Capivara, TatuEngine      ║
╚══════════════════════════════════════╝