Arachne Was Born from Frustration — Broken Scrapers and Lost Data

If Dogwalk was born from a need, Arachne was born from a deep frustration with broken tools.

I was tired of maintaining scrapers that would stop working out of nowhere. A site would change a CSS class, and boom — a whole day’s data lost. Services like ScrapingBee and Crawlbase solved part of the problem, but cost a lot and didn’t give me fine-grained control over the process.

The tipping point

June 3, 2026. I was debugging a scraper that broke for the third time in the same week when I thought: “I’ll build my own extraction system.”

At the time it seemed crazy. Building a scraping platform from scratch? But I already had experience with Crawl4AI, knew the limitations of each approach (Trafilatura, Playwright, raw requests), and knew exactly what a good system needed to have.

The first file

Curiously, the first file of the Arachne project wasn’t a scraper. It was an empty __init__.py in tests/.

# tests/__init__.py — the first file of Arachne
# 06/03/2026, 22:47
# Didn't have anything to test yet, but knew I would need it

Seems silly, but starting the project with a test directory set the tone: this project would be built with discipline. It wasn’t a weekend experiment — it was a platform I intended to actually use.

The initial architecture

The plan was simple in theory, complex in practice:

1. Extraction layer: Crawl4AI as main engine, Trafilatura as lightweight fallback
2. Smart cache: Avoid re-fetching already processed pages
3. Structured output: Clean Markdown, not garbage-filled HTML

# The first version of the extractor (simplified)
async def extract(url: str) -> str | None:
  try:
  result = await crawl4ai.extract(url)
  return result.markdown
  except Exception:
  # Fallback to Trafilatura if Crawl4AI fails
  return await trafilatura.extract(url)

This fallback logic, which today is automatic and has 4 layers, started exactly like this: a simple try/except.

The engine matrix — when to use each

One of the first things I did was map every available extraction engine and understand where each one shines. The result became a table that still guides routing decisions today:

The big insight was not competing between engines — each has a niche. The real problem was routing intelligently.

# The engine router — production version
class EngineRouter:
  def __init__(self):
  self.engines = {
  "trafilatura": TrafilaturaEngine(),
  "crawl4ai": Crawl4AIEngine(),
  "playwright": PlaywrightEngine(),
  "browser_evasion": BrowserEvasionEngine(),
  }
  self.cache = CacheEngine()

  async def extract(self, url: str, force: bool = False) -> ExtractionResult:
  # 1. Cache check first — always
  if not force:
  cached = await self.cache.get(url)
  if cached:
  return cached

  # 2. Try Crawl4AI (best effort, best markdown)
  try:
  return await self.engines["crawl4ai"].extract(url)
  except (ConnectionError, TimeoutError):
  pass

  # 3. Fallback to Trafilatura (lightweight, no JS)
  try:
  return await self.engines["trafilatura"].extract(url)
  except Exception:
  pass

  # 4. Playwright if JS is needed
  try:
  return await self.engines["playwright"].extract(url)
  except (TimeoutError, CloudflareBlock):
  pass

  # 5. Browser with evasion as last resort
  return await self.engines["browser_evasion"].extract(url)

Each fallback call costs time, but it’s better than returning empty. The natural progression (light → heavy) ensures 80% of pages are resolved on the first or second attempt.

The cache system that saves hours

Caching was one of the best decisions in the initial project. Every repeated request consumes resources and time — especially those going through Playwright (4-8s each).

# CacheEngine — SQLite with FTS5 for content search
import sqlite3
import json
import hashlib
from datetime import datetime, timedelta

class CacheEngine:
  def __init__(self, db_path: str = "cache/arachne_cache.db"):
  self.conn = sqlite3.connect(db_path)
  self.conn.execute("""
  CREATE TABLE IF NOT EXISTS cache (
  url_hash TEXT PRIMARY KEY,
  url TEXT NOT NULL,
  engine TEXT NOT NULL,
  result TEXT NOT NULL,
  content_type TEXT,
  status_code INTEGER,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  accessed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  ttl_hours INTEGER DEFAULT 24
  )
  """)
  self.conn.execute("""
  CREATE INDEX IF NOT EXISTS idx_cache_url
  ON cache(url)
  """)
  self.conn.commit()

  def _hash(self, url: str) -> str:
  return hashlib.sha256(url.encode()).hexdigest()[:16]

  async def get(self, url: str) -> ExtractionResult | None:
  url_hash = self._hash(url)
  row = self.conn.execute("""
  SELECT result, engine, created_at, ttl_hours
  FROM cache WHERE url_hash = ?
  """, (url_hash,)).fetchone()

  if not row:
  return None

  result_json, engine, created_at, ttl = row
  created = datetime.fromisoformat(created_at)
  if datetime.now() - created > timedelta(hours=ttl):
  self.conn.execute("DELETE FROM cache WHERE url_hash = ?", (url_hash,))
  self.conn.commit()
  return None

  # Update accessed_at
  self.conn.execute("""
  UPDATE cache SET accessed_at = CURRENT_TIMESTAMP
  WHERE url_hash = ?
  """, (url_hash,))
  self.conn.commit()

  return ExtractionResult(
  url=url,
  markdown=json.loads(result_json),
  engine=engine,
  cached=True
  )

  async def set(
  self,
  url: str,
  result: str,
  engine: str,
  ttl_hours: int = 24
  ) -> None:
  url_hash = self._hash(url)
  self.conn.execute("""
  INSERT OR REPLACE INTO cache
  (url_hash, url, engine, result, ttl_hours)
  VALUES (?, ?, ?, ?, ?)
  """, (url_hash, url, engine, json.dumps(result), ttl_hours))
  self.conn.commit()

  async def stats(self) -> dict:
  stats = self.conn.execute("""
  SELECT
  COUNT(*) as total,
  SUM(CASE WHEN engine = 'crawl4ai' THEN 1 ELSE 0 END) as crawl4ai,
  SUM(CASE WHEN engine = 'trafilatura' THEN 1 ELSE 0 END) as trafilatura,
  SUM(CASE WHEN engine = 'playwright' THEN 1 ELSE 0 END) as playwright,
  SUM(CASE WHEN engine = 'browser_evasion' THEN 1 ELSE 0 END) as evasion,
  AVG(ttl_hours) as avg_ttl
  FROM cache
  """).fetchone()

  return {
  "total_entries": stats[0],
  "by_engine": {
  "crawl4ai": stats[1] or 0,
  "trafilatura": stats[2] or 0,
  "playwright": stats[3] or 0,
  "browser_evasion": stats[4] or 0,
  },
  "avg_ttl_hours": round(stats[5] or 0, 1),
  }

The cache uses SHA256 hash truncated to 16 characters as the key — collision is virtually impossible for the volume we process. The 24-hour TTL prevents dynamic pages from going stale, but still saves repeated requests on the same day.

The fallback chain that saved N projects

The 4-layer fallback architecture wasn’t planned — it emerged from failures. Each layer covers a specific failure scenario:

# Async pipeline with exponential retry and automatic fallback
async def extraction_pipeline(
  url: str,
  max_attempts: int = 3
) -> ExtractionResult | None:
  """
  Full extraction pipeline with 4 fallback layers.
  Each layer has up to max_attempts with exponential backoff.
  """
  engines_chain = [
  ("crawl4ai", Crawl4AIEngine()),
  ("trafilatura", TrafilaturaEngine()),
  ("playwright", PlaywrightEngine()),
  ("browser_evasion", BrowserEvasionEngine()),
  ]

  for engine_name, engine_instance in engines_chain:
  for attempt in range(max_attempts):
  try:
  result = await engine_instance.extract(url)
  if result and result.markdown:
  logger.info(
  "%s extracted %s in %.2fs (attempt %d)",
  engine_name, url, result.time, attempt + 1
  )
  return result
  except CloudflareBlock:
  logger.warning("%s blocked by Cloudflare: %s", engine_name, url)
  break  # No point retrying — skip to next engine
  except TimeoutError:
  backoff = 2 ** attempt
  logger.warning(
  "%s timeout on %s — backoff %ds",
  engine_name, url, backoff
  )
  await asyncio.sleep(backoff)
  except Exception as exc:
  logger.error(
  "%s broke on %s: %s", engine_name, url, exc
  )
  break  # Non-recoverable error — next engine

  logger.error("ALL engines failed for %s", url)
  return None

What looks like simple code solved 90% of extraction problems. The key is the break on CloudflareBlock — there’s no point insisting on an engine the anti-bot has already detected. Better to jump straight to Playwright with evasion.

Architectural decisions that shaped the project

1. SQLite as cache, not Redis

I could have gone with Redis. It would be faster, more modern. But SQLite solves the problem with zero dependencies, zero extra processes, and more than enough performance for tens of thousands of cached URLs.

# Informal benchmark: 10K cache hits
# SQLite: 47ms total (4.7µs per hit)
# Redis (via docker): 112ms total (11.2µs per hit)
# Conclusion: SQLite wins on simplicity and performance

2. Async extraction from day one

I made sure to use asyncio from the start. Scraping is I/O-bound by nature — waiting for HTTP responses, waiting for rendering, waiting for cache writes. Blocking the main thread would be wasteful.

# Arachne processes N URLs in parallel with asyncio.gather
async def extract_batch(urls: list[str]) -> list[ExtractionResult]:
  tasks = [extraction_pipeline(url) for url in urls]
  results = await asyncio.gather(*tasks, return_exceptions=True)
  return [
  r for r in results
  if isinstance(r, ExtractionResult)
  ]

3. Engine as plugin, not coupling

Each engine implements a common interface. This allows adding (or removing) engines without touching the central pipeline:

class BaseEngine(ABC):
  @abstractmethod
  async def extract(self, url: str) -> ExtractionResult:
  pass

  @property
  @abstractmethod
  def name(self) -> str:
  pass

  @property
  @abstractmethod
  def supported_content_types(self) -> list[str]:
  pass

Today Arachne has 9 registered engines, from Trafilatura to PDF Vision. All because the interface was designed to be extensible from the start.

4. Structured logging as a debugging tool

Don’t underestimate the value of well-made logs. Each extraction step logs: engine used, elapsed time, result size, HTTP code.

logger = structlog.get_logger()
# Real log example:
# 2026-06-03 22:47:01 [crawl4ai] extracted https://example.com in 2.34s (2367 chars)
# 2026-06-03 22:47:02 [cache] cache HIT for https://example.com (remaining ttl: 18h)

Real numbers from the first month

Thirty days after the first commit, Arachne was already processing:

44.6% cache hit in the first month — almost half the requests never hit the network. Each cache hit saved ~3 seconds of processing. That’s 1,892 × 3s = ~94 minutes saved in one month, just from caching.

Why “Arachne”?

Spider that weaves webs. It made sense — scrapers are like digital spiders, crawling the web for information. The name stuck and became the project’s identity.

June’s frustration turned into what is today a platform with a multi-engine pipeline, SQLite cache, FTS5 indexing, and RAG integration. All because a scraper broke at the wrong time.

Useful commands

╔══════════════════════════════════════╗
║  Samuel Medeiros                    ║
║  Senior Software Engineer           ║
║  Stack: Python · TypeScript · Rust  ║
║  Projetos: Arachne, Dogwalk,        ║
║            Capivara, TatuEngine      ║
╚══════════════════════════════════════╝