Multi-engine pipeline in Arachne — 4 fallback layers

June 8th. Two days after Arachne’s first commit, I already had a clear problem: the web is hostile to scrapers.

Pages that worked in the morning broke in the afternoon. Sites that responded well with Trafilatura suddenly required JavaScript. Simple blogs turned into SPAs out of nowhere. And Cloudflare — oh, Cloudflare — was a monster that swallowed requests without mercy.

The first version of the pipeline was a half-baked try/except: try Crawl4AI, if it fails fall back to Trafilatura. It worked for about 60% of cases. The rest was blue screen.

I needed something that wouldn’t break. A system that would try the fastest path first, but knew how to escalate to the heaviest cannon if needed. That’s how the multi-engine pipeline with 4 fallback layers was born.

The architecture

The flow is a progressive staircase. Each step is slower than the previous, but also more capable:

┌──────────────────────────────────────────────────────┐
│  INPUT URL  │
└────────────────────┬─────────────────────────────────┘
  │
  ▼
  ┌─────────────────────────┐
  │  DomainHistory check  │ ← Have we visited this domain before?
  └────────┬────────────────┘
  │
  ┌────────────┴────────────┐
  │  Fastest engine  │ ← If history exists, start with the best one
  │  that has worked  │
  └────────────┬────────────┘
  │
  ╔════════════╪══════════════════════╗
  ║  ▼  ║
  ║  ┌──────────────────┐  ║
  ║  │ TRAFILATURA  │  250ms  ║  ─── Layer 1
  ║  │ Pure HTTP  │  ~70% sites ║
  ║  └───────┬──────────┘  ║
  ║  │ Failed? Low content?  ║
  ║  │ SPA detected?  ║
  ║  ▼  ║
  ║  ┌──────────────────┐  ║
  ║  │ CRAWL4AI SDK  │  1-2s  ║  ─── Layer 2
  ║  │ Chromium headless│  +JS sites ║
  ║  └───────┬──────────┘  ║
  ║  │ Failed?  ║
  ║  ▼  ║
  ║  ┌──────────────────┐  ║
  ║  │ DOCKER SIDECAR  │  3-8s  ║  ─── Layer 3
  ║  │ Isolated container│  Heavy  ║
  ║  └───────┬──────────┘  ║
  ║  │ Blocked?  ║
  ║  ▼  ║
  ║  ┌──────────────────┐  ║
  ║  │ CAMOUFOX  │  5-15s  ║  ─── Layer 4
  ║  │ Firefox stealth  │  Cloudflare║
  ║  └───────┬──────────┘  ║
  ║  │  ║
  ║  ▼  ║
  ║  ┌──────────────────┐  ║
  ║  │ CIRCUIT BREAKER  │  ║
  ║  │ Log + retry after│  ║
  ║  │ backoff  │  ║
  ║  └──────────────────┘  ║
  ╚══════════════════════════════════╝

Each layer attempts extraction, computes a confidence score, and decides whether the result is good enough or whether to escalate to the next engine.

Layer 1: Trafilatura (250ms, ~70% of lightweight sites)

Trafilatura is the sprinter of the group. It doesn’t open a browser — it downloads the HTML directly and extracts clean text. For blogs, documentation, and articles, it’s unbeatable.

async def _try_trafilatura(url: str, timeout: int = 5) -> Optional[dict]:
  cfg = trafilatura_settings.use_config()
  cfg["DEFAULT"]["EXTRACTION_TIMEOUT"] = str(timeout)

  downloaded = trafilatura.fetch_url(url)
  if not downloaded or not downloaded.strip():
  return {"status": "failed", "error": "Empty fetch", "engine": "trafilatura"}

  result = trafilatura.bare_extraction(
  downloaded,
  url=url,
  include_comments=False,
  include_tables=True,
  include_formatting=True,
  favor_precision=True,
  )

  if not result:
  # Fallback: try more aggressive extract()
  content = trafilatura.extract(downloaded)
  if not content or len(content.strip()) < MIN_CONTENT_CHARS:
  return {"status": "failed", "error": "Empty extraction",
  "html": downloaded, "engine": "trafilatura"}

  return {
  "status": "ok",
  "markdown": content,
  "html": downloaded,
  "chars": len(content),
  "engine": "trafilatura",
  }

  # ... extract title, description, author from result

The trick: I set favor_precision=True to avoid false positives. I’d rather fail fast (and escalate to the next engine) than return truncated content.

When it escalates to Layer 2:

• Fetch returned empty or None
• Extracted content < 100 characters (MIN_CONTENT_CHARS)
• Page looks like an SPA shell (< 3KB of HTML or React/Next.js markers)

Layer 2: Crawl4AI SDK (1-2s, JavaScript sites)

If Trafilatura fails, the Crawl4AI SDK opens a headless Chromium and renders the JavaScript for real.

async def _try_crawl4ai(url: str) -> Optional[dict]:
  from crawl4ai import AsyncWebCrawler
  from crawl4ai.async_configs import CrawlerRunConfig

  config = CrawlerRunConfig(
  word_count_threshold=5,
  excluded_tags=["nav", "footer", "script", "style"],
  wait_until="networkidle",
  page_timeout=5000,
  )

  async with AsyncWebCrawler() as crawler:
  crawl_result = await crawler.arun(url=url, config=config)

  if not crawl_result.success:
  return {"status": "failed", "error": crawl_result.error_message,
  "engine": "crawl4ai"}

  # Crawl4AI 0.8.9: markdown may come as fit_markdown or raw_markdown
  markdown = ""
  if crawl_result.markdown:
  if isinstance(crawl_result.markdown, str):
  markdown = crawl_result.markdown
  elif hasattr(crawl_result.markdown, 'fit_markdown') \
  and crawl_result.markdown.fit_markdown:
  markdown = crawl_result.markdown.fit_markdown
  elif hasattr(crawl_result.markdown, 'raw_markdown') \
  and crawl_result.markdown.raw_markdown:
  markdown = crawl_result.markdown.raw_markdown

The difference from Trafilatura is brutal on SPAs. A Next.js page that Trafilatura returns as empty <script>...</script>, Crawl4AI renders into clean markdown with titles, paragraphs, and tables.

When it escalates to Layer 3:

• SDK not installed (ImportError)
• Extracted content < 100 chars
• Timeout or network error

Layer 3: Docker Sidecar (3-8s, heavy pages)

The Sidecar is Crawl4AI running in a separate Docker container. Same engine, but isolated — it won’t affect Arachne’s memory if it crashes.

async def _try_sidecar(url: str) -> Optional[dict]:
  from app.engines.crawl4ai_sidecar import check_health, scrape_with_sidecar

  if not check_health():
  return {
  "status": "failed",
  "error": "Sidecar offline (check docker compose)",
  "engine": "sidecar",
  }

  result = await scrape_with_sidecar(url)
  if not result.get("success"):
  return {"status": "failed", "error": result.get("error"),
  "engine": "sidecar"}

  markdown = result.get("markdown", "") or ""
  cleaned_html = result.get("cleaned_html") or result.get("html", "") or ""

  # Extract structured data (JSON-LD, Open Graph)
  if cleaned_html:
  from app.scraper.extractors import extract_structured
  structured_data = extract_structured(cleaned_html, url=url)

The Sidecar runs on localhost:11235. The health check is a GET /health — if it doesn’t respond in 2s, we don’t even bother and escalate straight to Layer 4.

I discovered through practice that the Sidecar is especially good for e-commerce pages (Amazon, Shopee) that the Crawl4AI SDK processes but takes a while. The dedicated container doesn’t compete for resources with the main server.

When it escalates to Layer 4:

• Docker container not responding / not installed
• Sidecar returned an error
• Extracted content < 100 chars

Layer 4: Camoufox (5-15s, Cloudflare and blocked sites)

Camoufox is the last resort. It’s a Firefox with anti-detection patches — it gets past Cloudflare, Turnstile, and most WAFs.

async def _try_camoufox(url: str) -> Optional[dict]:
  from app.scraper.engine import fetch_camoufox
  from app.scraper.extractors import extract_all, extract_structured

  fetch_result = fetch_camoufox(url)

  if fetch_result.was_blocked or fetch_result.status == 0:
  return {"status": "failed",
  "error": f"Camoufox blocked: {fetch_result.reason}",
  "engine": "camoufox"}

  html = fetch_result.html
  pre_extracted = fetch_result.pre_extracted or {}

  extracted = extract_all(html, url=url,
  do_structured=True, do_text=True, do_structure=True)

  if extracted:
  content = extracted.text_content or ""
  title = extracted.title or ""
  else:
  # Fallback: Trafilatura on rendered HTML
  tr = trafilatura.bare_extraction(html, url=url)
  content = tr.text if tr and hasattr(tr, 'text') else ""

It’s slow — 5 seconds minimum, 15 on heavy pages. But it gets through where nothing else does. Sites with Cloudflare Advanced, pages requiring interaction, heavy single-page apps — Camoufox handles them.

The caching system

The cache is dual: SyncCache (Redis-like in-memory) + SQLite with configurable TTL.

def _get_cached(url: str, session: Session, max_age: int = 3600) -> Optional[dict]:
  # Try fast cache first (SyncCache in RAM)
  cached = sync_cache.get(f"pipeline:{url}")
  if cached is not None:
  return cached

  # Fallback: SQLite CacheEntry
  entry = session.exec(
  select(CacheEntry).where(CacheEntry.url == url)
  .order_by(desc(CacheEntry.created_at)).limit(1)
  ).first()
  if not entry:
  return None

  age = (datetime.now(timezone.utc) - entry.created_at.replace(tzinfo=timezone.utc)).total_seconds()
  if age > entry.ttl_seconds:
  session.delete(entry)
  session.commit()
  return None

  return json.loads(entry.data_snapshot)

The default TTL is 1 hour. Cached URL = response in milliseconds. New URL = full pipeline. In the future I want adaptive caching — pages that rarely change (documentation) get longer TTL, dynamic pages (news) get shorter TTL.

Confidence scoring — how we decide if the result is good

The heart of the pipeline is the score_result function. It evaluates result quality across 4 dimensions:

def score_result(result: dict) -> float:
  if not result or result.get("status") in ("failed", "blocked"):
  return 0.0

  content = result.get("markdown") or result.get("text") or result.get("content") or ""
  html = result.get("html", "")
  chars = len(content)
  html_len = len(html)

  # 1. Content size (weight 0.4)
  if chars < 100:  length_score = 0.0
  elif chars < 500:  length_score = 0.2
  elif chars < 2000:  length_score = 0.5
  elif chars < 8000:  length_score = 0.8
  else:  length_score = 1.0

  # 2. Text/HTML ratio (weight 0.3) — less junk, more content
  if html_len > 0 and chars > 0:
  ratio = chars / html_len
  if ratio > 0.5:  ratio_score = 1.0
  elif ratio > 0.2:  ratio_score = 0.7
  elif ratio > 0.05:  ratio_score = 0.4
  else:  ratio_score = ratio * 5
  else:
  ratio_score = 0.0

  # 3. Structured data bonus (weight 0.2) — JSON-LD, Open Graph
  # 4. Metadata bonus (weight 0.1) — title, description, author

  # Weighted combination
  final = (
  length_score * 0.40 +
  ratio_score * 0.30 +
  structured_score * 0.20 +
  metadata_score * 0.10
  )
  return round(min(final, 1.0), 4)

The constants are:

If Trafilatura returns a score > 0.75, we don’t even try the other engines — saving 1-15s per request.

DomainHistory — per-domain memory

One of the most important lessons learned: the web is not homogeneous. Each domain has its own behavior. What works on GitHub doesn’t work on Amazon.

class DomainHistory:
  def __init__(self):
  self._data: dict[str, dict[str, int]] = {}
  # domain -> {engine: score}

  def record_success(self, url: str, engine: str):
  domain = urlparse(url).netloc.lower()
  self._data.setdefault(domain, {})
  self._data[domain][engine] = \
  self._data[domain].get(engine, 0) + 1

  def best_engine(self, url: str) -> Optional[str]:
  domain = urlparse(url).netloc.lower()
  scores = self._data.get(domain, {})
  if not scores:
  return None
  positive = {e: s for e, s in scores.items() if s > 0}
  if not positive:
  return None
  return max(positive, key=positive.get)

DomainHistory becomes a before_fetch hook that reorders the engines: if example.com always worked with Crawl4AI, we start with it next time, skipping Trafilatura.

It’s a kind of simplified reinforcement learning — no network weights, just counting. But it works surprisingly well. After 5-10 requests to the same domain, the pipeline already knows which engine to use without error.

Hooks — the system that makes everything possible

The pipeline is extensible via hooks at 9 different points:

HOOK_POINTS = {
  "before_fetch":  "Before HTTP request",
  "after_fetch":  "After raw HTML",
  "before_extract":  "Before extracting content",
  "after_extract":  "After extraction",
  "before_cache":  "Before saving to cache",
  "before_crawl_url":  "Before each URL in a BFS crawl",
  "on_error":  "When an error occurs",
  "on_retry":  "When about to retry",
  "on_complete":  "Pipeline completed successfully",
}

The multi-engine’s after_extract hook, for example, detects when Trafilatura returned thin content and sets try_next_engine = True:

async def _after_extract_hook(ctx: dict) -> dict:
  result = ctx.get("result")
  if not result:
  return ctx

  confidence = score_result(result)
  result["confidence"] = confidence

  engine = result.get("engine", "")
  if confidence < MEDIUM_CONFIDENCE and engine == "trafilatura":
  need_js = _needs_js_rendering(result.get("html", ""))
  if need_js or result.get("chars", 0) < MIN_CONTENT_CHARS:
  ctx["try_next_engine"] = True

This allows the pipeline to be composed — you add a hook that filters ads, another that extracts assets, another that detects SPAs. Each handles its own concern without cluttering the main flow.

Progressive fetch — 4 levels of HTTP escalation

Parallel to the multi-engine pipeline (which focuses on extraction), there’s progressive_fetch, which is the escalation of HTTP request techniques:

_FETCH_REGISTRY = [
  (FetchLevel.BASIC,  "fetch_basic"),  # scrapling Fetcher.get()
  (FetchLevel.STEALTH,  "fetch_stealth"),  # scrapling StealthyFetcher
  (FetchLevel.DYNAMIC,  "fetch_dynamic"),  # scrapling DynamicFetcher
  (FetchLevel.CAMOUFOX,  "fetch_camoufox"),  # FetchManager pool
]

Each level increases the “disguise”:

• BASIC: Pure HTTP. Fast, but blocked by any WAF.
• STEALTH: Realistic headers, real Chrome User-Agent, fake screen resolution.
• DYNAMIC: Headless browser with fake WebGL, fonts, and canvas fingerprint.
• CAMOUFOX: Firefox with anti-detection patches. Gets past Cloudflare Turnstile.

And the coolest part: it automatically detects SPA shells. If the returned HTML is less than 3KB or looks like an empty React page (id="root", __next_data__), it escalates to the next level even with status 200.

def _needs_js_rendering(html: str) -> bool:
  if len(html) < 3000:
  return True  # Ultra-small = SPA shell

  markers = ["__next_data__", "__nuxt__", 'id="__next"',
  'id="app"', 'id="root"', "react-root"]
  if len(html) < 8000:
  for marker in markers:
  if marker.lower() in html.lower():
  return True
  return False

What I learned implementing this

1. The web lies about its content

Pages with status 200 can have irrelevant content. SPAs return empty HTML. Legitimate sites look like bots. I learned to trust content analysis more than status codes.

2. Cache solves 80% of performance problems

Without caching, the pipeline takes 250ms to 15s per URL. With caching (and intelligent TTL), repeated URLs come back in < 5ms. The difference is so large that I even put cache on the fallback — if Trafilatura fails but there’s stale cache, we use it anyway.

3. Engine ordering matters more than individual quality

A pipeline that starts with Camoufox (most capable) takes 5x longer per request than one that scales gradually. DomainHistory reduced average extraction time by ~40% after 100+ requests, simply because it learned the ideal order for each domain.

4. Confidence scoring is fragile if you don’t calibrate

At first I only used text size. Result: pages with 10KB of HTML junk but 5KB of text got high scores. Adding the text/HTML ratio and structured data improved accuracy from 72% to 94% in testing.

5. Docker Sidecar is brilliant until the container dies

The Sidecar isolates the heavy browser, but if the container goes down (OOM, deadlock), the entire Arachne loses Layer 3. Solution: health check on every request + automatic fallback to the native SDK. The container can be dead and the pipeline still works.

The real metrics

The combined success rate of all 4 layers is ~97% — out of 1000 tested URLs, 970 return usable content. Without the pipeline, with a single engine, it’d be about 60-70%.

Where the remaining 3% fail: Cloudflare Challenge requiring human interaction, explicit CAPTCHA, or sites that completely block the IP (network rate limiting, not application-level).

What I would change today

• Predictive cache: start fetching before the user clicks, based on navigation patterns. I already have the hooks, just need the prediction logic.
• Engine by site type: if the classifier detects it’s a blog, jump straight to Trafilatura. If it’s e-commerce, start with Crawl4AI. This would cut ~1s per request on average.
• Parallelism between engines: today each engine waits for the previous to fail. We could run Trafilatura and Crawl4AI in parallel and take the first that responds with quality. But then we’d lose the connection cache benefit (Trafilatura reuses keep-alive if called first).
• Exponential backoff per domain: sites that block shouldn’t be hammered. A 30s-5min backoff per domain would reduce blocks by ~40%.

Test commands

The multi-engine pipeline has a CLI helper that I use directly for debugging:

╔══════════════════════════════════════╗
║  Samuel Medeiros                    ║
║  Senior Software Engineer           ║
║  Stack: Python · TypeScript · Rust  ║
║  Projetos: Arachne, Dogwalk,        ║
║            Capivara, TatuEngine      ║
╚══════════════════════════════════════╝
      

In the end, the multi-engine pipeline is what makes Arachne more than just a scraper. It’s a machine that understands the web is unpredictable, and instead of trying to force a single method, it learns to adapt.

June 8, 2026 was the day Arachne stopped breaking. The day I stopped putting out fires and started building a system that puts them out on its own.