The Arachne story — from iMasters scraper to Knowledge OS with 42 tools
The scraper that became a knowledge operating system
It all started with a simple need: extracting content from sites that block scraping. Arachne was born as a scraper specialized for iMasters, with four evasion levels (Basic → Stealth → Headless → Camoufox).
Today it’s a platform with hybrid RAG, a chatbot with a 42-tool ReAct Agent, an MCP server, a visual pipeline with 38 handlers, computer vision, audio, and monetization — 2,818 tests making sure none of it breaks.
Context — from crawler to platform
The initial commit was on June 9, 2026:
2542caf chore: checkpoint antes de adicionar RAG + chatbotThat commit is revealing: Arachne was born already knowing it would go beyond scraping. On the same day came the RAG module with semantic search and vector store, plus the chatbot engine with API and widget.
The product shift came with the Knowledge OS: knowledge bases with a 3-step wizard, ingestion by URL or text, and hybrid search (FTS5 + vector + RRF) with cross-encoder reranking.
The struggle — 4 evasion layers and a search that couldn’t find
The initial battle was against Cloudflare. Each site had a different protection level — and the pipeline needed to scale:
# Arachne's progressive pipeline
progressive_fetch = Trafilatura → Stealth → Dynamic → Camoufox
# If all fail: Browser Agent with 4 evasion layersThe Browser Agent became a war machine: stealth JS, fingerprint rotation, cookie consent auto-dismissal (15+ frameworks), ad blocking with EasyList (104k rules), and multilingual block detection.
Then came hybrid search. FTS5 alone found too little; embeddings alone were imprecise. The answer was combining both with RRF (Reciprocal Rank Fusion) + reranking:
# Hybrid search: FTS5 + vector + RRF + cross-encoder
results = rrf_fusion(fts_results, vector_results)
results = cross_encoder_rerank(query, results, top_k=5)Resolution — the ReAct Agent and the MCP server
The highlight was when Arachne stopped being an API and became an agent. The ReAct Agent grew from 36 to 42 tools with router planner + fallback + cache. And the MCP server (stdio + SSE) exposed everything to external agents — Cursor, Claude, Hermes.
The cherry on top: the Python SDK published to PyPI (pip install arachne-sdk), and the vision-to-text v2 pipeline — 5 extractors (PySceneDetect, OpenCV K-Means, EasyOCR, Florence-2, Whisper) turning video into structured text for LLMs.
Metrics
| Metric | 09/06 (birth) | 31/07 (today) |
|---|---|---|
| Commits | 1 | 777 |
| Tests | 0 | 2818+ |
| ReAct tools | 0 | 42 |
| Pipeline handlers | 0 | 38 |
| Connectors | 0 | 18 |
| Crawlers | 1 (iMasters) | 13 specialized |
Takeaways
Arachne proved that scaling isn’t adding features — it’s adding architecture. Every evasion layer, every search layer, every agent tool was a response to a real problem, not speculation.
And the biggest lesson: when the tool gets good enough, external agents come to it — that’s how the MCP server and SDK were born.