Studies — when I studied the embeddings that think for the system

The question that started the study

In the middle of the ecosystem, two different systems need to understand what the user meant. Arachne searches the web, extracts content, answers questions. Yurumi stores the system’s memory — conversations, decisions, accumulated knowledge. Both depend on an invisible piece: the embedding model, the one that turns text into a vector.

I knew each one used a different model. What I didn’t know — and decided to study — was why each model was where it was, and whether the choice still made sense after months of evolution.

The study started with a simple question: which embedding model delivers the best result for each type of query the system makes? The answer, like almost every honest answer, came in the form of a tradeoff.

The context — two systems, one foundation

Both systems share the same search architecture: input text → embedding model → vector → similarity in the vector store → ranked results. But what each one searches for is different:

• Arachne receives user questions about web content. Queries are short, direct, about varied topics. The embedding needs to be fast and reasonable — an 80% match already gives the LLM enough context to assemble an answer.
• Yurumi searches the ecosystem’s memory. Queries are diffuse — “how was the WSL watchdog configured?” — and the result needs to be precise, because the agent that asks depends on that memory to act. The embedding needs to understand intent, not just words.

Two different problems, which — I suspected — called for different models.

The three models in the study

I picked three candidates that represented different cost-quality spectrums:

colibri (v1.0 local): the lightest model in the ecosystem. Runs on CPU, 50ms per embedding, 384 dimensions. It’s what Arachne uses by default. Perfect for short queries where speed matters more than semantic precision.

bge (base-en-v1.5): the local gold standard. 768 dimensions, runs on GPU, ~200ms. It’s Yurumi’s main embedding. It understands context, captures nuances, but weighs on the batch.

openai (text-embedding-3-small): the paid gold standard. 1536 dimensions, superior quality, but costs per token. It’s Yurumi’s fallback when bge doesn’t find a good enough match.

Each model has a signature that defines the tradeoff: speed, quality and cost form a triangle where you can only maximize two at a time.

The homegrown benchmark

I built a simple test — no academic papers, no standardized datasets. I used 50 real queries the ecosystem had already processed, split into three categories:

• Short queries (2-5 words): “watchdog WSL timeout”, “AI cover lifelog”, “deploy portifolio”
• Technical queries (6-15 words): “how to configure Arachne’s health check on k3s”, “what’s the secret of the /api/recusar rate limit”
• Diffuse queries (16+ words): “what was the decision about Yurumi’s embedding model and why was bge chosen”

For each query, I ran the embedding through all three models and compared the returned top-3 against what I considered the correct result (based on the actual base content).

def benchmark(query, expected, models):
    for name, model in models.items():
        vec = model.encode(query)
        results = search(vec, top_k=3)
        match = 1 if expected in results else 0
        print(f"{name}: {match}/1 — {results}")

What the study revealed

The results confirmed suspicions and revealed surprises:

colibri is surprisingly good at short queries — 80% match with 50ms latency and zero GPU cost. It’s the right model for Arachne, which makes hundreds of queries a day and needs fast responses.

bge is the balanced one: it loses little to openai on technical and diffuse queries, and runs for free on the local GPU. But the cost isn’t only financial — it’s RAM. Each embedding batch consumes ~1.2GB of VRAM, and on low-memory days on the host, bge fails silently.

openai is the gold standard, but the per-token cost makes it unviable as a primary embedding for the ecosystem’s volume (~3000 queries/day). It stays as a fallback — when bge returns low confidence, the query escalates to openai.

Where each one ended up

The study didn’t change the choices — it confirmed them with data. Each model is where it should be:

• Arachne: colibri v1.0. Speed matters more than precision (the LLM covers gaps). 50ms per query vs 200ms would make a difference in the response time the user perceives.
• Yurumi: bge as primary, openai as fallback. Precision matters because wrong memory generates wrong action. The fallback exists for the 16% of diffuse queries bge misses.
• Neither: a single model for both systems. The cost of standardizing would be losing performance in one of the two use cases.

Lessons learned

1. The right model depends on what you’re searching for, not what you have. colibri isn’t “worse” than bge — it’s the right model for short queries where the LLM covers the gaps. bge isn’t “worse” than openai — it’s the right model for the ecosystem’s volume.
2. A benchmark with real data beats a paper. The 50 real queries revealed a pattern no standardized dataset would show: colibri is consistently good at short queries, even with 384 dimensions.
3. Cost isn’t only money. bge is “free” but consumes 1.2GB of VRAM per batch. On low-memory days, “free” becomes unavailable — and the paid fallback saves the day.
4. Fallback isn’t plan B, it’s architecture. openai as bge’s fallback isn’t waste — it’s the valve that guarantees the system never goes blind when the local model fails.

The study didn’t answer “which embedding is best”. It answered something more useful: which embedding is best for each part of the system. And that, in the end, is what a real study should do.

╔══════════════════════════════════════╗
║  Samuel Medeiros                    ║
║  Senior Software Engineer           ║
║  Stack: Python · TypeScript · Rust  ║
║  Projetos: Arachne, Dogwalk,        ║
║            Capivara, TatuEngine      ║
╚══════════════════════════════════════╝