Studies — the week I measured my artificial memory
The question nobody asks about their own system
Every RAG system accumulates data at the same speed it generates false confidence. You index a document, ask a question, get a plausible answer, and move on. Nobody asks the embarrassing metric: if I actually tested this memory, how often is it right?
For months I treated my agent’s memory layer (Yurumi, the local-first memory layer of my ecosystem) as a functional black box. Ingestion went in, answers came out. Until the week I decided to do what I do with any project of mine: measure it. The result was an audit in three layers — the logical registry, the vector index, and a public benchmark — and the numbers told a more interesting story than I expected.
Layer 1 — the logical registry
Every memory starts in a ledger. Here, a simple SQLite database stores each knowledge base and its documents. The query is trivial, but it’s the first honest snapshot:
import sqlite3
c = sqlite3.connect("kb_registry.db")
print(c.execute("SELECT COUNT(*) FROM kb").fetchone()) # (32,)
print(c.execute("SELECT COUNT(*) FROM doc").fetchone()) # (551,)
print(c.execute("SELECT SUM(char_count) FROM doc").fetchone())
# (2242788,) — ~2.24 million characters indexed32 bases created, 551 documents, just over 2 MB of raw text. But looking only at the registry is reading the book’s cover. The schema has a classic pitfall of memory systems: creating a base doesn’t mean it has content.
Layer 2 — the vector index lies when unaudited
The second number came from the main Qdrant collection, the vector store backing semantic search:
collection yurumi_memory
points: 8,528
dimension: 768
distance: CosinePause on that comparison: 551 documents in the registry, 8,528 vectors in the index. The math only closes once you understand the pipeline: each document becomes dozens of chunks, each chunk becomes one 768-dimension vector generated by the local embedder.
And here lives the first auditing lesson: the collection has a single vector configuration (768, Cosine). That’s good news and a trap at the same time. Good, because everything stays consistent. A trap, because swapping the embedding model requires reindexing everything — dimension is fixed at collection creation, and the current embedder (a Portuguese-language embeddings model running locally) produces exactly those 768 dimensions.
curl -s http://127.0.0.1:6333/collections/yurumi_memory \
| jq '.result.config.params.vectors'
# {"size": 768, "distance": "Cosine"}
curl -s -X POST http://127.0.0.1:7997/embeddings \
-H "Content-Type: application/json" \
-d '{"input": ["sanity check"], "model": "colibri"}' \
| jq '.data[0].embedding | length'
# 768The sanity check on both sides — collection and embedder — must return the same number. If they differ, the entire next ingestion fails silently or, worse, indexes dimensionally incompatible garbage.
Layer 3 — the benchmark that handed back humility
With the inventory done came the uncomfortable part: measuring quality. I picked LoCoMo, a public benchmark for long conversations built to test exactly this — whether a memory system retrieves the right fact at the right time.
I ran a baseline using the default local embedder, no reranker, no tuning. The result reminded me why benchmarks exist:
| Metric | Baseline (PT-BR embedder, no rerank) |
|---|---|
| Recall @1 | 6% |
| Recall @5 | 13.2% |
| Recall @10 | 17.8% |
| MRR | 0.092 |
A recall@1 of 6% means: 94 out of 100 times, the first result is not the right evidence. A number like that, hidden, becomes hallucination dressed as confidence — the system retrieves something similar, and the LLM stitches a coherent answer on top of the wrong piece.
The good news came along with it: the same evaluation script exposes the levers. Swapping the embedder (--embed-name BGE) and enabling the reranker (--rerank) are ready-made flags in the pipeline. So the low baseline isn’t a sentence — it’s a measured starting line, something most personal memory systems never had.
What the audit changed day to day
Three habits survived the week:
- Dimensional invariant on both sides. Every time I touch the embedder or the collection, the
768 == 768check runs before any ingestion. One line of curl that prevents weeks of silent drift. - Registry and index are different sources of truth. SQLite says what should exist; Qdrant says what can be found. Auditing = comparing both numbers and explaining the difference (chunks).
- Baseline before optimization. The 6% recall@1 was uncomfortable to write down — but it became the denominator that turns “I swapped the embedder and it got better” into “it went from X% to Y%”.
Lessons
- A memory system without metrics is hallucination with delay. The pipeline worked “perfectly” until someone measured it. Working and being right are different properties.
- Aggregate numbers hide structure. 551 documents vs 8,528 vectors looks like an inconsistency until you understand chunking — and it’s a red flag if you don’t.
- Public benchmarks exist to humble you cheaply. Running LoCoMo cost an afternoon. Not running it would cost months of stacked wrong decisions.
- The difference between an amateur project and a real system is the baseline. It’s not the technology — it’s knowing your starting number and admitting it in public.
The next step is already mapped: rerun the same benchmark with an alternative embedder and the reranker active, then publish the comparison. The goal isn’t to break records — it’s to turn “I think it improved” into “it improved by 4.3 points”.