We ran an eval on our own search. Full post bodies barely moved the score.
We expected chunking full post bodies into /search's corpus to be an obvious precision win over embedding just titles and descriptions. At 21 posts, the aggregate score did not move. Here is the eval that showed it, and why we shipped the change anyway.
We built an eval harness for our own search and ran it. The number that mattered most was smaller than we expected.
The last post about /search ended with an honest limit: it only searched title, subtitle, and description, not full post bodies, to keep the corpus light. The obvious next move was to chunk in the full body. Before shipping that as an improvement, we wanted a number, not a vibe. So we built an eval harness, ran it against both corpus shapes on both model tiers, and the result was not the clean win we expected.
The question we were avoiding
"Full-text search should beat title-only search" sounds obvious. It also sounds obvious that a bigger model beats a smaller one, that more data beats less data, that a newer library beats an older one. Most of those obvious claims are true on average and false in a specific, small case, and this site is a specific, small case: 21 posts total. We did not want to ship a change on the strength of "it feels more thorough" without checking whether it actually retrieved better.
How we evaluated it
We wrote scripts/eval-search.mjs, a small Node script that runs entirely offline (ternlight ships a Node entry point alongside its browser one, so this needed no browser and no build step) and imports the real `posts.ts`, `news.ts`, and the same chunking function `/search` uses in the browser, so the eval can never drift from what actually ships.
The test set: one query per post, 21 total, hand-written to describe what the post is about without reusing its title's words. "Phone that can't prove it's safe" for the EU digital ID post, not "digital ID wallet." "Internet providers blocking shared IPs over football piracy" for the LaLiga story, not "LaLiga IP blocking." The point of semantic search is understanding meaning without keyword overlap, so the eval set had to actually test that instead of handing the model the answer.
For each of 4 configurations, mini and base tiers crossed with the old corpus (title + subtitle + description, one row per post) and the new one (full body, chunked to ternlight's 128-token limit, one row per ~70-word chunk), we embedded the corpus, embedded every query, ranked posts by their single best-scoring chunk, and recorded where the correct post landed.
const p1 = ranks.filter((r) => r === 1).length / n; // did the right post win outright?
const p3 = ranks.filter((r) => r <= 3).length / n; // did it make the top 3?
const mrr = ranks.reduce((sum, r) => sum + 1 / r, 0) / n; // how far off, on average?The results
Every single configuration landed the correct post first for 20 of 21 queries, and inside the top 3 for all 21. Chunking in the full body did not raise P@1, P@3, or P@5 at all, and on the base tier it took MRR down very slightly, not up. If we had shipped this purely on the aggregate table, the honest conclusion would be: do not bother.
What actually changed
The aggregate score being flat hid the more interesting fact: every configuration missed a different query, not the same one. Title-only search (mini) missed "internet providers blocking shared IPs over football piracy", ranking a Notion post above the actual LaLiga story. Full-body search fixed that exact query, its best-matching chunk scored 0.683 against the LaLiga post, but it introduced a new miss on "what's coming next for this studio's tools and content", which drifted toward a chunk in the selfhost.directory post that happens to share vocabulary about tools and directories.
Four configurations, four different single misses, zero overlap between them. At 21 queries, a single flipped result moves P@1 by about 5 points, so "flat aggregate" here really means "too small a sample to declare a winner", not "no difference exists." That is a different, more honest claim than either "full-text wins" or "full-text does nothing."
Why we shipped it anyway
Two reasons the P@1 table does not capture. First, explainability: each result on /search now shows the actual matching sentence, pulled from wherever in the post it matched, not the static description every result used to repeat verbatim. That is a real UX improvement invisible to a precision metric. Second, coverage: several posts have detail buried in the body that never made it into their descriptions, a title-only index simply cannot surface that detail no matter how a query is phrased, even if it does not move the aggregate score on this particular 21-query set.
The honest limits
21 posts and 21 queries is a smoke test, not a benchmark. Every number in that table has a margin of error wide enough to swallow the differences between rows. We are not claiming full-body chunking is proven better, or that mini and base are equivalent in general, those are claims that need hundreds of queries and a corpus with real duplicate/near-duplicate content to distinguish. What we are claiming is narrower: on our own real content, at our own real scale, this specific change was not the obvious win it looked like on paper, and we would rather publish that than a cherry-picked query where it clearly wins.
Next step: the eval script is in the repo and reproducible, run `node scripts/eval-search.mjs --verbose` to see every query's result. Try /search yourself, the mini/base toggle and the query latency panel are both still live.