Muhammad Fari Madyan
Hire me

Testing Elasticsearch Semantic Search: A Golden Set, Typo Tolerance and k6 Load Tests on a Real Blog

A reproducible case study on this blog's 55 articles: six Elasticsearch configurations measured with a labelled golden set, generated typos, negative queries and a k6 load test that also scores relevance under load. The biggest win came from what was embedded, not from the model.

Muhammad Fari Madyan

ยท 19 min read

Baca dalam Bahasa Indonesia โ†’

๐Ÿ‡ฎ๐Ÿ‡ฉ Baca dalam Bahasa Indonesia


The Question

Someone asked me: "If you had to stress test a search endpoint, what would you use? I want to judge its semantic search and its typo tolerance."

That sentence holds two different tests:

  • Quality: does the right article come back when the query is misspelled, paraphrased or in another language? That is an evaluation with labelled data, not a load test.
  • Load: how many searches per second does it handle before latency breaks, and do the results get worse when it is busy?

Rather than answer in the abstract, I ran both against something real: the search over this blog. 55 articles, 49 of them in English and Indonesian, 6 in only one language. Everything below can be reproduced from the repository (tools/search-eval), and every number comes from those runs.

What came out of it, in short:

  • Embedding the full text of each article (the default way to use semantic_text) scored no better than plain BM25: nDCG@10 0.739 against 0.756.
  • Embedding only the title and description of each article, with the same model, raised it to 0.888. Cross-language queries went from 0.034 to 0.643.
  • Typo tolerance mostly came for free. Plain BM25 kept its quality on misspelled multi-word queries; fuzziness: AUTO mainly made the top result more stable, and it turned a correct "no results" into wrong results.
  • No configuration could tell a nonsense query apart by its score. "Resep rendang padang" (a beef curry recipe) scored 0.918 in semantic search, the same as real answers.
  • A load test that repeats the same queries measures Elasticsearch's inference cache, not the model. With the cache off, one model allocation topped out at about 13 inferences per second, and past that point requests waited 10 seconds and failed with HTTP 408.

1. The Setup

PartChoice
EngineElasticsearch 9.1.0, single node in Docker (1 GB heap), on an Apple Silicon laptop
Model.multilingual-e5-small, Elastic's packaged multilingual E5, one allocation with two threads
Semantic fieldsemantic_text, which chunks text (sentence-based, 250 words, 1 sentence overlap) and embeds every chunk
Lexical fieldstitles, descriptions and bodies, with the english and indonesian analyzers
Corpusone document per article; both language versions go into the same document

I indexed the articles as plain text: code blocks, links and the "Series Navigation" blocks removed. Those blocks list the titles of every other part of a series, so leaving them in makes every part of a series match every other part's keywords.

Indexing cost

Indexing 55 articles took 875.6 seconds. Almost all of that is the model embedding about 94,000 words on CPU. BM25 indexing of the same text takes a second. Semantic search moves cost to ingest time: re-indexing after a model or chunking change is not free, so plan for it.

First pitfall: the model download

Elasticsearch downloads the 470 MB model itself. On my connection (about 350 KB/s) the download broke after five minutes with Model importing failed due to an IOException [java.io.IOException: Premature EOF]. The fix that works on any connection: download the three model files with a client that can resume (curl -C -) and point Elasticsearch at the folder with xpack.ml.model_repository=file://.... This is the documented approach for air-gapped clusters, and it is the one I would use in production anyway, so deployments do not depend on a download at startup.

2. The Golden Set

A golden set is a list of queries with the documents that should come back, graded:

  • 3: the article that answers the query
  • 2: clearly relevant, a good second result
  • 1: partly relevant, fine anywhere in the top 10

I wrote 47 queries in six kinds, because "is search good?" is really six different questions:

KindnExampleWhat it tests
exact13optimistic locking spring boot examwords from the article itself
paraphrase12stop two requests from overwriting the same exam answer at the same timesame intent, different words
crosslang6writing someone else's final thesis for money โ†’ an Indonesian-only article about joki skripsiquery language differs from the article's only language
slang5gmn cara deploy fastapi ke vps pake dockerinformal Indonesian with abbreviations
spacing6nest js api, rubyonrails clean architectureproduct names split or joined
negative5resep rendang padang, flutter bloc state managementnothing in the blog answers it

Three of the negatives are near misses on purpose. The blog has React Native, Python and Android-adjacent articles, so flutter bloc state management, golang goroutine worker pool and kotlin jetpack compose android app sound like they belong. A search engine that confidently returns something for them is misleading its users.

Typos are generated, not hand-written

Hand-written typos tend to be the ones the author thinks of. I generated them instead (lib/typos.js): for each exact, paraphrase and slang query, one variant with one edit and one with two edits. An edit is a deletion, a transposition, or an insertion or substitution with a neighbouring key on a QWERTY keyboard. The first letter is never touched, and words shorter than four letters are left alone. The generator is seeded, so the same seed always gives the same 60 variants:

optimistic locking spring boolt exam          (1 edit)
optimistic lockin sprint boot exam            (2 edits)
watching studengs live while they take an online tyest
gmn cara deploy fastapi ke vps pake docklee

Together that is 107 queries.

3. What Gets Measured

MetricQuestion it answers
nDCG@10Are the best documents near the top? Graded, so a 3 at rank 1 counts more than a 1.
Recall@10Did every relevant document make the first page?
MRRHow far down is the first good (rated 2+) result?
Zero-result rateHow often does a real query return nothing?
Typo robustnessFor each typo variant compared with its clean query: nDCG ratio, overlap of the two top-10 lists, and whether the top result stayed the same.
Negative leakageIf you cut results below the score of the weakest 10% of real answers, how many nonsense queries still show results?

With a few dozen queries per kind the numbers are noisy, so the evaluator also reports a 95% bootstrap confidence interval for the overall score. A difference smaller than that interval is not a finding.

4. Four Configurations, and a Surprise

I started with four configurations:

  • bm25: plain BM25 across titles (boost 3), descriptions (boost 2) and bodies
  • bm25_fuzzy: the same with fuzziness: AUTO
  • semantic: a semantic query on the full-text semantic_text field
  • hybrid: Reciprocal Rank Fusion (RRF) of bm25_fuzzy and semantic

The surprise was semantic search on cross-language queries, the one thing a multilingual model should be good at. It was the worst of the four. For four of the six cross-language queries, the right article ranked between 31st and 46th out of 55. The query about writing a thesis for money came back with an article about IDOR vulnerabilities.

Hypothesis 1: missing E5 prefixes (ruled out)

E5 models are trained with query: and passage: prefixes, and work worse without them. So I checked whether Elasticsearch adds them. With the inference API, an embedding requested with input_type: search is identical to one of the text with query: typed in front (cosine 1.0000), and differs from an ingest embedding of the same text. Elasticsearch handles the prefixes. Not the cause.

Hypothesis 2: chunk dilution (confirmed)

With semantic_text, an article becomes many 250-word chunks, and a query is scored against those chunks: a document is as good as its best-matching chunks. A long article has many generic passages (setup steps, conclusions, "in this series") and one of them is always somewhat close to any query. Short, single-language articles have fewer chances. All six cross-language targets are single-language articles.

To test it, I embedded only each article's title and description (both languages) and ranked the articles by cosine similarity to each paraphrase and cross-language query. Same model, same queries:

QueryBest answer's rank, title + descriptionBest answer's rank, full-text chunks
c04 when should an early stage company hire a contract frontend engineer246
c06 writing someone else's final thesis for money338
c01 alasan tidak memakai kontainer saat hosting aplikasi next1331
p07 hosting a react framework site on a cheap server without containers19
MRR of the best answer, 18 queries0.6930.486

The model understood the queries. The way the scores were aggregated over chunks buried the answers.

So I added a second field, summary_semantic (titles and descriptions in both languages), and two configurations that use it:

  • semantic_summary: semantic query on the summary field only
  • hybrid_summary: RRF of bm25_fuzzy, full-text semantic and semantic_summary

Adding the field did not need a full re-index. A mapping update plus a bulk partial update of that one field took 39.9 seconds, against 875.6 seconds for the full text, because only the new field was embedded.

5. Fixing the Golden Set Too

The evaluator lists every unlabelled document that showed up in a top 5 (out/unrated.json). After the first run there were unlabelled documents for 42 queries. Most were noise. Some were not: for watching students live while they take an online test, semantic search returned Laravel E-Learning Part 6, which has a whole section called "Live Exam Monitor Widget". I had not labelled it.

This is how evaluation sets are normally built (pooling): take the top results of every system under test as candidates, then judge them by hand. My rule was to add a label only if the article has a section that answers the query, checked in the article text. Mentions in passing did not count. That added 17 labels, for 84 in total. Without this step a configuration that finds relevant documents nobody labelled gets penalised for it, and semantic search does that more often than BM25.

6. Results

All numbers below are from the final run: six configurations, final labels, 107 queries.

nDCG@10 per query kind

Kindnbm25bm25_fuzzysemantichybridsemantic_summaryhybrid_summary
exact130.9800.9800.9700.9860.9870.990
typo, 1 edit300.8160.7960.8510.8820.8880.924
typo, 2 edits300.7790.7730.7840.8210.8120.877
paraphrase120.5990.5270.7490.7120.8150.845
crosslang60.2880.3980.0340.2710.6430.330
slang50.9100.9100.7290.9890.9730.988
spacing60.9270.8680.9360.9970.9920.997
all clean queries420.7560.7430.7390.8070.8880.855
95% CI0.65โ€“0.850.64โ€“0.840.64โ€“0.840.71โ€“0.900.82โ€“0.950.76โ€“0.93

Typo robustness

Each typo variant compared with its clean query:

Config1 edit: nDCG ratio1 edit: same top result2 edits: nDCG ratio2 edits: same top result
bm251.0090%0.9587%
bm25_fuzzy1.01100%0.9887%
semantic1.0180%0.9367%
hybrid1.0183%0.9470%
semantic_summary0.9790%0.8977%
hybrid_summary0.99100%0.9487%

(A ratio slightly above 1 is noise: a typo variant happened to rank a relevant document higher than its clean query did.)

Negative queries

ConfigScore cut-off (weakest 10% of real answers)Negatives above the cut-off
bm2512.6521 of 5 (flutter bloc state management)
bm25_fuzzy12.8761 of 5 (same)
semantic0.9211 of 5; the other four between 0.910 and 0.918
semantic_summary0.9052 of 5 (resep rendang padang 0.914, kotlin jetpack compose android app 0.909)
hybrid, hybrid_summaryn/aRRF scores come from ranks, not from similarity

7. What the Numbers Say

Typo tolerance: mostly free, and fuzziness has a price

On multi-word queries, plain BM25 already tolerates typos: one misspelled word out of five still leaves four that match. Its nDCG ratio for one-edit typos is 1.00. What fuzziness: AUTO adds is stability. The top result stayed the same for 100% of one-edit typos instead of 90%.

The price:

  • Paraphrases got worse, from 0.599 to 0.527. Fuzzy matching lets unrelated words that are one edit apart match too.
  • "No results" disappeared. resep rendang padang returns nothing with plain BM25, which is correct. With fuzziness it returns 38 articles: highlighting shows resep matching "reset", rendang matching "rancang" (Indonesian for "design") and padang matching "pandang" and "papan". Every one of those is a legitimate one- or two-edit match, and every one is wrong.
  • It costs latency: the median took went from about 2 ms to 16 ms.

Typo tolerance matters most for one- or two-word queries, where a single typo is the whole query. If your users mostly type short queries (product names, people), fuzziness earns its cost. For sentence-like queries the semantic side already covers it.

Semantic over full text was not better than BM25

Overall 0.739 against 0.756, with overlapping intervals. Semantic search did help paraphrases (0.749 against 0.599), and hurt cross-language and slang queries. Turning semantic_text on is not an upgrade by itself.

What you embed matters more than which model

The same small model, given one short passage per article instead of dozens of chunks, went from 0.739 to 0.888. Its interval (0.82โ€“0.95) barely touches the full-text one (0.64โ€“0.84), so this is the one overall difference in this study I would call real. For a blog, the title and description say what the article is about; the body says many other things too.

The trade-off shows in Recall@10: semantic_summary finds the best answer first (MRR 0.89 on paraphrases) but misses more of the secondary ones (Recall@10 0.65 against 0.85), because a summary does not mention everything an article covers.

RRF can also make things worse

hybrid_summary is the best or tied for best on five of the seven kinds, and the most typo-robust. But on cross-language queries it drops to 0.330, against 0.643 for the summary alone. RRF only sees ranks: when BM25 and full-text semantic both rank the wrong articles first, they outvote the one retriever that got it right. Fusion averages opinions; it does not pick the expert.

No score threshold separates nonsense from answers

E5 similarities sit in a narrow band: real answers and a beef curry recipe both land between roughly 0.90 and 0.94. A min_score would cut real answers before it cut the nonsense. If "no results" matters to your product, it needs a different mechanism: requiring a lexical match (for example minimum_should_match on the BM25 side), a reranker trained for relevance, or a classifier for "is this query in scope".

What I would ship for this blog

hybrid_summary, with two follow-ups: a better answer for cross-language queries (weighting the summary retriever higher, or translating titles at index time), and a separate "no results" rule. I would not tune weights on these 47 queries and call the result proof: with this few queries, tuning on the test set is how you overfit.

8. The Load Test

The k6 script (k6/search-load.js) does three things a plain "hammer one URL" test does not:

  • It sends the query mix, not one query. A single repeated query only tests a cache.
  • It scores every response with nDCG@10 against the golden labels, so "quality under load" is a metric next to latency, not a guess.
  • It tags every request with its load step (6 rps, 12 rps, ...), so the summary shows where latency breaks instead of one average over the whole run.

The stress profile ramps through four steps (30 seconds up, one minute held), with a p95 budget of one second and a 1% error budget.

The inference cache will lie to you

After a few runs, semantic queries that had taken 60โ€“80 ms took 2 ms. The deployment stats explained it: a trained model deployment keeps an inference cache, by default as large as the model (448 MB here), and 1,286 of the first 1,920 inferences were answered from it. A load test that replays the same 107 queries measures that cache.

My first fix was to give every request a fresh random typo. It was not enough: a short query only has a few dozen one-edit variants, and after a few thousand requests they repeat. In that run 53% of inferences still hit the cache. For the cold numbers below, I restarted the deployment with cache_size=0 and verified that the cache hit count stayed at zero.

Results

All runs on the same laptop, one model allocation with two threads.

RunStepp95 latencyFailednDCG@10 (successful responses)
bm25_fuzzy, no model10 / 20 / 30 / 40 rps40 / 32 / 31 / 30 ms0%0.78
hybrid_summary, golden queries, cache on (100% hits)10 / 20 / 30 / 40 rps90 / 89 / 88 / 89 ms0%0.88
semantic_summary, fresh typos, cache off6 / 12 rps113 / 115 ms0%0.87
18 / 24 rps10 s (timeouts)53% of the whole run
hybrid_summary, fresh typos, cache off3 / 6 rps299 / 368 ms0%0.90โ€“0.91
9 / 12 rps10 s (timeouts)57% of the whole run

What the load test says

The model is the bottleneck, and it has a hard ceiling. Without the cache, one inference took 75 ms on average. One allocation runs one inference at a time, so it tops out at about 1000 / 75 โ‰ˆ 13 inferences per second. That matches the data: semantic_summary (one inference per query) breaks between 12 and 18 rps, and hybrid_summary breaks between 6 and 9 rps. BM25 never came close to a limit at 40 rps.

hybrid_summary runs the model twice per query. The inference count rose by two per request: each semantic clause embeds the query text on its own, even when the text is the same. Two semantic retrievers halve the capacity. A query embedding computed once and reused would avoid it.

Past the ceiling, requests do not fail fast. They queue behind the model and fail after exactly 10 seconds with HTTP 408: timeout [10s] waiting for inference result. I reproduced it with 200 concurrent unique queries: 44 succeeded, 156 got the 408. For a user that is a ten-second spinner followed by an error. The application in front of Elasticsearch should use a shorter timeout and fall back to the BM25 results, which cost 30 ms and never hit the model.

Quality did not degrade before the failures started. Per kind, the golden-query run under load matched the offline evaluation almost exactly (typo1 0.927 against 0.924, paraphrase 0.846 against 0.845, crosslang 0.321 against 0.330). There was no silent degradation; the failure mode is errors, not worse results. That is also why nDCG of successful responses has to be read together with the error rate: a request that times out is a zero for the user, and it does not appear in that average.

Capacity planning uses the cold numbers. The cache makes popular queries cheap (90 ms flat at 40 rps), but long-tail queries are mostly unique, and the cold ceiling is what a burst of them hits. Throughput scales with number_of_allocations (each needs its own CPU threads) and per-query latency with threads_per_allocation; on a single small node, the cheapest win is not running the model twice per query.

9. Checklist

  1. Split quality from load. A load test answers "how fast", a golden set answers "how good". You need both.
  2. Label by query kind, including negatives and near misses, and report every kind separately. One overall number hides a 0.034.
  3. Generate typos, seeded, with a known number of edits. Compare each variant with its clean query.
  4. Pool unlabelled results from every configuration and judge them, or you will penalise the system that finds what you forgot.
  5. Report confidence intervals. With 5 or 6 queries per kind, most per-kind differences are noise.
  6. Check what is embedded, not only which model. Chunk dilution is invisible until you compare against a short summary.
  7. Do not trust min_score with embeddings without measuring negatives.
  8. Break the inference cache in load tests, or you are benchmarking a hash map.
  9. Measure relevance during the load test, not only latency.

10. Reproducing It

BASH
# Elasticsearch 9.1 with the model loaded from a local folder (see tools/search-eval/README.md)
cd tools/search-eval
node make-queries.mjs     # golden set + 60 seeded typo variants -> out/queries.json
node setup-index.mjs      # inference endpoint, index, 55 articles (~15 min on CPU)
node evaluate.mjs         # six configurations -> out/eval-report.md

# cold run: restart the model deployment with cache_size=0 first (README, "Measuring the cold cost")
docker run --rm -i --network container:es-search-eval -v "$PWD":/work -w /work \
  grafana/k6:1.0.0 run -e CONFIG=hybrid_summary -e QUERY_MODE=fresh \
  -e PROFILE=stress -e PEAK=12 -e P95=1000 k6/search-load.js

To test your own application rather than Elasticsearch directly, both tools take your endpoint: node evaluate.mjs --api "https://your-app/api/search?q={q}" and k6 run -e API_URL=.... Replace golden-set.json with queries and labels from your own domain; the rest stays the same.

Limitations

  • 55 documents, one labeller (me), 47 queries. The method transfers; the exact numbers do not.
  • One laptop running Elasticsearch, the load generator and other containers at once. Latency numbers are relative between configurations, not a capacity plan for a server.
  • The trial license was enabled so that every machine learning feature was available; check what your subscription includes before relying on these features in production.
ShareLinkedInXWhatsApp

Written by Muhammad Fari Madyan

Senior Full Stack Engineer and co-founder of LampungDev. I write up what I build so others can learn from it.

Work with me
more writingBack to all articles โ†’next โ†’System Design Roadmap Part 1: Networking Basics โ€“ Packets, TCP/UDP, TLS & REST

Muhammad Fari Madyan

Senior Full Stack Engineer building web and mobile systems that hold up in production. Co-founder of LampungDev. Based in Bandar Lampung, Indonesia (UTC+7), working remote.

Work

Case studiesCahTixExam platform, 4 stacksOpen source

ยฉ 2026 Muhammad Fari Madyan

Designed and built by Muhammad Fari Madyan