Ashish Tiwari | Search AI Specialist @ Elastic
Advertised length is not usable length.
Embedding recall has a hard ceiling.
Inside Agent Builder, with the constants.
gap between the largest advertised window and its measured effective length.
| Model | Claimed | Effective |
|---|---|---|
| LWM (7B) | 1M | <4K |
| GPT-4 | 128K | 64K |
| Yi (34B) | 200K | 32K |
Accuracy by answer position, 20 document QA.
NoLiMa: needle and distractor share no words with the question. No keyword overlap to latch onto.
18 models. 194,480 calls. Same result.
| System | Evidence only | Full history | Drop |
|---|---|---|---|
| GPT-4o | 0.870 | 0.606 | -30% |
| Llama 3.1 70B | 0.744 | 0.334 | -55% |
Same model and same question, so the only thing that changed is who picked the context.
80,887 threads
65,624 threads
9,732 threads
An empty result fails loudly. Half an answer just sounds right but could be risky.
One vector of size d can only ever express so many different answers.
BEIR datasets where plain keyword search beat every dense retriever tested.
Adding keyword search to embeddings cut failed retrievals by nearly half.
Add a reranking pass on top and it drops again, to 1.9%.
Same pipeline, same corpus, and the stores land twenty points apart.
80 keyword + 80 vector
→ rank fusion
→ cross encoder rerank
time decay + usage boost
per user access control
Skills load only when the agent needs them. Everything else stays a lightweight stub.
fewer input tokens
fewer input tokens
"Eventually the context window fills up, fidelity drops and token costs climb."
Keep what matters for the next turn, rather than dropping the oldest.
Results sit in an in-memory file store and enter context only when needed.
Chats past 30 turns, at up to 40% lower token cost.
Five to watch: recall@10, MRR, tokens per turn, compaction events, and partial hits.
Ashish Tiwari | Search AI Specialist @ Elastic
Connect on LinkedIn