I read a write-up on a DeepMind paper that got into the limits on how many documents a vector database can practically work with, and the numbers land on the smaller-than-expected side of the scale: a 512-dimension embedding starts breaking down around 500K documents, 1024 dimensions gets you to roughly 4 million, and 4096 dimensions caps out around 250 million. Worse, those are best-case figures for embeddings trained directly on the test labels. Real embeddings hit the wall sooner.
The results are bleak enough that they’re reporting 20% success for a 50K document corpus (and recall that “document” refers to the chunk, not the whole). On a 46-document corpus, the best performer only hit 54%. A vector database failing on 46 documents is the kind of number that makes you stop and re-read the method section.
BM25 and ColBERT-style multi-vector approaches sidestep the limitation to some extent, but in a deep-knowledge corpus setting, simple vector RAG won’t reliably cover the tail.