When I’m in my Zen garden (mowing), I’m often mentally sketching an idea I’ve had for a while around the notion of a pyramid of summaries of varying detail that will help a RAG pipeline provide context to queries in a useful way, such that a hing-level inquiry can stay high-level, and a series of inquiries can get an increasingly fuller picture. My vaporous implementation involves an LLM that occupies a very busy part of the ingestion pipeline.
SproutRAG is a project I’ve just ran across that appears to build a similar structure in a clever way without the LLM. It builds a binary tree grown bottom-up from sentence-level chunks, and instead of asking a model to summarize what’s beneath a node, it just averages the embeddings of that node’s two children.
The tree grows by merging whichever sentence pair has the highest mutual attention, using learned weights across heads and layers so the merges track semantic co-relevance instead of just favoring whatever sits next to what on the page, a trap the paper calls proximity bias. Every internal node ends up holding a “progressive embedding,” nothing more than the average of its children, so retrieval can stop at one leaf for a narrow fact or climb several levels for something broader, off the same structure.
No extra LLM calls at index time, and none at query time either, since the small encoder doing the attention work is just running the encoding pass it already does. It’s a pretty cool idea that’s both efficient and easy to think about.