So if I'm reading this right, it's basically a specialized embedding of a RAG, and then a model in the middle takes the agent's queries and returns responses in a form that is injecting into the main model's input token stream directly i.e. skipping actual text. And the injection format is INCREDIBLY token efficient. Sort of injecting a vibe rather than quotable text. Their results of "good on concepts, less good on quotes/exact text retrieval" seem to intuitively line up with that.
It looks like another difference with RAGs is that instead of reranking the doc vectors and taking the top 5 or so, this paper mixes all the docs (weighted by relevance) into a fixed number of "memory tokens".
It looks like this helps avoid a problem I've had with RAG - when a fact isn't local to a single patch of text.
It looks like this helps avoid a problem I've had with RAG - when a fact isn't local to a single patch of text.