I read a writeup on MemRL out of Shanghai Jiao Tong University and it’s going down a road that I’ve thought a lot about. Instead of dinking with the model weights, the LLM stays as delivered and all the tweaking happens in the RAG algorithm instead. The researchers call the frozen model the *cortex*, which is probably something smarter folks than me will take exception to, but the mechanism is slick enough that I’ll allow it.
Instead of storing plain text chunks and embeddings the way most RAG setups do, MemRL organizes memory into [intent, experience, utility] triplets: the query, the solution trajectory that was tried, and a Q-value scoring how well it actually worked. Retrieval happens in two passes. First a normal semantic-similarity pass like RAG, then a re-rank by Q-value that prioritizes strategies with a track record over strategies that just sound close. Every time the agent tries something and gets feedback from the environment, that Q-value updates. Over enough iterations the agent starts ignoring memories that read well but perform badly.
The numbers back it up more in the hard cases than the easy ones. They saw roughly a 56% relative improvement over MemP on ALFWorld, which is the most exploration-heavy of the four benchmarks they tested. What’s maybe more interesting was co-author Muning Wen’s answer on cost: “Our Q-value calculation is performed entirely on the CPU.” No GPU tax for the part of the system that’s processing the feedback. On my AI lab machine, my GPUs are pegged, but I’ve got CPU to burn.
Wen conceded bad feedback “may spread more widely” through the memory bank, but that’s a system-design caveat, not a deal-breaker. To me, it’s a nice solution to improving RAG outcomes over time, without the cost or downsides of a fine-tuning effort that goes wrong.