This is part one of two posts on how a model can work with an input that is much longer than its context window and also improve how its reasons over this long context.
Language models have a fixed context window. Even inside that limit answer quality often drops as the input gets longer. A common approach is to summarize older content when the context window fills up. That works for some tasks but it can fail when the answer depends on details spread across the full input.
A recent paper on Recursive Language Model (RLM) takes a different approach. It does not put the long context into the model at all. The input is stored as a variable in a REPL, and the model works on it by writing code.
A REPL as you may know is eval, print, loop. It is an interactive programming session. You type a line of code and run it and you see the result. Then you type the next line. The REPL keeps the variables and state you create, so each step can build on the previous one. If you have typed Python in a terminal and seen the >>> prompt you have used REPL.
In an RLM the model starts with only basic information about the input, such as its length and its first few lines. It can then write code to read parts of the input, split it into chunks, search it, or transform it. It can also call a model on any part of the input from inside of the code. Let's call this sub-call. The results of these sub-calls are saved in variables while the main model only needs to see a short preview. A sub-call can itself be another RLM with its own REPL.
This has benefits
a) The input size is no longer limited by the models context window. The approach can work with inputs containing millions of tokens.
b) The models active context stays relatively small because the original input and intermediate results remain in variables instead of being repeatedly placed into the context window.
c) Code handles much of the bookkeeping.
However here is most important gain.
RLMs are not only about processing inputs that are too large for the context window. They can also increase how much semantic work the model performs over an input. A normal model gets one context window and one generation to reason over everything. An RLM can make additional model calls based on the structure of the task. It can make a semantic decision for every record, process different sections independently, or compare many items with one another.
One downside is cost.
Each sub-call adds tokens and latency. A model that splits the work too finely could make hundreds or thousands of sub-calls for a task that does not require them.
In part 2 I will go over a follow up study which tries to find how much gain comes from recursion versus just use of REPL.