This is part two of two posts on how a model can work with an input that is much longer than its context window.
In part one I explained the recursive language model (RLM). The model keeps the input in a REPL variable, works on it with code and makes sub-calls to a model on chunks of the input.
The question we ended with was how much of the gain comes from recursion and how much comes from the structured interface (REPL and code) the model uses to work with its input.
A follow-up paper from Apple researchers tested this question.
It proposes a keeping the REPL and makes sub-calls optional. It runs eight attempts in parallel on the same problem. Each attempt has access to the same REPL and can take a different path through the input. It then picks one answer using three signals that come from the model itself.
a) The first signal is agreement. The method keeps only the attempts that reached the most common answer.
b) The second signal is confidence. At each step, the model states how confident it is and the method combines these into one score for the attempt.
c) The third signal is length. The method counts the tokens each attempt used, because a model that is unsure tends to write more. Among the attempts that agree, the method picks the one that was confident at every step and used fewer tokens.
Compared with RLM, this method shows some benefits
a) Accuracy is higher on the same benchmarks under the same time limit.
b) Recursion provides marginal improvement in their benchmarks although there are some benchmarks I feel there is inconsistency between original RLM paper and this one.
c) Results are more stable when the input fits in the context window and gains are more even across task types.
My observations
a) Eight attempts in parallel take about the same time as one RLM run would use roughly eight times the tokens. Getting better results in the same time is not the same as getting better results for the same cost but generally I think this approach limits number of attempts and would be more cost effective in general.
b) But my biggest question was for very long answers there is no simple way to decide that two attempts agree so not sure how we would pick attempts that agree cheaply.
My view after reading both of these is that just like in programming recursion has value in solving certain types of tasks and there could be a decision point in RLM methodology whether a task at hand really needs the recursive sub-call or can use fixed number of parallel attempts.