SRDN: A Non-Linear RNN Based on RWKV

July 8, 2026

The transformer architecture dominant in LLMs today has memory requirements that scale linearly with context size during inference. This memory inefficiency is among the most widely tackled problems in foundational model research today. While approaches to handling memory constraints span from infrastructure to the application layer, the most promising opportunity to achieve unlimited context length is in architecture design. RNNs use a fixed number of vectors or matrices to store information instead of a per-token cache. Although Transformers perform much better at scale than the RNN designs that they tended to replace, modern linear RNNs show competitive performance on benchmarks as well as high compute utilization through sequence parallelizable training.

Previously abandoned because of issues with training efficiency, non-linear RNNs such as M²RNN have recently seen increased attention for demonstrating strong state tracking abilities that are absent from both linear RNNs and Transformers. These abilities have been justified theoretically through work noting the limits of Transformers and other architectures through the lens of computational complexity. A proposed "no free lunch" rule asserts that it is impossible for any architecture to be both parallelizable across sequence length and capable in principle of solving arbitrary non-associative sequential problems.*

This conversation is especially relevant in light of discussions about continual learning. While common conception holds that a continual learning system is one in which the base model weights must be updated over time, it seems equally possible that in-context "meta-learning" can achieve the same goal with fixed weights. For this to work, solving the state tracking limitations of modern LLMs is a top priority.

The Architecture

The RWKV architecture is a performant linear RNN centered around a diagonal plus rank one matrix update. The architecture's primary theoretical motivation is a generalization of the delta update rule, established by the DeltaNet architecture as an inductive bias that matches a formulation of the state matrix as updating through online gradient descent. This, when combined with specialized write and erase gates, gives the architecture the ability to retain information selectively.

SRDN (short for State Reading DeltaNet, pronounced "sardine") modifies RWKV by following an intuitive method to 'serialize' the parallelizable token mixer: instead of having each of the projections take in the output of the previous layer alone, we additionally feed information from the state at that layer. To be more specific, we use 4 separate low-rank query-conditioned projections from our state matrix to feed to the key, value, decay, and in-context learning rate vectors, and a similar projection of the diagonal of each matrix into the read vector.

The main delta rule operation in both RWKV-7 and SRDN follows the generalized form

With the projections differing between the 2 architectures:

where $\hat{x}_t$ is RWKV-7's per-projection token-shift mix of $x_t$ and $x_{t-1}$.

Because our low-rank adapters are initialized to 0, SRDN is equivalent to RWKV at step 1 of training. Unlike RWKV, the state update function of SRDN can be non-linear with respect to the state itself, allowing for it to represent non-reducible chains of operations that scale with the sequence length, rather than the number of stacked layers in the architecture. This separates SRDN from most of the other architectures that we test below, with the exception of M²RNN. However, the nonlinearity of M²RNN is limited to a single fixed tanh operation between updates to the state matrix, rather than a variable function of the state matrix itself.

Above: the token mixers of RWKV-7 (left) and SRDN (right).

Results

We test baseline LM abilities on the enwik8 dataset, training with sequence length 256 and batch size 32 for 1,500 steps, measuring loss in terms of bits-per-character (bpc). The SRDN performs competitively, beating every architecture except for the baseline RWKV.

None of the models are fully converged after 1,500 steps. Most of the architectures were still improving at a similar rate to each other at the cutoff, so the ordering between them is stable - but Mamba-3 and the Transformer were improving more quickly than the rest, so their gaps in this table are likely overstated.

Keep in mind that the above test has matched parameters, so the RWKV architecture used here has a much larger FFN size than SRDN. The difference in the FFN ratio decreases with scale because the parameter count of the SRDN adapters grows linearly with hidden dimension while the param count of the overall architecture grows quadratically. As we scale parameter count and training length, the language modeling gap between RWKV and SRDN decreases towards 0.

The best test of the recurrence mechanism itself is the Forward-Referencing Jumps Task (FRJT). While there are evals like parity that are meant to test state tracking abilities, many architectures that can succeed at those will fail here. This task involves repeated iterations of forward jumps and optional changes to the value of a 1-bit register, executed based on the current value of the register itself. It tests the ability to track data-dependent control flow over arbitrary lengths. For each architecture, we train on programs of mixed depths 1-16 for 10,000 steps and measure length extrapolation out to depth 128 across 3 seeds. The SRDN is the only architecture that masters the task at our scale.

Future Work

It's unproven that the demonstrated performance of SRDN will translate to success at larger scale; further experimentation, optimization, and testing will be needed to take it there.

The largest problem facing the development of non-linear RNNs, one that affected this project itself, is the previously mentioned training efficiency constraint - linear RNNs can be trained in parallel across sequences because linear functions are associative; it's for this same reason that those same linear RNNs fail at non-associative computational tasks. There are a few plausible ways to get over this, though.

1. Training methods beyond standard backprop meant to enable parallelizable training, such as SMT. While useful for training on language modeling, they are not capable of training capabilities on nonassociative tasks on their own, so they need to be combined with some amount of standard BPTT to achieve high expressivity.

2. New hardware. Many specialized hardware designers target improved speed on bandwidth-heavy tasks. Even though these improvements are marketed towards improving inference, they also speed up all forms of non-parallelizable training, including our own. In theory, an entirely compute-bound training run would eliminate the gap between a sequentially trained model and one trained in parallel entirely.

3. A larger focus on RL & "hybrid RL" methods. Since our training constraint removes the relative throughput advantage of standard teacher forcing, the ideal training stack for a frontier model may shift towards techniques that combine online exploration with pre-existing data. I think there's a lot of room for very high impact research here.

All the code needed to reproduce the project can be found here.

*I should note that the research threads I'm referencing tend to endorse Chain of Thought on top of Transformers as the ultimate solution, rather than non-parallelizable architectures. CoT has been proven to enable Turing completeness in Transformers under the right conditions, as an internalized scratchpad allows for them to perform inherently serial tasks. However, CoT in its current form can only be applied after pretraining is complete. I argue that in the same way that undersized models will fail to benefit from using overly complicated training data, models which are unable to represent certain patterns during pretraining will receive very little benefit from training on them at all, having to essentially learn to represent them from scratch during RL. Extending this expressive capacity to pretraining should allow for models that can integrate a much wider variety of complex strategies that are especially useful alongside test-time compute.