Beyond the Transformer Stack

Mamba, RWKV, RetNet. Architectures that are not Transformers wearing a hat.

Published: 2 August 2026
Reading time: 2 min read
Equations: 2 equations

KAN (Kolmogorov-Arnold Networks)

KANs (Liu et al., 2024) challenge the fundamental structure of MLPs. In a standard MLP, there are fixed activation functions (like ReLU or SiLU) on nodes, and learnable linear weights on the edges.

KANs invert this based on the Kolmogorov-Arnold representation theorem. In a KAN, the nodes simply sum their inputs, and the edges contain learnable non-linear functions (parameterized as 1D B-splines).

Benefits: KANs are highly interpretable (you can inspect the learned 1D curves) and avoid catastrophic forgetting in continual learning. They can achieve matching accuracy to MLPs with far fewer parameters, though their training wall-clock time is significantly slower due to the lack of hardware-optimized spline operations.

Standard MLP

y = σ (W x)

KAN Edge

ϕ (x) = w (b (x) + \sum c_{i} B_{i} (x))

Liquid Neural Networks

Liquid Neural Networks (Hasani et al., 2021) are continuous-time recurrent networks inspired by the nervous system of C. elegans.

Unlike traditional RNNs where the 'time constant' (how fast the state updates) is fixed, Liquid networks feature input-dependent time constants. Their internal equations literally adapt their structure to the incoming data stream, making them highly resilient to noise and out-of-distribution shifts (like changing the frame rate of a video or encountering turbulence in a drone flight).

While highly parameter-efficient (solving autonomous driving tasks with ~20,000 parameters), scaling them to LLM sizes remains an open research problem due to the complexity of solving their differential equations during training.

Test-Time Compute (o1 / o3)

For years, scaling laws focused on training compute (pre-training flops). In late 2024, models like OpenAI's o1 introduced scaling laws for inference compute (test-time compute).

Instead of generating an answer instantly, the model uses RL-trained latent reasoning processes to 'think'. It generates multiple possible reasoning chains, critiques its own steps, backtracks on errors, and explores a search tree of solutions before outputting the final answer.

This mirrors AlphaGo's Monte Carlo Tree Search (MCTS). The longer the model is allowed to 'think' at inference time, the higher its accuracy on complex math and coding tasks. This shifts the compute bottleneck from massive pre-training runs to dynamic allocation at inference.