What Remains Unsolved

The RULER benchmark (Hsieh et al., 2024) exposed the gap between a model's claimed context window and its functional context — the length over which it can actually reason reliably.

RULER goes beyond simple Needle-in-a-Haystack (NIAH) — where a key fact is planted in a long document — to 13 task types: multi-hop tracing (follow chains of references), aggregation (count all occurrences), variable tracking, and selective filtering, all tested from 4K to 128K tokens.

Headline finding: of 17 long-context models benchmarked, only half could effectively handle 32K tokens despite all advertising 32K+. GPT-4 degraded least but still dropped 15.4 points from 4K to 128K. Most models show a sharp performance cliff well before their advertised limit.

MRCR v2 (Multi-Range Context Retrieval): tests finding and integrating multiple non-adjacent pieces of evidence — significantly harder than single-needle NIAH and a better proxy for real-world usage.

SubQ (Subquadratic Sparse Attention, 2026) represents the newest architectural bet: content-dependent sparse attention that selects which positions matter for each query, computing exact (not approximated) attention only over those positions. This aims to preserve the retrieval precision of full attention while achieving sub-quadratic scaling.

Vendor-published figures (May 2026, from an Appen white paper): • 56.2× prefill speedup at 1M tokens vs FlashAttention-2 (21,410ms → 380.96ms on B200) • 62.8× reduction in attention FLOPs at 1M tokens • RULER@128K: 95.6% • MRCR v2: 86.2% • SWE-Bench Verified: 81.8%

Architectural position: distinct from prior approaches — unlike fixed-pattern sparsity (Longformer/BigBird) its routing is content-driven; unlike SSMs it preserves exact arbitrary-position retrieval; unlike hybrids (Jamba) it has no load-bearing dense attention layers.

⚠ These are vendor-published figures pending broad independent reproduction as of mid-2026.

Epoch AI's analysis ('Will We Run Out of Data?', Villalobos et al.) estimates the effective stock of quality, repetition-adjusted human-generated public text at approximately 300 trillion tokens, with a 90% CI spanning 100T–1000T.

Their projection: this stock will be fully utilized between 2026 and 2032, with a median around 2028 — possibly sooner if training continues at the 'overtrained' rate. LLaMA 3 was estimated to be trained ~10× the Chinchilla-optimal number of passes on some datasets.

Mitigations under active research: • Synthetic data: generate training data using existing models — especially valuable for math, code, and reasoning where correctness is verifiable • Multimodal data: video, images, and audio are largely untapped • Data efficiency: architectural improvements that learn better from fewer tokens • Curriculum learning: better ordering and weighting of training examples

We cannot reliably explain what individual attention heads, neurons, or layers 'do' in a trained model. We can detect that a layer encodes syntactic structure using probing classifiers, but we cannot trace a specific output to specific mechanisms with confidence.

Current research directions: • Sparse Autoencoders (SAEs): decompose residual-stream activations into monosemantic 'features' — units that activate for specific interpretable concepts (e.g., 'this token is part of a DNA sequence'). More interpretable than raw neurons, which tend to be polysemantic. • Circuit analysis: identify the minimal set of attention heads and MLP neurons implementing a specific capability. • Probing: train linear classifiers on hidden states to detect what information is encoded at each layer.

The alignment gap: RLHF and DPO successfully align surface behavior — models follow instructions, refuse harmful requests, sound more helpful. But they do not reliably fix factual accuracy. An RLHF-trained model can be confidently and fluently wrong. The mechanisms for truthfulness and for instruction-following are largely orthogonal.

Hallucinations are not bugs in the code; they are a fundamental consequence of how next-token prediction works. They fall into distinct categories:

• Closed-Domain (Factual Contradiction): The model contradicts information directly provided in the prompt. • Open-Domain (World Knowledge): The model confidently states something factually incorrect about the world. • Logical (Reasoning Flaw): The model makes a mathematical or deductive error.

The root cause: LLMs do not have a grounded world model or a 'database of truth'. They have a statistical map of language. When they don't 'know' an answer, they don't naturally output 'I don't know' unless heavily RLHF-trained to do so. Instead, they sample the most plausible-sounding tokens, which seamlessly constructs a lie.

'When a measure becomes a target, it ceases to be a good measure.'

Classic benchmarks like MMLU (Massive Multitask Language Understanding) are increasingly saturated (models scoring 90%+). However, this often doesn't correlate with real-world usefulness.

Two problems: 1. Data Contamination: Benchmark questions frequently leak into the massive pre-training sets. The model isn't reasoning; it's retrieving memorized answers. 2. Over-optimization: RLHF and specialized fine-tuning heavily optimize for benchmark formats (like multiple choice) at the expense of general capabilities (the 'alignment tax').