QUERAIS RESEARCH LABS // ARCHITECTURE DISCOVERY

Intelligence, measured per compute.

We hunt neural architectures with measurably better useful intelligence per unit of compute, memory, energy, latency and network bandwidth — with decentralized heterogeneous inference as a first-class target.

EXP-001 · RUNNINGENVELOPE ≤ 2M PARAMSFAILURES: ON RECORD

CH.01 // METHOD

One loop, falsifiable.

01
Hypothesis
Rank falsifiable questions by expected information gain per unit of compute.
02
Prediction
Commit to a measurable outcome before running anything.
03
Implementation
Pure-PyTorch transformer / ssm / hybrid blocks built from the model registry.
04
Controlled experiment
Matched total parameter count (+/-5%) and identical tokens/steps/optimizer/schedule unless explicitly disclosed.
05
Measurement
Accuracy on task-relevant positions only; param counts, FLOPs/token estimates, host fingerprint, JSON record per run.
06
Analysis
Read curves against the committed prediction; negative results count as evidence.
07
Conclusion
No claim enters STATUS.md without a results/*.json artifact.
08
Next hypothesis
Failures gate retests and feed the next ranked question in the cycle.

Negative results are recorded, not discarded.

CH.02 // LIVE EXPERIMENTS

On the bench.

exp000PASS

Infrastructure smoke test

Do all three model families construct, train-step, and write JSON records end-to-end?

smoke_ok=true — transformer 411,520 / ssm 249,472 / hybrid 330,496 params all step cleanly.

▸results/exp000_smoke.json

exp001RUNNING

MQAR — attention vs selective SSM vs hybrid

At ~800k matched params, which primitive solves multi-query associative recall — and does one attention layer recover the gap in a hybrid?

Transformer d128/L4 reaches 100% eval accuracy by step ~2250 on zoology-faithful MQAR-4 (4000 steps, 822,912 params); SSM and hybrid_1attn runs diverged mid-run (NaN loss, acc 0.0 at cutoff) — rerun pending under the post-F-0002 protocol.

▸results/exp001_all.json
▸results/exp001_transformer_p4.json
▸results/exp001_ssm_p4.json
▸results/exp001_hybrid_1attn_p4.json
▸results/exp001_mqar_p8_transformer.json

CH.03 // THE BOARD

Ranked by information gain.

01H1

Attention is the recall specialist at matched budget

RUNNING

Softmax attention outperforms selective SSMs on multi-query associative recall when total params (~800k), tokens, optimizer, and schedule are matched.

PREDICTION: Transformer MQAR accuracy beats SSM accuracy by >15 points after convergence on kv_pairs=8, vocab 64.
FALSIFIED IF: SSM lands within 5 points of attention — or beats it.
EXPERIMENT: exp001

02H2

One attention layer mostly recovers recall in an SSM stack

RUNNING

Replacing one of six SSM layers with attention recovers >=80% of the H1 accuracy gap while adding <25% of the attention-layer compute share.

PREDICTION: hybrid_1attn >> ssm alone, approaching transformer.
FALSIFIED IF: Hybrid ≈ SSM (no recovery).
EXPERIMENT: exp001

03H3

Recurrence matches attention where recall is absent

PLANNED

On char-level language modeling (low exact-recall demand), selective SSM reaches transformer val loss at matched params/tokens.

PREDICTION: |val loss difference| < 0.05 nats/char.
EXPERIMENT: exp004 — tiny corpus under disk constraints.

04H4

Copying exposes recurrent state bottlenecks

PLANNED

On the copy task, attention reaches near-perfect accuracy once induction forms, independent of prefix length within context; SSM accuracy decays with prefix length.

EXPERIMENT: exp002 — copy task, prefix lengths {8,16,32}.

05H5

Recall gap grows with number of KV pairs (state capacity law)

PLANNED

SSM MQAR accuracy falls monotonically as kv_pairs grows (4→8→16); attention stays high until context limits bind.

EXPERIMENT: exp001 extended to kv_pairs ∈ {4,8,16}.

06H6

State size buys recall sublinearly

PLANNED

Increasing n_state (8→16→32→64) narrows the SSM recall gap but with diminishing, log-like returns — predicting hard capacity walls for fixed-state models.

EXPERIMENT: exp003 — n_state sweep at fixed other budgets.

07H7

Locality decomposition: window attention fails global recall alone

PLANNED

Sliding-window attention (w=12 < distance to needed keys) fails MQAR; a window+SSM hybrid partially recovers via state-mediated indirection.

EXPERIMENT: exp005 — add SWA block type; cheap variant of exp001.

08H8

Learned routing: pay attention only sometimes

PLANNED

A token-level learned router choosing attention vs SSM per layer matches dense-hybrid accuracy at <=70% of its attention compute.

PREDICTION: Routing overhead accounted as quality per additional FLOP.
EXPERIMENT: exp006 — after H1/H2 establish reference points.

09H9

Wall-clock reality of cache-free inference

PLANNED

Despite worse asymptotics, optimized attention beats our naive SSM scan at short T on CPU, and the crossover where SSM throughput wins sits at T >> typical training lengths.

EXPERIMENT: exp007 — throughput sweep T ∈ {23,64,128,256,512}, batch fixed.

10H10

Distributed probes (deferred)

DEFERRED

Expert/state sharding communication volume vs pipeline on simulated heterogeneous nodes.

Expand a row for its committed prediction and falsifier.

CH.04 // LITERATURE LEDGER

Standing on the map.

FAMILY	MECHANISM	NOTE	STATUS
Softmax Transformers F1	Content-based soft addressing over full history; exact token-level retrieval. Train O(T²·d); inference KV cache O(T·d·L) grows without bound.	Vaswani et al. 2017. Baseline implemented (lab/models/transformer.py); basics reproduced in exp000 — learns MQAR above chance with a sane loss curve.	REPRODUCED
Linear attention / fast-weight family F2	phi(q)phi(k)^T outer-product state, O(T); gated/decayed variants keep the same fundamental fixed-state recall limit at small scale.	Katharopoulos 2020, RetNet/GLA/DeltaNet. 'Matches transformer LM quality' holds for perplexity but not recall-heavy evals. Planned GLA-style variant if SSM results warrant.	UNVERIFIED
State-space models (S4/S5/Mamba/Mamba-2) F3	Input-dependent selectivity (Δ, B, C functions of x); linear time, constant state. Mamba-2/SSD connects SSMs to linear attention via chunkwise parallel form.	Gu & Dao 2023/2024. Minimal faithful S6 implemented with custom verified autograd scan; exp001 is the first controlled comparison. Our pure-torch CPU scan runs ~10–15x slower per step than SDPA attention at T=23 — asymptotic wins do not transfer to naive implementations.	PARTIAL
Recurrent LMs (RWKV, xLSTM) F4	RNN formulations competitive on perplexity, sharing the same recall limits as other fixed-state recurrence.	Lower priority than SSMs — similar primitive class.	UNVERIFIED
Hybrids (attention + recurrence interleaved) F5	A small fraction of attention layers (10–25%) recovers most recall/in-context ability while keeping recurrent efficiency (H3/Hyena, Based, Jamba, Zamba, Samba, Griffin/RecurrentGemma, MiniMax-01, Nemotron-H).	The single most replicated practical result across independent groups. Hybrid stack implemented (lab/models/hybrid.py); exp001 tests it — conclusion pending.	PARTIAL
Memory / retrieval augmentation F6	External/compressed memory decouples context cost from state capacity (Memorizing Transformer, Infini-attention, Titans neural memory, TTT layers, RMT).	Could cut KV-cache and long-context processing costs — central to the deployment goal. Not started; requires reliable base models first.	UNVERIFIED
MoE / sparse experts F7	More capacity per active FLOP via conditional expert routing (Switch, Mixtral, DeepSeek-V3); expert parallelism maps onto distributed heterogeneous nodes but all-to-all routing traffic is bandwidth-heavy.	Tension with the decentralized-inference goal. Not started; blocked behind solid dense baselines.	UNVERIFIED
Adaptive computation F8	Spend compute conditionally (ACT, PonderNet, early exit, Mixture-of-Depths): can a router decide WHEN attention is needed, at token or segment level?	Open question we care about. Routing overhead must not exceed savings — measure quality per additional FLOP. Not started.	UNVERIFIED
Long-context mechanisms F9	Sliding window, BigBird/Longformer sparse patterns, ring attention (distributed). Local attention alone cannot do exact long-range recall beyond its window bound.	Cheap to verify locally; interacts with H7.	UNVERIFIED

Softmax Transformers

REPRODUCED

Content-based soft addressing over full history; exact token-level retrieval. Train O(T²·d); inference KV cache O(T·d·L) grows without bound.

Vaswani et al. 2017. Baseline implemented (lab/models/transformer.py); basics reproduced in exp000 — learns MQAR above chance with a sane loss curve.

Linear attention / fast-weight family

UNVERIFIED

phi(q)phi(k)^T outer-product state, O(T); gated/decayed variants keep the same fundamental fixed-state recall limit at small scale.

Katharopoulos 2020, RetNet/GLA/DeltaNet. 'Matches transformer LM quality' holds for perplexity but not recall-heavy evals. Planned GLA-style variant if SSM results warrant.

State-space models (S4/S5/Mamba/Mamba-2)

PARTIAL

Input-dependent selectivity (Δ, B, C functions of x); linear time, constant state. Mamba-2/SSD connects SSMs to linear attention via chunkwise parallel form.

Gu & Dao 2023/2024. Minimal faithful S6 implemented with custom verified autograd scan; exp001 is the first controlled comparison. Our pure-torch CPU scan runs ~10–15x slower per step than SDPA attention at T=23 — asymptotic wins do not transfer to naive implementations.

Recurrent LMs (RWKV, xLSTM)

UNVERIFIED

RNN formulations competitive on perplexity, sharing the same recall limits as other fixed-state recurrence.

Lower priority than SSMs — similar primitive class.

Hybrids (attention + recurrence interleaved)

PARTIAL

A small fraction of attention layers (10–25%) recovers most recall/in-context ability while keeping recurrent efficiency (H3/Hyena, Based, Jamba, Zamba, Samba, Griffin/RecurrentGemma, MiniMax-01, Nemotron-H).

The single most replicated practical result across independent groups. Hybrid stack implemented (lab/models/hybrid.py); exp001 tests it — conclusion pending.

Memory / retrieval augmentation

UNVERIFIED

External/compressed memory decouples context cost from state capacity (Memorizing Transformer, Infini-attention, Titans neural memory, TTT layers, RMT).

Could cut KV-cache and long-context processing costs — central to the deployment goal. Not started; requires reliable base models first.

MoE / sparse experts

UNVERIFIED

More capacity per active FLOP via conditional expert routing (Switch, Mixtral, DeepSeek-V3); expert parallelism maps onto distributed heterogeneous nodes but all-to-all routing traffic is bandwidth-heavy.

Tension with the decentralized-inference goal. Not started; blocked behind solid dense baselines.

Adaptive computation

UNVERIFIED

Spend compute conditionally (ACT, PonderNet, early exit, Mixture-of-Depths): can a router decide WHEN attention is needed, at token or segment level?

Open question we care about. Routing overhead must not exceed savings — measure quality per additional FLOP. Not started.

Long-context mechanisms

UNVERIFIED

Sliding window, BigBird/Longformer sparse patterns, ring attention (distributed). Local attention alone cannot do exact long-range recall beyond its window bound.

Cheap to verify locally; interacts with H7.

Statuses defined in LEDGER.md — UNVERIFIED literature-only · PARTIAL small-scale local evidence · REPRODUCED replicated · REFUTED_LOCAL our test disagreed.

CH.05 // FAILURE DATABASE

What we eliminated.

Every failed or inconclusive experiment is recorded. No idea gets retested without reading this file first.

F-0001FAILURE2026-08-22

Under-trained pilot (exp001 @ 800 steps)

Val acc reached only 0.206 and was still climbing steeply at cutoff — NOT converged. 800 steps is eliminated as a sufficient budget at d128/batch64 on MQAR-8; conclusions from this regime would be meaningless for either architecture. Everything remains plausible; recalibrate convergence before any comparison.

F-0002FAILURE2026-08-22

Tiny-vocab/short-seq MQAR is a degenerate basin — root cause: embedding init scale

All models pinned exactly at loss=ln(#pairs) — uniform over values present — because GPT-2-style std=0.02 embedding init leaves attention scores ~0: a symmetric saddle where content matching never forms. Earlier copy-task 'successes' were positional shortcuts. Re-initializing embeddings to N(0,1) lets the same 4-layer d128 transformer solve zoology-faithful MQAR to 100% eval accuracy by step ~1500-2000. Big-embedding init is now lab default (embed_init_std=1.0).

ENV-0001ENVIRONMENT2026-08-22

CUDA PyTorch unavailable on this machine

System torch is CPU-only; CUDA wheel installs failed twice (no working cu126/cu128/cu130 build; torch 2.7.1+cu126 aborted on disk full — ~2GB free vs ~7GB needed). All experiments stay CPU-bound on shared desktop cores; scale capped at ~2M params and throughput numbers are indicative only. Re-enable condition: >=7GB free disk, then rerun exp007 on GPU where it actually matters.

ENV-0002PROTOCOL2026-08-22

Benchmark validation protocol adopted

Rule added after F-0002: every benchmark generator must ship (a) a ground-truth oracle solver scoring 100%, AND (b) a reference-model sanity check showing the task is learnable within a modest budget BEFORE any architecture comparison runs on it.

CH.06 // INSTRUMENT LIMITS

The envelope.

GPU: RTX 2060 SUPER 8GB
CPU: i7-10700K 8C/16T
RAM: 32 GB
DISK: ~2 GB free
STACK: Python 3.11.9, torch 2.13.0+cpu, numpy

Consequences: models stay at or under ~2M parameters, tasks are synthetic by necessity, and CPU throughput numbers are indicative only.

Intelligence, measured per compute.

One loop, falsifiable.

Hypothesis

Prediction

Implementation

Controlled experiment

Measurement

Analysis

Conclusion

Next hypothesis

On the bench.

Infrastructure smoke test

MQAR — attention vs selective SSM vs hybrid

Ranked by information gain.

Attention is the recall specialist at matched budget

One attention layer mostly recovers recall in an SSM stack

Recurrence matches attention where recall is absent

Copying exposes recurrent state bottlenecks

Recall gap grows with number of KV pairs (state capacity law)

State size buys recall sublinearly

Locality decomposition: window attention fails global recall alone

Learned routing: pay attention only sometimes

Wall-clock reality of cache-free inference

Distributed probes (deferred)

Standing on the map.

What we eliminated.

Under-trained pilot (exp001 @ 800 steps)

Tiny-vocab/short-seq MQAR is a degenerate basin — root cause: embedding init scale

CUDA PyTorch unavailable on this machine

Benchmark validation protocol adopted

The envelope.