MNIST — DRAM-Native Binary Classifier

1. Experiment Data

• Code: github.com/aotto1968/forward-prop (otto-score-ifc/ tree + mnist-1/ local trainers)
• Dataset: MNIST (60,000 train + 10,000 test, 28×28 grayscale, IDX format) — forward-prop.nhi1.de/data/mnist/ (train-images-idx3-ubyte.gz, train-labels-idx1-ubyte.gz, t10k-images-idx3-ubyte.gz, t10k-labels-idx1-ubyte.gz) — fetch_mnist.sh
• Split: 70,000 total samples, shuffled random split → 60,000 train / 10,000 eval (--evalN 10000 is the standard, always used). --evalN 100 and --evalN 1000 are special experiments to test the data-limited ceiling (more training data per eval sample).
• Note on old 50K/10K split: Earlier results used a fixed 50,000 train / 10,000 eval split without shuffle — this was a bug. All current results use the correct 60K/10K shuffled split. The difference is minor (~0.1pp).
• Hardware: 8-core CPU, no GPU. All timing on identical hardware for fair comparison.
• Project: forward-prop.nhi1.de

2. Leaderboard (2026-07-16)

Note: On MNIST, --encoding latest = --encoding exp8 (single grayscale channel). All thermometric encodings produce identical output on binary pixels.

Core insight: exp8,log8,sig8 multi-encoding ensemble outperforms single exp8 by ~0.1-0.2pp by providing additional W0 diversity. The cos-time step mode dominates — gently reducing step over time prevents the oscillation of pow-based schedules.

3. Architecture

Input:  784 px → 1 encoding block × 196 uint32 = 196 containers
Layer 0: W0 (frozen, random MAJ3): NC × H       [bit-string, 32 bits per neuron]
Layer 1: Bayes log-Score: H → K=10 (target + offset)
Grid:    ENSEMBLE[EN] × XFORM[XF] × COLOR[C] × HN[H]
Voting:  Σ_members score[k]  →  argmax

Each W0 row picks 3 random containers from the 196-container encoding block. MAJ3 outputs 1 if ≥2 of the 3 container bits are set → a frozen binary hash. The correction loop iterates over epochs:

1. Score: score[k] = Σ_active target[k][bit] + offset[k] — Bayes log-odds per class
2. Correct: If argmax ≠ true, target[true] += step, target[pred] −= step
3. Iterate N epochs with cosine step decay (warmup + cos-time)

3.1 The Member Grid

The total member count is the product of four orthogonal dimensions:

Members = ENSEMBLE[EN] × XFORM[XF] × COLOR[C] × HN[H]

• EN (ensembleN): independent copies of W0 with different random seeds — stochastic diversity
• XF (xform): geometric transforms of the input image — structured diversity
• C (color/encoding): different encoding schemes for the same pixels
• HN (splitHN): horizontal input slicing (each member sees different containers)

Each member has its own W0, target matrix, and offset — trained independently. Scores are summed across all members (product of experts), not hard majority vote.

3.2 Exponential Encoding — The Breakthrough

Raw 8-bit pixel values produce near-uniform container bit patterns (~4 bits set per byte regardless of brightness). The exponential thermometer encoding (exp8) converts each pixel into a brightness-proportional bit string where bright pixels set more bits. This lets the MAJ3 layer perceive continuous intensity variation rather than binary on/off. The effect was a +7pp jump from 96% to 97%+.

4. Key Findings

4.1 Bit-Mass Theory — H×EN Determines Accuracy

The single most important discovery: total neuron count H×EN determines accuracy, not H or EN alone. Keeping total H × EN fixed (constant bit-mass) produces identical accuracy regardless of distribution:

All configurations reach ~97.3%. H=128×EN=8 is the sweet spot — slightly best result with faster forward-passes. Doubling bit-mass gives logarithmic returns:

4.2 Exponential Encoding Broke the 96% Wall

Before exp8, the method wall stood at 96%. The key realization: raw byte containers produce uniform popcount (~4 bits/byte) regardless of pixel brightness — the MAJ3 layer could not distinguish a faint digit from a saturated one. Thermometer encoding converts each pixel to a brightness-proportional bit mask:

raw pixel 200 (78% brightness) → exp8 → 0b11111110 (7 bits set)
raw pixel 30  (12% brightness) → exp8 → 0b00000011 (2 bits set)

The MAJ3 layer now has meaningful variance to work with. Combined with cosine step decay, this pushed MNIST accuracy from 96% to 97%+ in a single change.

4.3 Small H + Ensemble Beats Large Single Model

Many small independent MAJ3 banks outperform one large bank at equal bit-mass. A 16× smaller W0 (98 KB vs 1.57 MB) reaches comparable accuracy:

Architecture insight: W0 noise IS the feature. Diverse MAJ3 projections from many small banks give better generalization than one large projection. Each small bank fits on a separate DRAM row — fully parallel at chip level.

4.4 All Thermometric Encodings Are Equivalent on MNIST

MNIST pixels are binary (0 or 255 after binarization). All thermometer encodings (exp8, log8, sig8, lin8) produce identical output: 0→0x00, 255→0xFF. Multiple encodings bring nothing but additional W0 diversity — exp8,exp8,exp8 reaches the same accuracy as exp8,log8,sig8. On MNIST, only bit-mass matters, not encoding choice.

4.5 Target Init Does Not Matter

Seven initialization modes were tested (count, dampen, inverse, laplace, random, uniform, prior). All converge to the same final accuracy after sufficient correction epochs — only the initial 1-pass accuracy differs (10-86%). The correction loop always finds the same attractor.

4.6 Deep MAJ3 Loses Information

Each additional MAJ3 layer compresses N uint32 → 32 bits. A second MAJ3 on already-compressed H0 loses ~4% accuracy. Three layers collapse to 59.7%. Ensemble (parallel) beats depth (sequential) every time.

5. Geometric Transform Ensemble — Structured Beats Stochastic

The most impactful architectural insight: a geometric transform creates a genuinely new view of the data, which is fundamentally more valuable than a different random seed on the same view.

The --xform flag adds a new dimension to the member grid. Each active transform produces an independent pixel buffer with its own W0, Target, and Offset — exactly like adding an --ensembleN copy, but the member operates on a different image rather than a different random projection of the same image.

5.1 Available Transforms

--xform all activates all 8 transforms → 8× member multiplier.

5.2 Architecture — Each Xform Has Its Own W0

This is not test-time augmentation. Each xform creates an independent structural member:

1. ki_xform_raw() transforms raw pixel data before channel computation.
2. Each xform allocates its own container buffer via load_input_cached_xform().
3. Each member mem->input_buf points to its xform's buffer — different MAJ3 triples for the same W0 row.
4. Training corrects each xform's targets independently based on that xform's view.
5. Evaluation sums scores across all xform members.

Critical distinction: An EN=5 member sees the same pixels through 5 different random filters. An XF=5 member sees 5 different pixel arrangements through the same filter. The MAJ3 triples fall on completely different spatial positions — a horizontal flip maps pixel (7, y) → (21, y), so the MAJ3 triple that was (1,2,3) now operates on (28,27,26). This is genuinely independent information, not just another random projection of the same pattern.

5.3 Xform vs EnsembleN: The Efficiency Gap

At equal member count (5 members each), xforms outperform ensembleN by +0.4pp. But the real story is the time efficiency — achieving the same accuracy with fewer members:

Why the 7× speedup?

• EN=7 = 7× W0 computation on the same input. Seven identical forward-passes through different hash functions on the same pixels. Each member computes h0 from exactly the same X_all buffer. The errors are highly correlated — if member 0 struggles with a particular 3, members 1-6 struggle too (just through different noise).
• XF=8 = 1× W0 computation on 8 different inputs. Each forward-pass requires a different buffer (mirrored/rotated). But the decorrelation is massive: a horizontal-flipped 3 presents the difficult curves at completely different pixel positions. Errors are nearly independent — voting across 8 views is much more effective than voting across 7 noise realizations of the same view.
• Net effect: 8 xform members (5.2s) provide more independent signal than 21 ensemble members (36.1s) — 4× fewer members, 7× less time.

Direct comparison at equal member count:

Both use H=64, Ep=7, 5 members total. Xforms win by +0.4pp at equivalent time. Structured beats stochastic.

5.4 H-Scaling With Xforms

Sweet spot: H=128, EN=1, 8 Xforms → 97.7% in 5.2s. The three pure rotations (rot90/180/270) add +0.3pp at H=128 over the 5-flip set. H=512+ reaches 97.9% with 8 xforms — the architecture ceiling.

5.5 Key Architectural Insight: Xforms Replace EnsembleN

The member grid now has three diversity sources, listed by effectiveness:

1. Encoding — different pixel-to-container transformations (exp, log, sig, lin, spatial)
2. Xform — different geometric views of the image (flip, rotate, transpose)
3. EnsembleN — different random seeds on the same data (least effective)

Encoding and Xform together provide enough independent views that additional EN contributes minimal accuracy at linear time cost. The new design principle: maximize structural diversity first (encoding + xforms), use ensembleN only as a last-resort multiplier when all other dimensions are exhausted.

6. Time Scaling Over H and Epochs

The following data shows how runtime evolves with --hiddenN and --epochsN for the champion config (--encoding latest --ensembleN 1 --xform all, i.e. 8 xforms on exp8).

Key observations:

• Per-epoch time scales linearly with H. H=256 costs ~2.5× per epoch vs H=64 (0.85s → 2.0s per epoch). This is expected — double the neurons means double the h0 computation, double the scoring, double the correction.
• Eval plateaus at epoch 7. Additional epochs only reduce training error (err), not generalization. H=64 stays at 97.4%, H=128/256 reach their maximum by epoch 7.
• Training error drops exponentially. Each epoch roughly halves the remaining errors. H=64: 576→143 (−4×). H=128: 269→9 (−30×, because higher capacity absorbs more training signal). H=256 reaches err=2 by epoch 10.
• H=64 is capacity-limited. Even 10 epochs yield only 97.4% with err=143. 128 neurons cannot separate all 10 classes cleanly.

Practical conclusion: The standard recommendation is H=128, Ep=7, 8 xforms — 97.7% in 5.2 seconds. For the maximum 97.9%, use H=512 + 8 xforms at 19.5s. Anything beyond H=512 enters diminishing returns.

7. Why 99% is Data-Limited, Not Method-Limited

The single most important finding of the MNIST research: the 98% wall at evalN=10000 is a data ratio problem, not an algorithm ceiling.

At the standard 6:1 split (60K train, 10K eval, shuffled from 70K total), each eval sample receives ~6 correcting votes during training — insufficient for a fully confident decision. At 599:1, each eval sample gets ~60 votes because 599× more training data informs its target weights. The limit is the amount of training data per evaluation sample, not the architecture.

Absolute architecture ceiling: 97.9% at standard 6:1 split (err ≤ 10, training 100%). The remaining errors are samples that the frozen MAJ3 projection cannot represent — no amount of target retraining can fix what the hash never captured. This is the fundamental limit of a frozen random projection layer.

Solution for higher accuracy: Larger datasets (500K+ samples) or concentrated evaluation (fewer eval samples = more training data per class). For a DRAM chip with millions of rows, data volume is never the bottleneck — MNIST and CIFAR are simply too small to saturate the available representational capacity.

7.1 Progression of Ceilings

8. Practical Recommendations

All configs use default count target init and cos-time step schedule. --xform all = 8 transforms (id, hflip, vflip, dflip1, dflip2, rot90, rot180, rot270). The transform ensemble is the key architectural insight: structured geometric diversity replaces stochastic ensemble seeds, achieving the same accuracy with 7× fewer compute resources.