MNIST — DRAM-Native Binary Classifier

1. Experiment Data

• Code: github.com/aotto1968/forward-prop (otto-score-ifc/ tree + mnist-1/ local trainers)
• Dataset: MNIST (60,000 train + 10,000 test, 28×28 grayscale, IDX format) — fetch_mnist.sh (downloads train-images-idx3-ubyte.gz, train-labels-idx1-ubyte.gz, t10k-images-idx3-ubyte.gz, t10k-labels-idx1-ubyte.gz)
• Split: 70,000 total samples, shuffled random split → 60,000 train / 10,000 eval (--evalN 10000 is the standard, always used). --evalN 100 and --evalN 1000 are special experiments to test the data-limited ceiling (more training data per eval sample).
• Note on old 50K/10K split: Earlier results used a fixed 50,000 train / 10,000 eval split without shuffle — this was a bug. All current results use the correct 60K/10K shuffled split. The difference is minor (~0.1pp).
• Hardware: 8-core CPU, no GPU. All timing on identical hardware for fair comparison.
• Project: forward-prop.nhi1.de

2. Leaderboard (2026-07-30)

Note: On MNIST, --encoding latest = --encoding exp8 (single grayscale channel). The 98.4% record uses --encoding sweep (all 13 encodings) × --xform sweep@avg4,avg3,avg2,spiral,colswap pipe chains, with --min-gain 0.03 selecting 9 of 26 members.

Core insight: Pipe xforms (running-average filters combined with pixel reordering) create fundamentally different input statistics. The avg4@avg4 double-running-average produces a heavily smoothed image where the MAJ3 layer captures overall digit shape without overfitting to stroke texture — +0.5pp over the previous record.

3. Architecture

Input:  784 px → 1 encoding block × 196 uint32 = 196 containers
Layer 0: W0 (frozen, random MAJ3): NC × H       [bit-string, 32 bits per neuron]
Layer 1: Bayes log-Score: H → K=10 (target + offset)
Grid:    ENSEMBLE[EN] × XFORM[XF] × COLOR[C] × HN[H]
Voting:  Σ_members score[k]  →  argmax

Each W0 row picks 3 random containers from the 196-container encoding block. MAJ3 outputs 1 if ≥2 of the 3 container bits are set → a frozen binary hash. The correction loop iterates over epochs:

1. Score: score[k] = Σ_active target[k][bit] + offset[k] — Bayes log-odds per class
2. Correct: If argmax ≠ true, target[true] += step, target[pred] −= step
3. Iterate N epochs with cosine step decay (warmup + cos-time)

3.1 The Member Grid

The total member count is the product of four orthogonal dimensions:

Members = ENSEMBLE[EN] × XFORM[XF] × COLOR[C] × HN[H]

• EN (ensembleN): independent copies of W0 with different random seeds — stochastic diversity
• XF (xform): geometric transforms of the input image — structured diversity
• C (color/encoding): different encoding schemes for the same pixels
• HN (splitHN): horizontal input slicing (each member sees different containers)

Each member has its own W0, target matrix, and offset — trained independently. Scores are summed across all members (product of experts), not hard majority vote.

3.2 Exponential Encoding — The Breakthrough

Raw 8-bit pixel values produce near-uniform container bit patterns (~4 bits set per byte regardless of brightness). The exponential thermometer encoding (exp8) converts each pixel into a brightness-proportional bit string where bright pixels set more bits. This lets the MAJ3 layer perceive continuous intensity variation rather than binary on/off. The effect was a +7pp jump from 96% to 97%+.

4. Key Findings

4.1 Bit-Mass Theory — H×EN Determines Accuracy

The single most important discovery: total neuron count H×EN determines accuracy, not H or EN alone. Keeping total H × EN fixed (constant bit-mass) produces identical accuracy regardless of distribution:

All configurations reach ~97.3%. H=128×EN=8 is the sweet spot — slightly best result with faster forward-passes. Doubling bit-mass gives logarithmic returns:

4.2 Exponential Encoding Broke the 96% Wall

Before exp8, the method wall stood at 96%. The key realization: raw byte containers produce uniform popcount (~4 bits/byte) regardless of pixel brightness — the MAJ3 layer could not distinguish a faint digit from a saturated one. Thermometer encoding converts each pixel to a brightness-proportional bit mask:

raw pixel 200 (78% brightness) → exp8 → 0b11111110 (7 bits set)
raw pixel 30  (12% brightness) → exp8 → 0b00000011 (2 bits set)

The MAJ3 layer now has meaningful variance to work with. Combined with cosine step decay, this pushed MNIST accuracy from 96% to 97%+ in a single change.

4.3 Small H + Ensemble Beats Large Single Model

Many small independent MAJ3 banks outperform one large bank at equal bit-mass. A 16× smaller W0 (98 KB vs 1.57 MB) reaches comparable accuracy:

Architecture insight: W0 noise IS the feature. Diverse MAJ3 projections from many small banks give better generalization than one large projection. Each small bank fits on a separate DRAM row — fully parallel at chip level.

4.4 All Thermometric Encodings Are Equivalent on MNIST

MNIST pixels are binary (0 or 255 after binarization). All thermometer encodings (exp8, log8, sig8, lin8) produce identical output: 0→0x00, 255→0xFF. Multiple encodings bring nothing but additional W0 diversity — exp8,exp8,exp8 reaches the same accuracy as exp8,log8,sig8. On MNIST, only bit-mass matters, not encoding choice.

4.5 Target Init Does Not Matter

Seven initialization modes were tested (count, dampen, inverse, laplace, random, uniform, prior). All converge to the same final accuracy after sufficient correction epochs — only the initial 1-pass accuracy differs (10-86%). The correction loop always finds the same attractor.

4.6 Deep MAJ3 Loses Information

Each additional MAJ3 layer compresses N uint32 → 32 bits. A second MAJ3 on already-compressed H0 loses ~4% accuracy. Three layers collapse to 59.7%. Ensemble (parallel) beats depth (sequential) every time.

5. Geometric Transform Ensemble — Structured Beats Stochastic

The most impactful architectural insight: a geometric transform creates a genuinely new view of the data, which is fundamentally more valuable than a different random seed on the same view.

The --xform flag adds a new dimension to the member grid. Each active transform produces an independent pixel buffer with its own W0, Target, and Offset — exactly like adding an --ensembleN copy, but the member operates on a different image rather than a different random projection of the same image.

5.1 Available Transforms

--xform all activates all 8 transforms → 8× member multiplier.

5.2 Architecture — Each Xform Has Its Own W0

This is not test-time augmentation. Each xform creates an independent structural member:

1. ki_xform_raw() transforms raw pixel data before channel computation.
2. Each xform allocates its own container buffer via load_input_cached_xform().
3. Each member mem->input_buf points to its xform's buffer — different MAJ3 triples for the same W0 row.
4. Training corrects each xform's targets independently based on that xform's view.
5. Evaluation sums scores across all xform members.

Critical distinction: An EN=5 member sees the same pixels through 5 different random filters. An XF=5 member sees 5 different pixel arrangements through the same filter. The MAJ3 triples fall on completely different spatial positions — a horizontal flip maps pixel (7, y) → (21, y), so the MAJ3 triple that was (1,2,3) now operates on (28,27,26). This is genuinely independent information, not just another random projection of the same pattern.

5.3 Xform vs EnsembleN: The Efficiency Gap

At equal member count (5 members each), xforms outperform ensembleN by +0.4pp. But the real story is the time efficiency — achieving the same accuracy with fewer members:

Why the 7× speedup?

• EN=7 = 7× W0 computation on the same input. Seven identical forward-passes through different hash functions on the same pixels. Each member computes h0 from exactly the same X_all buffer. The errors are highly correlated — if member 0 struggles with a particular 3, members 1-6 struggle too (just through different noise).
• XF=8 = 1× W0 computation on 8 different inputs. Each forward-pass requires a different buffer (mirrored/rotated). But the decorrelation is massive: a horizontal-flipped 3 presents the difficult curves at completely different pixel positions. Errors are nearly independent — voting across 8 views is much more effective than voting across 7 noise realizations of the same view.
• Net effect: 8 xform members (5.2s) provide more independent signal than 21 ensemble members (36.1s) — 4× fewer members, 7× less time.

Direct comparison at equal member count:

Both use H=64, Ep=7, 5 members total. Xforms win by +0.4pp at equivalent time. Structured beats stochastic.

5.4 H-Scaling With Xforms

Sweet spot: H=128, EN=1, 8 Xforms → 97.7% in 5.2s. The three pure rotations (rot90/180/270) add +0.3pp at H=128 over the 5-flip set. H=512+ reaches 97.9% with 8 xforms — the architecture ceiling.

5.5 Key Architectural Insight: Xforms Replace EnsembleN

The member grid now has three diversity sources, listed by effectiveness:

1. Encoding — different pixel-to-container transformations (exp, log, sig, lin, spatial)
2. Xform — different geometric views of the image (flip, rotate, transpose)
3. EnsembleN — different random seeds on the same data (least effective)

Encoding and Xform together provide enough independent views that additional EN contributes minimal accuracy at linear time cost. The new design principle: maximize structural diversity first (encoding + xforms), use ensembleN only as a last-resort multiplier when all other dimensions are exhausted.

5.6 Pipe Xform Ensemble — 98.4% Record (July 2026)

The most accurate Otto Score classifier ever trained on MNIST: 98.4% (9837/10000) with 9 members, H=196, 10 epochs, 14.2 seconds.

The breakthrough is the xform pipe chain (@ syntax): applying a running-average filter after a pixel reordering transform. A pipe like avg4@spiral first applies the spiral pixel permutation, then the 4-pixel running average on each row. The result is a fundamentally different image structure — not just pixels moved around, but pixel values blended with neighbors in a transformed coordinate system.

Build recipe

# Step 1a: Sweep — one command per xform pipe (9 total)
./mnist-mlp-bin32-otto-trn-xnor.exe --hiddenN 196 --epochsN 10 \
  --encoding sweep --xform sweep --sweep --export-merge-scores scores-196-10
./mnist-mlp-bin32-otto-trn-xnor.exe --hiddenN 196 --epochsN 10 \
  --encoding sweep --xform sweep@spiral --sweep --export-merge-scores scores-196-10
./mnist-mlp-bin32-otto-trn-xnor.exe --hiddenN 196 --epochsN 10 \
  --encoding sweep --xform sweep@colswap-1-4 --sweep --export-merge-scores scores-196-10
./mnist-mlp-bin32-otto-trn-xnor.exe --hiddenN 196 --epochsN 10 \
  --encoding sweep --xform sweep@colswap-2-4 --sweep --export-merge-scores scores-196-10
./mnist-mlp-bin32-otto-trn-xnor.exe --hiddenN 196 --epochsN 10 \
  --encoding sweep --xform sweep@colswap-3-4 --sweep --export-merge-scores scores-196-10
./mnist-mlp-bin32-otto-trn-xnor.exe --hiddenN 196 --epochsN 10 \
  --encoding sweep --xform sweep@svg2 --sweep --export-merge-scores scores-196-10
./mnist-mlp-bin32-otto-trn-xnor.exe --hiddenN 196 --epochsN 10 \
  --encoding sweep --xform sweep@avg2 --sweep --export-merge-scores scores-196-10
./mnist-mlp-bin32-otto-trn-xnor.exe --hiddenN 196 --epochsN 10 \
  --encoding sweep --xform sweep@avg3 --sweep --export-merge-scores scores-196-10
./mnist-mlp-bin32-otto-trn-xnor.exe --hiddenN 196 --epochsN 10 \
  --encoding sweep --xform sweep@avg4 --sweep --export-merge-scores scores-196-10
# → 1053 archive files (13 enc × 9 xforms × 9 pipe variants)

# Step 2: Beam search selects best subset
./mnist-merge-ensemble.exe scores-196-10/ --beam 10 \
  --member-out member.out --min-gain 0.03

# Step 3: Train selected 9 members → 98.4%
./mnist-mlp-bin32-otto-trn-xnor.exe --hiddenN 196 --epochsN 10 \
  --member-file member.out --debug-member

Selected members (sorted by addition order)

Key insight: The pipe avg4@avg4 (double running average) appears three times (lin8, sig8, sqrt8 encodings). Double-avg4 heavily smooths the image — eliminating fine stroke texture. The MAJ3 layer cannot overfit to noise it cannot see. Blurring is a feature, not a bug.

--min-gain 0.03: Only members with ≥0.03% cumulative gain are included. 9 of 26 potential members are selected — 65% fewer, 0.0pp loss. See Status Report for details.

6. Time Scaling Over H and Epochs

The following data shows how runtime evolves with --hiddenN and --epochsN for the champion config (--encoding latest --ensembleN 1 --xform all, i.e. 8 xforms on exp8).

Key observations:

• Per-epoch time scales linearly with H. H=256 costs ~2.5× per epoch vs H=64 (0.85s → 2.0s per epoch). This is expected — double the neurons means double the h0 computation, double the scoring, double the correction.
• Eval plateaus at epoch 7. Additional epochs only reduce training error (err), not generalization. H=64 stays at 97.4%, H=128/256 reach their maximum by epoch 7.
• Training error drops exponentially. Each epoch roughly halves the remaining errors. H=64: 576→143 (−4×). H=128: 269→9 (−30×, because higher capacity absorbs more training signal). H=256 reaches err=2 by epoch 10.
• H=64 is capacity-limited. Even 10 epochs yield only 97.4% with err=143. 128 neurons cannot separate all 10 classes cleanly.

Practical conclusion: The standard recommendation is H=128, Ep=7, 8 xforms — 97.7% in 5.2 seconds. For the maximum 97.9%, use H=512 + 8 xforms at 19.5s. Anything beyond H=512 enters diminishing returns.

7. Why 99% is Data-Limited, Not Method-Limited

The single most important finding of the MNIST research: the 98% wall at evalN=10000 is a data ratio problem, not an algorithm ceiling.

At the standard 6:1 split (60K train, 10K eval, shuffled from 70K total), each eval sample receives ~6 correcting votes during training — insufficient for a fully confident decision. At 599:1, each eval sample gets ~60 votes because 599× more training data informs its target weights. The limit is the amount of training data per evaluation sample, not the architecture.

Absolute architecture ceiling: 97.9% at standard 6:1 split (err ≤ 10, training 100%). The remaining errors are samples that the frozen MAJ3 projection cannot represent — no amount of target retraining can fix what the hash never captured. This is the fundamental limit of a frozen random projection layer.

Solution for higher accuracy: Larger datasets (500K+ samples) or concentrated evaluation (fewer eval samples = more training data per class). For a DRAM chip with millions of rows, data volume is never the bottleneck — MNIST and CIFAR are simply too small to saturate the available representational capacity.

7.1 Progression of Ceilings

8. Random Pixel Shuffle — Relation Beats Position

--xform shuffle1..shuffle10 performs a random pairwise pixel permutation (Fisher-Yates, fixed seeds 1–10, compiled into the code). Unlike geometric xforms (hflip, rot90), shuffle destroys spatial neighborhood structure completely — but preserves all information (bijection, no loss).

Result: only 0.4pp below identity. H=196 with 6 shuffles reaches 97.6% vs 97.6% with id. The W0 random projection partially recovers spatial structure because each h0-neuron mixes random pixels. Otto Score compares pixel relations, not pixel positions.

8.1 Complete Test Series (10 epochs)

8.2 Optimal Config per Accuracy Target

For every eval% target, the fastest combination of H, ensembleN, and shuffle count (10 epochs, MNIST standard split):

Sweet spots:

• 97.0% in 2.2s — H128 + 3 shuffles (best cost/benefit)
• 97.5% in 3.5s — H256 + 3 shuffles
• 97.9% in 9.9s — H512 + 6 shuffles, 6 epochs (max accuracy record 🏆)

8.3 Shuffle vs Ensemble: Diversity Source

Different shuffles (xform diversity) outperform same shuffle (W0 diversity) by +1.5–1.7pp. Same shuffle + different W0 = pure ensemble effect. Different shuffles + same W0 = different pixel neighborhoods. The latter is more effective because targets learn true pixel relations rather than position-dependent patterns.

8.4 Generalization: Diversity Needs Structure

Three sources of member diversity, sorted by effectiveness:

1. Xform (different shuffles) — different pixel neighborhoods, maximal error decorrelation
2. Geometric xforms (hflip, rot90) — structured variation of pixel arrangement
3. W0 ensemble (--ensembleN) — different random projections on the same arrangement, weakest diversity

9. Practical Recommendations

The sweep + merge-ensemble workflow is the new standard for maximum accuracy: train all encoding×xform combinations, export per-member scores, let the beam search select the optimal subset. --min-gain 0.03 keeps only members with measurable contribution — 65% fewer members, 0.0pp loss.

Legacy configs use default count target init and cos-time step schedule. --xform all = 8 transforms (id, hflip, vflip, dflip1, dflip2, rot90, rot180, rot270). The transform ensemble is the key architectural insight: structured geometric diversity replaces stochastic ensemble seeds, achieving the same accuracy with 7× fewer compute resources.