Fashion-MNIST — The Trap Is Broken
DRAM-Native Bit-Logic Benchmark — 2026-08-19
The 91% trap is BROKEN. Beam 20 --max + filter eval gt 20% reaches 92.96% on Fashion-MNIST. Every search strategy lands in the 88.7–92.96% range — the 91% trap is BROKEN. The ceiling is architectural (frozen random W0 + maj1 voting, no convolution, no trainable features), not a search or diversity problem. What speaks for us: speed — a 221-member sweep runs in ~1 minute, orders of magnitude faster than CNN training.
1. The 91% Trap
| Strategy | eval range | What we tried |
|---|---|---|
--beam 10 --max (plain) | 90.8–91.6% | H32..512, maj1-thresh 98..108 |
--tries N --tries-no-lock | 91.3–91.6% | full-pool search, best-of-N |
--tries 3 + bestN 1/2/3 | 91.4–91.5% | per-try candidate offset |
--tries-random-seed | 91.3–91.4% | splitmix64 seeds |
--optimal (2-opt exchange) | — | 1-swap-stable, zero gain |
--expansion-sort clarity | 86.8–89.2% | member clarity (WORSE) |
--greedy | 90.2–90.8% | single path |
| Bit-Voting (no W0, no maj1) | 88.7–89.0% | lower technology bound |
The honest assessment: we play in the amateur league on accuracy — Fashion-MNIST SOTA (CNN) is 94–96%, we sit at ~92.96%. The 2.1pp gap is architectural, NOT a search or diversity problem — every lever we pull moves the needle by tenths of a percent only.
What speaks FOR us: SPEED. The PRF trainer does a 221-member sweep in ~1 minute, orders of magnitude faster than CNN training. For a DRAM-native bit-logic chip, the question is not "can we reach 95%" but "how little hardware do we need for 92.96%".
2. Leaderboard (all runs, INT32 bit-exact)
Top results per corpus/mode — --beam 10 --max unless noted. The lr=
field carries the winning --beam-bestN (reproducibility). Differences <0.1pp
are within noise.
| H | m1t | eval | err | Members | Mode |
|---|---|---|---|---|---|
| 512 | 105 | 92.96% | 704 | 59 | --beam 20 --max --filter eval gt 20% |
| 512 | 105 | 92.96% | 704 | 59 | + --filter eval gt 83% |
| 512 | 105 | 92.96% | 704 | 59 | + --filter eval gt 84% |
| 512 | 105 | 92.80% | 720 | 32 | --beam 10 --max |
| 256 | 105 | 92.51% | 749 | 42 | --beam 10 --max |
| 196 | 105 | 92.47% | 753 | 40 | --beam 10 --max |
| 512 | 105 | 92.59% | 741 | 19 | --beam 20 --filter eval gt 30 --tries 5 --tries-no-lock |
| 196 | 107 | 91.45% | 855 | 59 | + --tries 5 --tries-no-lock |
| 196 | 105 | 91.44% | 856 | 36 | + --optimal |
| 256 | 105 | 91.43% | 857 | 55 | + --tries 5 --tries-random-seed |
| 196 | 105 | 91.41% | 859 | 56 | --max --tries 3 (bestN=1) |
| 512 | 105 | 91.38% | 862 | 17 | --beam 10 (strict) |
| 196 | 106 | 91.34% | 866 | 39 | --beam 10 --max |
| 196 | 107 | 91.34% | 866 | 57 | --max --min-gain 0 |
| 196 | 107 | 91.33% | 867 | 56 | --beam 10 --max |
| 196 | 108 | 91.30% | 870 | 57 | --beam 10 --max |
| 196 | 104 | 91.20% | 880 | 58 | --beam 10 --max |
| 128 | 105 | 91.18% | 882 | 43 | --beam 10 --max |
| 196 | 98 | 90.91% | 909 | 35 | --beam 10 --max (true half) |
| 64 | 105 | 90.82% | 918 | 66 | --beam 10 --max |
| 32 | 105 | 89.70% | 1030 | 27 | --beam 10 --max |
| BV | — | 88.94% | 1106 | 17 | --beam 10 --max (no W0) |
Ceiling: ~92.96% (H=512). The H dimension is fully measured and flat (+0.14pp from H196→H512). The 91% trap is BROKEN — beam 20 --max + filter eval gt 20% → 92.96%.
3. The Speed Argument
The PRF trainer (mlp-bin32-otto-trn-seq-prof.c, N members in parallel,
1 thread each) makes sweeping blazing fast:
| Task | Time |
|---|---|
| Full member sweep (221 members: 17 xforms × 13 encodings) | ~1 min |
| Merge beam search (1885–2093 archives) | 3–60 s |
| Bit-Voting sweep (lower-bound research) | ~1 min |
Compare: a CNN epoch on Fashion-MNIST takes minutes to hours on a GPU. The DRAM-native bit-logic approach trades accuracy (~92.96% vs 94–96%) for hardware simplicity + speed that no CNN can match on-chip. For a DRAM-native chip, the question is not "can we reach 95%" but "how little hardware do we need for 92.96%".
4. maj1-thresh Calibration
--maj1-thresh is a sharp calibration point. The auto value is 107 for
n=196 — but the merge attractor peaks at 105. Lower thresh → fewer members
at equal/better eval: 105 gives 36 members vs 56 at 107.
| m1t | beam10 | beam10 --max | members | err |
|---|---|---|---|---|
| 98 (=n/2) | 90.73% | 90.91% | 35 | 909 |
| 104 | 90.99% | 91.20% | 58 | 880 |
| 105 | 91.04% | 91.44% | 36 | 856 |
| 106 | 91.11% | 91.34% | 39 | 866 |
| 107 (auto) | 90.86% | 91.33% | 56 | 867 |
| 108 | 90.84% | 91.30% | 57 | 870 |
5. Ensemble Search Findings
| Finding | Evidence |
|---|---|
| the 91% trap is BROKEN | --beam 20 --max --filter eval gt 20% reaches 92.96%; the basin is 88.7–92.96%, the ceiling is architectural (frozen random W0 + maj1), not search-limited |
--tries is pool partitioning | TRY 2..10 lock found members; --tries 10 == --tries 1 |
--tries-no-lock enables full-pool search | each try gets a fresh pool → different seeds, same-space attractors |
--beam-bestN N breaks 1st-best fixation | per-level selection starts at the N-th best candidate; bestN=3 single run: 91.16 vs 90.94 (bestN=1) |
--min-gain 0 is a real lever | 867→866 (57 members) on Otto, 1106→1105 (24) on BV |
--optimal (2-opt) is a dead end | 1-swap-stable attractor — no improving single swap |
--member-start = monotone pool extension | beam starts from a saved attractor (floor) — new members can only ADD, never redirect below it |
| clarity (class-score std-dev) | --expansion-sort clarity + --greedy-clarity; DIVERSITY beats pure clarity |
| beam is NOT pool-monotone | adding 312 shuffle members WORSENED 6/10 corpora — MORE members ≠ better |
6. Reproducible Reference Run
The best result to date (scores-H512-E10-OT8-M1-105-INT32,
--beam 20 --max --filter eval gt 20%) reaches 92.96% with 59 members
in the merge search. The --member-out file feeds the IFC delivery path
(export → model.otto → import). The v8 model is self-describing: the
import reads maj1-thresh=105 from the model itself (bugfix 2026-08-09 — before, the
import fell back to CLI defaults and lost ~50pp). The IFC-verified H512 model currently
in the log is the 55-member export at 92.95%:
| Step | Config | eval | err | Members |
|---|---|---|---|---|
| Merge search (best) | scores-H512-E10-OT8-M1-105-INT32 --beam 20 --max --filter eval gt 20% | 92.96% | 704 | 59 |
| Retrain (IFC export) | --member-file member-H512-E10-OT8-M1-105-INT32.out --export-default | 92.95% | 705 | 55 |
| IFC inference | --import export-H512-E10-OT8-M1-105-INT32 --evalN 10000 (no CLI flags) | 92.95% | 705 | 55 |
| Merge search (m1t=105, H196) | scores-H196-E10-M1-105-INT32 --beam 5 --filter @ | 91.10% | 890 | 16 |
| Retrain (m1t=105) | --member-file fashion-105.member --export model/ | 91.10% | 909 | 16 |
| IFC inference (m1t=105) | --import model/ --evalN 10000 (no CLI flags) | 91.10% | 890 | 16 |
| alt. | scores-H196-E10-M1-104-INT32 --beam 10 --filter not shuffle | 91.09% | 891 | 14 |
| alt. | scores-H196-E10-M1-106-INT32 --beam 10 | 91.11% | 889 | 21 |
IFC-verified: 55 members, 92.95% — zero train-merge gap. The retrained members (92.95%) and the IFC inference (92.95%) are identical: the v8 model carries the exact maj1-thresh=105 calibration and the trainer runs MODE_INT32 (bit-exact, chip-faithful). The best merge search (92.96%/59) is one member set beyond the currently exported model — the 92.95%/55 export is the verified chip delivery.
7. Technology Boundary
Bit-Voting (-DKI_BITVOTING, identity W0, direct pixel-bit → class) is the
lower technology bound for the DRAM chip: no W0, no majX. Its sweep is blazing
fast (221 members in ~1 min with the PRF trainer).
| Run | members | eval | err |
|---|---|---|---|
--beam 10 --max | 17 | 88.94% | 1106 |
--beam 10 --max --min-gain 0 | 24 | 88.95% | 1105 |