Bit-Voting Baseline

1. Motivation — Why a Baseline?

Every claim about Otto Score's accuracy raises a natural question: "Is the performance due to the W0 + majority architecture, or would a simpler linear model achieve the same result?"

The Bit-Voting baseline answers this definitively. It is a linear perceptron operating at the bit level — no W0, no majority tree, no hidden layer, no nonlinearity. Each of the 6272 input bits (after encoding) has 10 trainable weights (one per class), exactly like each of the 256 h0-neurons in Otto Score.

2. Architecture — Identity W0

Bit-Voting removes the W0 matrix entirely. Each input bit flows directly to the voting layer:

Both models use the exact same training algorithm: ki_batch_correct() from lib/ki-train.h with Bayesian gap-scaling. Both use the same voting mechanism: accumulated int64 scores → argmax. The only difference is the presence (Otto) or absence (Bit-Voting) of the W0 random projection and majority layer.

2.1 Training Loop

For each sample in the batch:

1. Forward: for each set bit b, accumulate score[k] += target[b][k]
2. Compare predicted label vs true label
3. If misclassified: target[b][true] += step, target[b][predicted] -= step
4. Step scales with OT_F: step = lr × OT_F

This is identical to Otto's training — the only difference is that Otto forwards through gb_buf (768 h0-bits per sample after W0 + majority) while Bit-Voting forwards through all 24,576 input bits directly.

2.2 Key Equivalence — Same Bit Mass in Target

Same memory footprint, same training algorithm. Otto's 768 compressed gb-bits beat Bit-Voting's 24,576 raw bits at equal target capacity — proving feature quality matters more than feature quantity.

2.2 Multi-Member Ensemble

Like Otto, Bit-Voting supports multiple members via --encoding and --xform. Each (encoding × xform) combination creates one independent member. Members are trained serially, votes are accumulated:

# 1 member (baseline)
./mnist-mlp-bin32-otto-trn-bitvoting.exe --epochsN 10 --encoding latest --xform id

# 6 members (2 encodings × 3 xforms)
./mnist-mlp-bin32-otto-trn-bitvoting.exe --epochsN 10 --encoding exp,log \
  --xform id,dflip1,rot90

3. Information Equivalence

Both models receive the same input bits and train the same number of weight bits. The only difference is how input bits are grouped before voting.

3.1 The Identity W0 Thought Experiment

Imagine replacing Otto's random W0 with an identity matrix: every input bit b maps to exactly one h0-neuron b, with no mixing. The majority layer then receives only 1 bit per neuron — a majority of 1 bit is the identity function. The rest of Otto's architecture (target weights, voting, training) is unchanged.

This identity-W0 Otto Score is exactly Bit-Voting. In the historical single-member measurement (unequal width: Otto H=256 vs identity) accuracy drops from 96.1% to 91.4%. At the current equal-H point the two are on par (§5.6) — the drop is a width effect, not a linearity effect. Historical figures:

• W0 randomness is not noise — it is the feature extraction mechanism.
• The majority gate alone does not create nonlinearity — it needs multiple inputs to vote on.
• The overcomplete random projection (each input bit mapped to ~8 neurons) creates the nonlinear combinations that the majority gate exploits.

4. Results

4.1 MNIST — 60K train / 10K test, 10 epochs, single member

Historical single-member comparison (unequal width: Otto H=256 vs identity H=196). At the equal-H point the two are on par — see §5.6. Kept for reference:

4.2 CIFAR-10 — OUTDATED (historical, kept for reference)

This section is outdated. Bit-Voting on CIFAR has since been superseded; the current, honest baseline is the equal-H result on MNIST/Fashion (see §5.6). Kept below for historical reference only.

Why CIFAR is not the current story: the large CIFAR gap (55.2 vs 64.4) is a width-ratio artifact — CIFAR compares H=768 (W0) against identity at a different effective width, not an equal-H point. On MNIST/Fashion at the same H the two are on par (§5.6), which is the honest baseline. CIFAR BV was not carried forward.

4.3 Fashion-MNIST — current equal-H result (2026-08-19)

At equal H Bit-Voting is on par with W0 on Fashion (91.94 vs 92.47, +0.53pp); W0's edge appears only when it scales to H=512. (The old ~80% vs ~88% gap was an unequal-width comparison.)

4.4 Encoding Sensitivity

Historical (unequal-width, single-member MNIST). The qualitative point still holds — Bit-Voting depends more on encoding quality than Otto Score, because it has no feature extraction to compensate for poor input structure:

Insight: Otto Score is robust to encoding (±0.5pp variance). Bit-Voting swings ±2pp. The W0 random projection homogenizes input structure — without it, the voter is at the mercy of the input encoding.

4.5 Ensemble Effect

Historical (unequal-width, MNIST). At equal H the ensemble diversity point is weaker — see §5.6. Kept for reference: multi-member ensembles help Otto Score significantly but help Bit-Voting only marginally:

Why the smaller ensemble gain? Otto members have independent random W0 matrices — different random seeds create completely different projections, producing uncorrelated errors. Bit-Voting members differ only in encoding and data transform (xform), which provides limited diversity. The W0 randomness is the actual source of ensemble diversity, not the data augmentation.

5. Speed Analysis — The Bit-Density Surprise

5.1 The raw-operation accounting (forward path)

Otto's forward path is the dense W0 × I0 computation. Bit-Voting scans only the set bits (sparse):

That is ~40× more raw operations in the forward path — but the forward is precomputed once (gb cache), and the shared training loop dominates.

5.2 The bit-density surprise (measured 2026-08-10)

Both models have the same gb dimension (H=I=196 on Fashion-MNIST), but the values differ in density:

The while (_b) loop in KT_SCORE/KT_CORRECT iterates over every set bit. Otto's W0+majority (maj1.h, half=107 at n=196) compresses the 32-bit containers to ~2.76 set bits — a 5.2× reduction of the training-loop work. Bit-Voting skips the compression and pays the full input density in the training loop.

5.3 Measured phases (3 members, serial, --prof)

5.4 IFC inference: maj1(W0×I) is dominant and cannot be cached

In IFC (--import) the picture flips: Bit-Voting is 14.7× faster. Measured with the same 3-member ensemble (Fashion-MNIST, H=196, 16 threads):

5.5 The honest accuracy balance — same member count (2026-08-10)

The W0 accuracy advantage depends on the member count. The headline "more accuracy" must be compared at the SAME number of members. Fashion-MNIST, H=196, both beam-selected on their own corpora (merge = TRN = IFC, zero gap):

The W0 gap GROWS with the member count: +1.34pp at 3:3, +2.01pp at 6:6. Otto's advantage is not a fixed per-member property — it comes from W0 diversity: each member projects through an independent random W0, producing complementary votes that only pay off with more members. The earlier "+2.5pp" (6-member Otto 90.49% vs 3-member BV 88.01%) was inflated by the member-count mismatch; the 6:6 comparison is the honest version of the same number.

Still the smallest gap of all datasets: MNIST +4.3pp (ensemble +2.4), CIFAR +9.2pp, Fashion +1.34pp (3:3) / +2.01pp (6:6). (Historical, 2026-08-10 — unequal-width. The newer equal-H result in §5.6 reduces this to ±0.5pp at H196.)

Why: the IFC evaluation path (ki_evaluate_member with use_gb=0) has no gb_buf cache. For every sample it calls scores_otto, which recomputes maj1(W0×I) per h per sample:

maj1(W0×I) is the dominant cost factor in IFC — and it is used exactly once per member, which allows NO caching. In training the same computation runs once into gb_buf and is reused over all 10 epochs (amortized → the ~1.0× result of §5.2); in IFC every sample pays it fresh. Bit-Voting skips it entirely, so the sparse bit-vote (~14 bits/container) is the whole cost.

5.6 Equal-H parity: Bit-Voting ≈ W0 at the SAME H (2026-08-19)

New result: at the same hidden size H, Bit-Voting is on par with W0 — the gap collapses to a fraction of a percent. The W0 advantage is not a per-member property: it only becomes visible once W0 is allowed to scale to more H. Measured on the OT8 corpus, beam 10 --max (INT32, bit-exact):

Why BV cannot follow: Bit-Voting sets W0 = identity, so its hidden size is frozen at the input width (H = I = 196 for these encodings). There is no free H knob — the sparse bit-vote width is fixed by the input. Otto's random projection projects to an arbitrary H, so it can scale H196 → H256 → H512, and that extra capacity is where the W0 advantage appears (Fashion 92.47 → 92.51 → 92.80, up to 92.96 with beam 20 --max). At the single shared point (H196) the two are essentially tied.

Interpretation: the "W0 nonlinearity" is best described as W0 = a free width expansion. Its value is not at a fixed dimension but in the ability to grow H beyond the input. The equal-H comparison is the honest baseline: it shows the nonlinear projection buys ~0.5pp at fixed capacity, and the real win comes from scaling width.

Comparison speed (Vergleichsgeschwindigkeit)

Wall-clock comparison at the shared H196 point (3-member ensemble, Fashion-MNIST, 16 threads). The merge-search runtime reflects member count; the IFC inference is the true forward-cost measure for the DRAM chip:

At equal H, training and merge search run in ~equal time. The only decisive speed difference is IFC inference — Bit-Voting's identity skips the uncacheable maj1(W0×I) and is 14.7× faster. So the product trade-off at equal H is: +0.5pp accuracy (W0) vs 14.7× faster inference (BV) — and the accuracy win only grows once W0 scales H past the input width, at the cost of a denser, slower forward path.

6. W0 as Generalizer

The W0 random projection does more than create nonlinearity — it generalizes.

6.1 Distributed Representation

Each input bit maps to ~8 random h0-neurons. No single pixel can dominate a decision. This forces the network to combine information from multiple pixels, creating robust, distributed features that generalize better than any single-pixel detector.

6.2 Encoding Invariance

Otto Score achieves ~96% on MNIST regardless of encoding choice (raw, sig8, exp8, log8 — all within ±0.5pp). Bit-Voting varies ±2pp depending on encoding quality. The random projection homogenizes input structure — bad encodings get mixed into useful features anyway.

6.3 Overfitting Resistance

Historical (CIFAR-10, 48 members — CIFAR BV is superseded; see §4.2). Kept for reference: Bit-Voting reaches 55.2% eval but trains to 78.2% (train-eval gap: 23pp). Otto with the same 48 members reaches 64.4% eval and trains to 96.7% (gap: 32pp — larger absolute gap because it CAN overfit more, but the absolute eval is 9.2pp higher because the features are better). At the current equal-H points (MNIST/Fashion H196) both train to high accuracy with small gaps, consistent with the parity finding.

6.4 Why This Matters for DRAM

A chip that implements W0 as a fixed random addressing pattern (not stored weights) gets the generalizer for free:

• No weight storage for W0 (just addressing)
• No training for W0 (frozen random)
• Nonlinearity + generalization at zero chip cost
• The same addressing hardware works for any dataset

7. Interpretation — The Npp Gap

7.1 The Gap is a Width Effect, Not a Linearity Effect

At the same H, Bit-Voting and W0 are on par — the "gap" is essentially zero. The gap appears only when W0 is allowed to scale H beyond the input width:

• MNIST (H196): 98.17% vs 98.75% — +0.58pp
• Fashion (H196): 91.94% vs 92.47% — +0.53pp
• Fashion (H512): 92.80% — W0 scales width, BV cannot follow

Historical (outdated, CIFAR-BV superseded): the old per-dataset gaps (MNIST +4.7pp at unequal width, Fashion +8pp, CIFAR +9.2pp) came from comparing W0 at a much larger H than the identity baseline, not from an equal-H point.

The W0 matrix acts as a random nonlinear expansion:

1. Each input bit is XOR-ed with ~31 random container values → nonlinear mixing
2. Majority over each container → threshold activation (popcount > 16)
3. Result: 256 bits that are nonlinearly transformed from the original 6272 bits

Without training W0, this expansion is random. It buys ~0.5pp at fixed width, but its real value is the free width knob: projecting to an arbitrary H lets the voter separate classes that the fixed input width cannot. This matches the Reservoir Computing / Random Projection literature: a random expansion into a higher-dimensional space improves linear separability, even without training the expansion weights.

7.2 The Identity W0 Proof

The thought experiment from Section 3.1 can be tested empirically: if we set W0 to the identity matrix, Otto Score becomes Bit-Voting. At equal width the accuracy difference is small (±0.5pp, §5.6) — proving W0's advantage is not the nonlinearity per se but the width expansion it enables. When W0 is allowed to scale (H196→H512) the gap grows; when held at the identity width it disappears. (Historical: the old 4.7pp MNIST / 20pp CIFAR / 8pp Fashion figures were unequal-width comparisons, superseded.)

7.3 Why Not Train W0?

Learning W0 (backprop through the random projection) adds ~10pp on CIFAR-10 (58% → 68%, see XNOR-Net literature), but requires:

• Floating point storage for real-valued weights
• Backpropagation through XOR operations
• GPU or specialized floating-point hardware

The DRAM-native constraint forbids these. A frozen random W0 is a compromise: ~0.5pp at equal width and the ability to scale width to higher H, at zero additional hardware cost — the random projection is hard-wired in the DRAM addressing logic. (Historical: the old "5pp below the learnable ceiling on MNIST" figure was an unequal-width comparison.)

7.4 Relation to Reservoir Computing

The Otto Score architecture is a discrete reservoir computer:

• Reservoir: W0 (frozen random matrix, never trained)
• Nonlinearity: Majority over XOR-mixed bits (popcount threshold)
• Readout: Linear voter (target weights, trained)

The bit-voting baseline shows the reservoir's contribution is a width expansion: at equal H the readout matches (98.17 vs 98.75 MNIST, 91.94 vs 92.47 Fashion); the reservoir's real value appears when it scales H beyond the input width. (Historical: the old "91.4% instead of 96.1%, 5pp gap" was an unequal-width comparison.)

8. Key Findings

9. Conclusion

The Bit-Voting baseline shows that W0 is not the core nonlinear mechanism per se — it is a free width expansion. At the same hidden size H, Bit-Voting matches Otto Score (MNIST 98.17 vs 98.75, Fashion 91.94 vs 92.47, ±0.5pp). W0's real contribution is the ability to scale H beyond the input width (Fashion 92.47→92.80→92.96), which Bit-Voting's identity cannot do — its width is frozen at the input.

This result has practical implications for DRAM-native chip design:

• At equal width, W0 buys only ~0.5pp. The honest accuracy trade-off is +0.5pp (W0) vs 14.7× faster IFC inference (Bit-Voting) — a product decision, not a technical necessity.
• W0's value is width scaling. If the application needs accuracy beyond the input-width baseline, the random projection to a larger H is the lever; if it values inference speed, Bit-Voting wins.
• Bit-Voting is the DRAM-native complexity floor. Any chip that implements Otto Score can be compared against Bit-Voting at zero architectural cost — and at equal H the two are essentially tied.

Bit-Voting is now a standard experiment in the Otto Score workflow, alongside Hebbian and AdamW baselines. With a single make bitvote target, any dataset can be checked for linear separability before committing to the full Otto Score architecture.

Appendix A: Quick Start

# Build
make bitvote

# MNIST baseline
./mnist-mlp-bin32-otto-trn-bitvoting.exe --epochsN 10 --encoding latest --xform id \
  --evalN 10000 --debug-epoch

# CIFAR-10 baseline
./cifar-mlp-bin32-otto-trn-bitvoting.exe --epochsN 10 --encoding latest --xform id \
  --evalN 10000

# Fashion-MNIST baseline
./fashion-mlp-bin32-otto-trn-bitvoting.exe --epochsN 10 --encoding latest --xform id \
  --evalN 10000

# Multi-encoding ensemble
./mnist-mlp-bin32-otto-trn-bitvoting.exe --epochsN 10 --encoding exp,log \
  --xform performance --debug-member --member-threshold 90

# Compare with Otto Score
./mnist-mlp-bin32-otto-trn-xnor.exe --hiddenN 196 --ensembleN 1 \
  --epochsN 10 --xform id --lr 0.05 --debug-epoch

The --debug-epoch flag shows per-epoch progress with training and eval accuracy. The --debug-member flag shows per-member accuracy with encoding and xform metadata.

Source: Bit-Voting is a compile-time variant of the shared trainer otto-score-ifc/mnist/mlp-bin32-otto-trn-seq-prof.c, built with -DKI_BITVOTING (identity W0, I=H width). The bitvote Makefile target produces the *-bitvoting.exe binary from it. See plans/plan-2026-08-02-ki-bitvoting-flag.md.