Leaderboard — Ensemble Subset Selection
Public challenge leaderboard — 2026-08-31
Status: OPENING. The challenge is live, the score corpora and the reference search tool are published. First submissions are accepted via GitHub — we validate them independently on the .ens corpora and add them to this table. See the full challenge specification →
Reference Baselines (INT32 bit-exact)
All baselines run on Fashion-MNIST (10,000 test samples) via
merge-ensemble.exe over the published .ens score archives. The
standard member cap is |S| ≤ 12; the greedy run is uncapped.
| # | Method | eval | members | config / notes |
|---|---|---|---|---|
| 1 | Greedy forward selection | 91.84% | 17 | full pool 24,192, --greedy |
| 2 | Beam search (30/50) + filter | 92.32% | 12 | H392-OT8, --beam 30 --max 12 --eff --filter m1th eq 105 --filter eval gt 66% |
| 3 | Beam 20 + filter eval gt 20% | 92.96% | 12 | H196-OT8, --max --filter eval gt 20% |
| 4 | Row 2 + --exclude 6 (Shirt omitted) | 94.24–94.83% | 12–18 | ⚠️ proviso: 9-class accuracy, class 6 removed from train+eval |
⚠️ Row 4 is a 9-class result (proviso: omitting class "Shirt"). It quantifies how much the single problem class costs (−2.4 to −3.0pp) and is not comparable to the 10-class rows 1–3. See the Fashion-MNIST paper.
Community Submissions
| # | Method | eval | members | submitter | date |
|---|---|---|---|---|---|
| — open for submissions — | |||||
How to Submit
- Read the challenge: ensemble-selection-challenge.html — problem statement, corpus access, submission protocol (§6).
- Produce a subset: your algorithm must output a subset of member indices (or a reproducible ranking) over one of the published .ens corpora.
- Report: accuracy, #members, wall-time, hillclimb/test split used, determinism, and the exact command line / seed.
- Send: open an issue or PR on github.com/aotto1968/forward-prop. We re-run it independently on the corpora and publish the verified numbers here.
Evaluation Rules
- Metric: majority-vote accuracy on the test set, INT32 bit-exact, reproducible from the .ens archives alone.
- Standard cap:
|S| ≤ 12(also report the best result at|S| ≤ 100). - To qualify for the 10-class leaderboard, the result must use all 10 classes; 9-class (class-excluded) results are listed separately and marked with the proviso.
- We may reject submissions that overfit the reported set (no held-out verification protocol) — state your split explicitly.
Forward-Prop Research — DRAM-native bit-logic classification. Leaderboard
opened 2026-08-31. Baselines INT32 bit-exact, reproducible via
run-grep --date-start 2026-08-30 | run-sort eval.