▣ fractal_zip — Hutter Prize submission
Submitted by Scott Brooks · September 2026 · ← fractal_zip project
A submission to the Hutter Prize (Human Knowledge Compression Contest: losslessly compress the first 109 bytes of English Wikipedia, enwik9). The compressor is a cmix-lex-transformer derivative: the online LSTM of the fx2-cmix lineage is replaced by a 6M-parameter transformer trained offline, quantized to packed-int4 dense weights, and run entirely on one CPU core. The PPMD model's 14 GiB heap lives on fvmem, a compressed in-RAM store with real page release, to honor the 10 GiB memory line without a disk-backed temp file.
Measured result
| Quantity | Bytes |
|---|---|
| archive9 (self-extracting, fixed decompressor stub) | 97,617,144 |
| compressor C | 3,836,817 |
| S (archive9 + C) | 101,453,961 |
| Current 1% improvement bar (vs deepmix) | 107,203,662 |
| Margin under the bar | 5,749,701 |
| enwik8, included with the mail (compressed_enwik8, decode-verified against its lineage binary cmix_enwik8, shipped alongside) | 12,303,845 |
archive9 sha256 5bf6021376b23ae3067110b994fbbfdbc560404d4143a41ced5ce2621f64d4d4 cmix sha256 a9f2894d54f7eed9f6d7373248a830c8b97082e8e366b588f1151028719e3969 enwik9 sha256 159b85351e5f76e60cbe32e04c677847a9ecba3adc79addab6f4c6c7aa3744bc (restored)
Time legality (wall_hours × Geekbench5 T < 70,000, clock-normalized projections from quiet solo prefix retimes): encode 60,105 / decode 63,359 on the Lenovo i7‑1165G7 named in the rules; 55,177 / 58,164 on the AMD Ryzen 7 — all under 70,000 with 9–14% margin, with decode figures taken from deliberately conservative contaminated measurements. Full method and numbers in the packet document.
Method in one paragraph
Articles are split and fed to an autoregressive byte-level transformer (offline-trained, weights appended to the binaries — 2.9 MB each after int4 quantization of weights and activations, with AVX2 kernels for the low-precision GEMMs). The train-set 205 vocabulary is baked in so out-of-vocabulary bytes never force a mask. Everything else follows the fx2-cmix / cmix-lex record lineage, unmodified. Details, training setup, and honest caveats (this is specialized to enwik9, not a general-purpose compressor): writeup.md.
Say NO to the RAMpocalypse
The part of this submission that did not exist anywhere before: the PPMD heap wants ~14 GiB; the judging machines have 10. fvmem keeps the heap in RAM as compressed 64 KiB blocks with a capped uncompressed working pool, evicting under pressure. Getting this actually legal took three attempts, all disclosed in the packet:
| Build | Defect | Fate under an enforced memory cage |
|---|---|---|
| Sep 14 | bump-only arena leaked every dirty re-eviction | kernel-OOM-killed at 17.6% in 16 min |
| Sep 15 | arena fixed, but eviction only mprotect()ed pages — released zero resident memory | kernel-OOM-killed at 55% in 92 min |
| Sep 16 (mailed) | in-place arena reclamation + madvise(MADV_DONTNEED) page release | completed bit-identical, peak 6.35 GB, ~423 GB cumulatively returned to the kernel |
systemd-run -p MemoryMax=… -p MemorySwapMax=0) — no credit
taken for a big-RAM development box. The failing builds above are the
negative controls proving the test can detect the bug. enwik9-scale RSS is
an argued extrapolation from the 10M-prefix measurement plus the store's
structural bounds; that distinction is stated plainly in the mail.fvmem outlived the submission — it is already used elsewhere in the
fractal_zip project. Source: tools/fractal_compute/fvmem.h.
Why submit
I probably have a very compelling story about the circumstances I've gone through in my life, but to me that is not what this competition is about. For me, it's about incentivizing discovery and exploration that lead to real improvements to humanity and people's lives down the line. I've already used fvmem in my own projects, and I wouldn't have been pushed to come up with things like it without the Hutter competition.
The conditions of creating these extreme compression tools have changed since the beginning of this competition. While it may once have made sense to award someone, or a group, as solely responsible, that really isn't the case today — especially at the time of writing, with so many beautiful submissions and innovations! We're all pushing each other to be better, and that's something to be encouraged.
Documents
- writeup.md — transformer method writeup (LSTM → transformer, training, quantization)
- PACKET.md — packet identity, hashes, RSS incident disclosures, time-legality tables
- Source zip, containment evidence, and verification logs ship with the mail; lineage: fx2-cmix / cmix-lex (Kaido Orav, Byron Knoll, Ibrahim Marcouch) and the unpublished Ivanov transformer path, per the writeup.
- Mailed artifacts (archive9, cmix, source.zip, compressed_enwik8, SHA256SUMS): mail/
- Mirrors: GitHub release (all files) · FFS public shelf (folder “hutter”: archive9 + checksums)
fractal_zip · Hutter Prize submission page · September 2026.