go-compressions

Two adjacent concerns live here — lossless compression codecs and content-addressable hash functions — because they sit at the same layer of the stack: allocation-aware, zero-copy-friendly primitives that turn one byte stream into another, smaller or fingerprinted. Different math, same place in the stack, same engineering bar. Codecs are fuzz-tested against the upstream reference implementations (pierrec/lz4, Apple's liblzfse); hashes are cross-checked against the spec test vectors.

Pure Go, no cgo, multi-arch. The numbers are measured and honest — wins and the cases where we trade speed for ratio. lz4's hot match-extension loop and blake3's mixing ride go-asmgen SIMD — the matchlen common-prefix kernel now runs vectorized on all six 64-bit Go SIMD targets (amd64, arm64, riscv64, loong64, ppc64le VSX and s390x vector facility, big-endian). ppc64le is now natively measured on real POWER10 silicon (GCC Compile Farm, Go 1.26.4): lz4 encode runs 1.8× scalar (1174 vs 644 MB/s) and beats pierrec/lz4 (1174 vs 1012 MB/s), with blake3 mix4 at 4.5×. riscv64 is now natively measured too on a SpacemiT X60 (RVV 1.0, GCC Compile Farm, Go 1.26.4): lz4 encode runs 1.45× scalar (110 vs 76 MB/s) and beats pierrec/lz4 (110 vs 83 MB/s, ~1.32×), with blake3 mix4 at 2.9×. The s390x path stays qemu-validated (byte-identical to scalar), with native throughput pending an IBM Z runner. Beyond the six SIMD targets, every library also builds and passes its tests bit-exact on a seventh architecture, ppc64 (big-endian) on real POWER9 silicon — six SIMD targets, validated on seven architectures. 100% statement coverage is a CI gate on every module.

Repositories

lz4 libcodec

LZ4 block codec (CompressBlock / DecompressBlock)

SIMD on all 6 arches (match extension via matchlen)

wire-compatible with pierrec/lz4 — beats it on compression ratio (text ~4.6% smaller) and now decodes at parity with its hand-written arm64-asm decoder (~1.0-1.4x on real files, beating it on ooffice/sao); honestly slower on encode speed. Match extension delegates to matchlen's SIMD common-prefix kernel.

lz4c clicodec

LZ4 CLI (compress / decompress)

pure-Go CLI (rides the lz4 lib's SIMD)

single static-binary CLI over the lz4 library — stdin/stdout by default, -d decompress, -o output, -v summary (ratio + timing). Pure Go, no C toolchain. The reference CLI for the lz4 lib.

lzfse libcodec

Apple LZFSE / LZVN codec (Compress / Decompress)

pure-Go (scalar)

byte-compatible with the reference liblzfse: LZFSE = LZVN + FSE entropy stage. Auto-picks LZVN for ≤4 KiB inputs, LZFSE V1/V2 for larger; round-trips against Apple's C reference. Ratio within ~1-3% of Apple; decode now within ~1.6-2.1x of Apple's -O3 C reference (was ~2-3x).

lzfsec clicodec

LZFSE CLI (compress / decompress)

pure-Go (scalar)

single static-binary CLI over lzfse — stdin/stdout by default, optional -v summary (ratio + timing). Pure Go, no C toolchain.

deflate libcodec

DEFLATE / RFC 1951 codec (Deflate / Inflate, Writer / Reader)

match extension via matchlen (SIMD common-prefix kernel)

bidirectionally wire-compatible with the standard library's compress/flate — flate decodes every stream our Writer produces and we decode every stream flate.NewWriter produces (differential-fuzzed both ways). All three block types (stored / fixed / dynamic Huffman), 32 KiB sliding window, typed errors on every malformed-input path. The LZ77 parse delegates match extension to matchlen's SIMD common-prefix kernel. Correctness-first: competitive encode on text, honestly slower decode than stdlib flate.

blake3 libhash

BLAKE3 cryptographic hash (Sum256 / Sum512 / XOF)

SIMD mix4 on all 6 arches (VSX / vector facility)

pure-Go, cgo-free BLAKE3, verified against the official test vectors; the mix4 core runs go-asmgen SIMD on all 6 of Go's 64-bit targets. Streaming hash.Hash-style API plus arbitrary-length extendable output (XOF). All three BLAKE3 modes ship: unkeyed hashing, keyed hashing (MAC, via NewKeyed) and key derivation (NewDeriveKey / KDF).

b3sum clihash

BLAKE3 checksum CLI (print / verify)

SIMD on all 6 arches (via blake3)

pure-Go, cgo-free b3sum — output compatible with the reference Rust b3sum (<hex> <name>). Hashes files/stdin, --check verification, --length / --no-names. Single static binary.

Every module is pure Go with CGO_ENABLED=0, allocation-aware on its hot paths, and held at 100% statement coverage as a CI gate. Codecs are fuzz-tested for byte-exact round-trips against their upstream reference implementations; hashes are verified against the official spec vectors. SIMD acceleration (via go-asmgen) is layered behind build tags where it helps — matchlen's common-prefix kernel covers all six of Go's 64-bit SIMD targets. BSD-3-Clause throughout.