Hey r/cybersecurity,

I've been building a zero-disk-footprint pipeline (nist-live-cvt1-feed, open source) that carves MD5/SHA-1/SHA-256 hashes out of the NIST NSRL (National Software Reference Library) RDSv3 "modern" dataset — the reference set used for known-good file whitelisting in DFIR/malware triage workflows — entirely on GitHub Actions free-tier runners (6h job limit, 14GB disk).

I want to focus less on "here's my project" and more on the constraint-driven engineering decisions, including two things that broke in production and forced a redesign.

The actual constraint

The published archive (RDS_*_modern.zip) is ~124GB compressed. Once inflated, the real SQLite database is ~425GB — I verified this directly two ways that had to agree: the ZIP64 extra field's declared uncompressed size, and independently, page_size (4096) × page_count (111,560,036) read straight from the SQLite file header. Both gave the identical 456,949,907,456 bytes.

Extracting that to disk is impossible under a 14GB limit. Loading it into RAM is impossible under a 7GB limit. So this meant dropping the sqlite3 engine entirely and parsing the raw B-tree binary format ourselves, streamed, never touching disk.

Two things that broke in production (the actually interesting part)

1. Compression doesn't work on hash output, and we proved it before shipping the fix. Our SHA-256 output cleared GitHub's 2GB single-asset limit (~72M unique digests). Instinct says "just gzip it." We measured first, on 3,000,000 real digests: gzip -9 and zip -9 both landed at 100.0% of the original size — literally zero benefit. Cryptographic hashes are uniformly-distributed bytes by construction, so there's no redundancy for DEFLATE to exploit. The real fix was splitting into numbered parts with a header-preserving format, so cat clean_sha256_part*.bin > clean_sha256_full.bin reconstructs a byte-exact original — not compression at all.

2. Unconditional 3-way parallelism hurt small runners. Our checkpoint-compaction step (sort + dedup, one thread per hash algorithm) originally spawned exactly 3 threads, always. Standard GitHub-hosted runners often only expose 2-4 vCPUs and constrained disk I/O — always launching 3 concurrent read/write streams can cause CPU/IO contention instead of a speedup on a small runner. Fixed with a bounded semaphore sized to min(3, detected_core_count). Verified empirically: with 4 detected cores the checkpoint takes identical time to 3 cores (confirming no wasted 4th thread for only 3 fixed tasks), while 1-2 cores correctly fall back toward sequential execution instead of thrashing.

Architecture

Parsing - SQLite B-tree page carving — never starts the SQLite engine; parses table leaf pages (0x0D) and cell pointer arrays directly, plus custom SQLite varint decoding. - Schema-positional matching — instead of scanning for "anything hex-shaped," a record only counts if its trailing column signature exactly matches text(8)/text(32)/text(40)/text(64) (crc32/md5/sha1/sha256). This structurally rejects unrelated hash-shaped columns and cache-style filenames elsewhere in the row, by construction rather than by heuristic. - Zero-copy streaming — HTTP → zlib inflate → parse, continuously, with the full file never written to disk.

Concurrency - Ping-pong double buffering to overlap network fill with CPU-bound parsing. - Thread-local sharding — each worker thread owns its own 256 buckets (partitioned by first hash byte), eliminating lock contention during the hot path. - Bounded semaphore concurrency for checkpoint compaction, as above — dynamically sized to the runner's actual detected core count, not a hardcoded constant.

Memory/disk bounding - Scatter-gather: carve (map) and merge (reduce) are fully separate phases/jobs. - External sort-merge + k-way merge for the final cross-checkpoint deduplication — deliberately not an in-memory hash set, since total unique count can run into the tens of millions and RAM is capped at 7GB. - Periodic checkpoint compaction (std::sort + std::unique, then flush) bounds local disk usage regardless of total database size — closer to LSM-tree compaction than a naive single final merge.

Output format - A compact custom binary format (magic/version/algo/timestamp/count header + sorted, fixed-width digest array) designed for mmap() + binary search lookups from consuming tools, with no parsing overhead.

Networking - HTTP Range-based reconnect: if the TCP connection drops mid-download, the pipeline reopens with a byte-exact Range header and keeps feeding the same in-memory zlib decompressor — this works because the process itself never dies, only the socket does. Worth noting: I checked whether this specific file's deflate stream has any Z_SYNC_FLUSH restart points (which would allow resuming decompression after a full process restart, not just a dropped connection) — scanned 30+MB of real compressed data and found zero. So this trick is strictly a same-process connection-drop mitigation, not a general resumable-download mechanism; a killed process genuinely has to restart the decompression from byte zero. - Pre-flight throughput probe: before committing to a multi-hour carve, a dedicated job measures this runner's actual sustained throughput to the host and skips the expensive job entirely if the measured speed projects it won't finish in time — cheap failure over expensive failure.

Repo's linked in my profile if anyone wants to dig into the actual carving/varint code or the schema-positional matching logic specifically — genuinely curious if anyone here has hit similar constraints processing large reference datasets (NSRL, VirusTotal exports, etc.) in constrained CI environments, and how you approached it.