Engineering Log: Binary Delta Patcher

Context

To secure leadership buy-in, the immediate priority was demonstrating a significant reduction in update payloads to prove the ROI of binary diffing. The objective was to find an algorithm that yields optimized patch sizes out of the box.

Decision

Implemented the qbsdiff crate (Bsdiff algorithm) as the core diffing engine.

Rationale

• Pros: Successfully proved the business case for CDN cost reduction. Benchmarks showed a 120 MB base updating to a 128 MB target required only a 1.95 MB patch.
• Cons (The RAM Flaw): Bsdiff uses suffix sorting, requiring both the old and new files to be held in memory simultaneously. Patch creation requires RAM equal to or greater than the new file size.
• Next Steps: This technical debt is documented; for production scaling, the diffing engine will be hot-swapped for xdelta3 or HDiffPatch to balance patch size with a flat memory footprint.

Context

Standard directory diffing pipelines often generate large intermediate .tar archives before comparing them, introducing disk I/O overhead and consuming excess storage.

Decision

Architected a custom directory archive format (starting with magic bytes ADIR). The system walks directories to generate a manifest of specific DiffTask operations: Patch, Create, or Delete.

Rationale

• Pros: Eliminated intermediate file storage overhead. During benchmarking, the engine traversed 188 files across 21 folders, calculated diffs, and streamed the payload natively through a global zstd encoder into a unified 1.95 MB archive.
• Cons: Introduces a proprietary archive format requiring a specific CLI tool to unpack, limiting interoperability with standard unzipping utilities.

Context

Processing large binary files chunk-by-chunk through standard file I/O creates bottlenecks. Minimizing system call overhead is essential to maintaining speed in the patching pipeline.

Decision

Utilized the memmap2 library to map target binaries directly into virtual memory.

Rationale

• Pros: Accelerates I/O reads by delegating paging to the OS kernel, efficiently feeding byte arrays to the diffing algorithm and integrity hashers.
• Cons: Requires handling unsafe blocks in Rust to initialize memory maps. The system's virtual memory limits strictly dictate the maximum file size processed during the prototype phase.

Context

Applying a patch to an incorrect base file version corrupts the file. Ensuring state integrity is critical, but hashing large datasets can delay patching.

Decision

Integrated the blake3 algorithm for high-performance hashing. The 32-byte hash output is embedded into the patch header.

Rationale

• Pros: Before applying a patch, the tool verifies the local file hash matches the embedded hash. blake3 is highly parallelizable and fast, ensuring safety checks do not bottleneck I/O throughput.
• Cons: Fails strictly on hash mismatch, meaning users cannot force-apply patches to modified base files.