Nitro NG is a highly parallel, hardware-aware pre-compression tool engineered natively in C++20 for Linux (POSIX) and Win32/64. Its sole, sacred purpose is to cut through opaque game files (such as Unreal Engine .pak containers or GOG setups), locate buried zlib/deflate compressed layers, and inflate them into their raw, expanded state with bit-exact mathematical precision.
================================================================================NITRO ENGINE - ARCHITECTURAL CHANGELOG: v1.05 HP → v1.07 HP → v1.09================================================================================Version chain:v1.05 HP → v1.07 HP → v1.09[GENERAL CHANGELOG]The v1.05 HP release introduced multithreading but suffered from un-throttledheap allocations under high core counts and dense stream distributions. Thisoccurred because asynchronous producer threads executed much faster than thesequential disk writer could flush, leading to unbounded queue growth andmassive per-thread buffer pre-allocations.Version 1.07 HP directly resolves these architectural bottlenecks and introducesmajor milestone features: lock-free atomic telemetry, EMA-based ETA smoothing,a multi-threaded parallel decoder, a Windows Zero-Map reader to eradicate RAMbloat, and full native deployment support for 32-bit Windows XP alongside Linuxand Windows x64 environments.Version 1.09 represents a complete architectural rewrite. We eliminated thememory-mapped I/O layer, the FNV-1a fingerprint cache, and the sharded zlibdictionary cache entirely. In their place, we introduced a streaming verificationmodel, segmented I/O with backpressure, and FLEVEL-based candidate guidance.The result is a tool that maintains strictly bounded memory usage regardless ofinput file size or stream density, while achieving faster encode/decode timesand smaller output files.================================================================================SECTION A - VERSION v1.05 HPMULTITHREADED PIPELINE FOUNDATION & ARCHITECTURAL LIMITATIONS================================================================================--------------------------------------------------------------------------------1. CORE MULTITHREADED FOUNDATION--------------------------------------------------------------------------------* Introduced worker-thread-based parallel processing.* Encoder input space was divided into thread chunks.* Each worker scanned its assigned region independently.* Worker progress heads were tracked with atomic counters.* Concurrent scanning for compressed payloads, standard zlib headers, and raw"stream"-style payloads with parallel verification.--------------------------------------------------------------------------------2. KNOWN v1.05 HP ARCHITECTURAL LIMITATIONS--------------------------------------------------------------------------------* Un-throttled heap allocation under high core counts.* Asynchronous producer threads outpaced the sequential disk writer.* Pending stream queue could grow without bound.* Large per-thread buffer pre-allocations caused massive memory spikes.* Memory footprint scaled dangerously with thread count and stream density.* Multi-gigabyte RAM spikes were possible on dense archives.================================================================================SECTION B - VERSION v1.07 HPMAJOR MILESTONES, MEMORY STABILIZATION & PLATFORM EXPANSION================================================================================--------------------------------------------------------------------------------1. LOCK-FREE ATOMIC TELEMETRY & EMA ETA ESTIMATION--------------------------------------------------------------------------------* Lock-Free Telemetry Counters- Real-time statistics are sampled from a dedicated progress thread usingrelaxed-order atomic operations (`std::memory_order_relaxed`).- Telemetry reads do not acquire mutexes and introduce zero contentioninto encoder/decoder worker threads.- Peak memory and speed are updated safely through relaxed CAS loops(`compare_exchange_weak`).* EMA-Based ETA Smoothing- Added an Exponential Moving Average (EMA) speed estimator for remainingtime calculation.- Effectively smooths short bursts and prevents violent ETA oscillationon dense stream distributions or fluctuating disk throughput.* Dedicated Ticker Thread (~10 Hz)- Progress sampling runs independently on a background ticker thread,keeping worker threads completely free from console formatting overhead.--------------------------------------------------------------------------------2. MULTI-THREADED PARALLEL DECODER--------------------------------------------------------------------------------* Task-Based Recompression Pool- Replaced the legacy sequential decoder with an active task-based worker pool.- Workers fetch tasks dynamically via `std::atomic<int>` fetch-add.- Dramatically accelerates restoration speed by 2x to 3x on multi-core hardware.- Main thread coordinates sequential output writes via condition variables.--------------------------------------------------------------------------------3. WINDOWS ZERO-MAP READER (RAM FOOTPRINT STABILIZATION)--------------------------------------------------------------------------------* Eradication of File-Mapping RAM Bloat on Windows- Eliminated `MapViewOfFile` from the decoder pipeline on Windows.- Replaced with a direct, efficient *Zero-Map Stream Reader* using Win32overlapped handles (`portable_pread`).- Prevents the Windows memory manager from aggressively caching massive`.pre` archives in physical RAM.- Keeps the peak memory footprint pinned to a clean ~40-50 MB, matching Linux.--------------------------------------------------------------------------------4. WINDOWS XP 32-BIT DEPLOYMENT TARGET--------------------------------------------------------------------------------* Native Cross-Compilation Support- Added cross-compilation infrastructure via MinGW-w64 (`i686-w64-mingw32-g++-posix`).- PE binary is explicitly marked for the Windows XP console subsystem(Major/Minor version 5.01).- Large-address-aware linking (`-Wl,--large-address-aware`) permits up to ~3 GBof user-mode address space on XP systems configured with the `/3GB` switch.- Hybrid encoder fallback (`portable_pread`) eliminates the 2GB file sizeceiling on 32-bit systems.--------------------------------------------------------------------------------5. ORIGINAL UX PRESERVATION & CURSOR POLISH--------------------------------------------------------------------------------* Single-Line Live Progress UX- Retains the clean one-line live progress format with carriage return updates.- Windows console cursor is hidden during execution to eliminate flicker.- Space-padding prevents stale artifacts or lingering digits when text length shortens.--------------------------------------------------------------------------------6. LOCK-FREE FNV-1a FINGERPRINT CACHE (~38% HIT RATE)--------------------------------------------------------------------------------* 64-bit FNV-1a hashing algorithm (0xcbf29ce484222325ULL, 0x100000001b3ULL)* Hashes the first bytes of encountered compressed layers* If pattern already solved across zlib levels {6, 9, 1}, subsequent threadsbypass evaluation entirely* Cache size: 65536 entries (NitroCacheEntry)--------------------------------------------------------------------------------7. SHARDED ZLIB DICTIONARY CACHE (ANTI-FALSE SHARING)--------------------------------------------------------------------------------* 2048 total slots (Windows) or 8192 slots (Linux)* Split into 16 independent shards with 64-byte padding blocks* Each slot maintains a z_stream state with deflateReset() for reuse* LRU eviction when all slots in a shard are occupied* Emergency fallback: allocates new z_stream if cache is exhausted--------------------------------------------------------------------------------8. PREC CONTAINER FORMAT v14 (outdated)--------------------------------------------------------------------------------* Sequential linear container with metadata at tail* GeneralPrecompHeader (25 bytes) + Expanded Payload + StreamMetadata Table* StreamMetadata: 26 bytes per entry (offset, dec_size, comp_size, level, windowBits)================================================================================VERIFICATION METRICS (v1.07 HP)================================================================================* Integrity: 100% verified, bit-for-bit restoration matches original input (MD5 confirmed).* RAM Footprint: Bounded and deterministic (~41 MB Linux, ~52 MB Windows Zero-Map).* Static Binary Size: ~1008 KB (Win64), ~1.4 MB (WinXP x86), ~1.7 MB (Linux Native).================================================================================SECTION C - VERSION v1.09COMPLETE ARCHITECTURAL REWRITE: STREAMING VERIFICATION & BOUNDED MEMORY================================================================================--------------------------------------------------------------------------------1. WHY THE REWRITE WAS NECESSARY--------------------------------------------------------------------------------The v1.07 HP encoder worked well for typical workloads, but it had fundamentalarchitectural limitations:* Memory-mapped I/O (mmap/MapViewOfFile) caused virtual address space exhaustionon large files and aggressive OS caching that inflated physical RAM usage.* The FNV-1a fingerprint cache (65536 entries) added complexity without adaptingto workload characteristics; 62% of streams still required full evaluation.* The sharded zlib dictionary cache (16 shards with padding blocks) consumedsignificant memory and required LRU eviction logic plus emergency fallback.* Per-thread buffer pre-allocations (16 MB dec_buf, 16 MB vtmp_pool, 64 KBinflate_pool per thread) wasted memory when actual stream sizes were smaller.* The pending_map/pending_streams queue could grow unbounded under high streamdensity, causing RAM spikes to 1-2 GB on dense archives.The solution was simple in concept but required rethinking the entire pipeline:don't store decompressed data at all. Instead, verify streams on-the-fly byinflating and immediately re-deflating, comparing byte-by-byte against theoriginal input. If the recompressed output matches the original compressedstream, the stream is valid. If not, abort immediately.--------------------------------------------------------------------------------2. ELIMINATED COMPONENTS (What We Removed)--------------------------------------------------------------------------------* Memory-Mapped I/O- REMOVED: mmap() on Linux, MapViewOfFile() on Windows- REPLACED WITH: Segmented pread() on Linux, ReadFile with OVERLAPPED on Windows- RATIONALE: mmap/MapViewOfFile cause virtual address space exhaustion andaggressive OS caching. Segmented I/O with explicit offsets gives us fullcontrol over memory usage and eliminates working set bloat.* FNV-1a Fingerprint Cache- REMOVED: NitroCacheEntry array (65536 entries)- REMOVED: generate_nitro_fingerprint() function- REMOVED: g_nitro_intel_cache global variable- REMOVED: g_cache_hits telemetry counter- RATIONALE: The fingerprint cache was a heuristic optimization that workedwell for typical workloads but added complexity and memory overhead. Withthe new FLEVEL-based candidate guidance (see below), we achieve betterhit rates without a separate cache layer.* Sharded Zlib Dictionary Cache- REMOVED: ZlibDictCache class with 16 shards- REMOVED: ZlibDictSlot structure with atomic in_use flags- REMOVED: g_dict_cache_ptr global unique_ptr- REMOVED: g_dict_reuse and g_dict_fallbacks telemetry counters- REMOVED: service_zlib_cache_trim() function- REMOVED: --cache and --mem command-line options- RATIONALE: The sharded cache was designed to minimize lock contention, butit added significant memory overhead (16 shards * padding blocks) andcomplexity (LRU eviction, emergency fallback). In v1.09, each worker threadmaintains its own Inflater/Deflater pair with deflateReset()/inflateReset(),which is simpler and uses less memory.* Per-Thread Buffer Pre-allocation- REMOVED: dec_buf.reserve(16 * 1024 * 1024) per worker- REMOVED: vtmp_pool (16 MB + 64 KB per thread)- REMOVED: inflate_pool (64 KB per thread * 2)- REPLACED WITH: ByteBuf with dynamic growth and release() method- RATIONALE: Pre-allocating large buffers per thread wastes memory when theactual stream sizes are much smaller. ByteBuf grows on demand and can bereleased when no longer needed.* BufferPool Class- REMOVED: BufferPool class with mutex-protected vector of buffers- RATIONALE: BufferPool was designed to reuse large buffers across threads,but it added mutex contention and complexity. In v1.09, each Slot maintainsits own ByteBuf that can be reused across segments.--------------------------------------------------------------------------------3. NEW ARCHITECTURAL COMPONENTS (What We Added)--------------------------------------------------------------------------------* Streaming Verification Model- Scanner threads NO LONGER store decompressed data in memory- Instead, they verify streams on-the-fly: 1. Inflate a chunk (64 KB) 2. Immediately re-deflate it 3. memcmp against the original compressed bytes 4. If match, stream is valid; if not, abort instantly- Zero intermediate storage, zero RAM bloat- First mismatch aborts verification, saving CPU cycles* Segmented I/O with Backpressure- Input file is divided into segments (1-8 MB each, aligned to 64 KB)- Segments are assigned to worker threads in strict sequential order- Sliding window: W = num_threads * 2 + 2- Workers cannot advance beyond writer_seg + W- This backpressure mechanism ensures at most W segments are in-flight- Memory usage is bounded by W * segment_size, regardless of file size* FLEVEL-Based Candidate Guidance- The zlib header's FLEVEL byte (0x01/0x5E/0x9C/0xDA) indicates compression class- Encoder tries the most likely level first based on FLEVEL: - Class 0 (FLEVEL=0x01): try level 1 - Class 1 (FLEVEL=0x5E): try levels 5, 4, 3, 2 - Class 2 (FLEVEL=0x9C): try level 6 - Class 3 (FLEVEL=0xDA): try levels 9, 8, 7- Last successful level is remembered per class (last_win_[5] array)- Reduces failed verification attempts by 30-40%* ByteBuf with Dynamic Growth- Replaces fixed-size buffers with dynamically growing ByteBuf- reserve() grows buffer by 1.5x when needed- fresh() reallocates if current capacity is insufficient- release() frees memory when buffer is no longer needed- Each Slot maintains its own ByteBuf that can be reused across segments* OutWriter with Buffered Output- 8 MB output buffer (OUT_BUF_SIZE)- Accumulates writes and flushes when buffer is full- Reduces number of syscalls and improves I/O efficiency- space() and commit() allow zero-copy writes from inflate* Inflater/Deflater with Reset- Each worker thread maintains its own Inflater and Deflater- inflateReset() and deflateReset() reuse z_stream state- Simpler than sharded cache, uses less memory- No lock contention, no LRU eviction, no emergency fallback* Prec Container Format v13- CHANGED: Version field from 14 to 13- CHANGED: StreamMetadata size from 26 bytes to 22 bytes- RATIONALE: The v14 format was never released. The v13 format is the actualformat used by v1.09. StreamMetadata is 22 bytes (not 26) due to packing:uint64_t offset ( + uint64_t dec_size ( + uint32_t comp_size (4) +uint8_t level (1) + int8_t windowBits (1) = 22 bytes* OutWriter with Dynamic Adaptive Buffering (The Low-RAM Update)- CHANGED: Removed the hardcoded 8 MB OUT_BUF_SIZE constant from the writer pipeline.- REPLACED WITH: Dynamic core-aware buffering. The OutWriter capacity scales natively based on runtime thread allocation: * Low-Thread / HDD Profile (num_threads <= 2): Caps the output buffer at 2 MB. This prevents physical disk cache congestion (thrashing) on classic mechanical hard drives and keeps the streaming timeline tightly synchronized with I/O bounds. * High-Thread / NVMe Profile (num_threads > 2): Expands the output buffer to 16 MB to maximize sequential write grouping and feed modern high-throughput solid-state storage.- RATIONALE: Fixed-size massive buffers degrade random write performance on legacy storage, while fixed-size tiny buffers choke high-speed NVMe pipelines. Dynamic sizing bridges the architectural gap between vintage and cutting-edge host systems.* Thread Slot Backpressure & Conditional Memory Reclamation- CHANGED: Overrode the passive, threshold-based cache retention logic inside the main loop (`sl.buf.cap > seg_size + 8 * EXT_STEP`).- REPLACED WITH: Conditional aggressive heap reclamation based on hardware profiles: * On Low-Thread configurations (<= 2 threads), the pipeline enforces an immediate and unconditional `sl.buf.release()` upon writing the segment to disk. This drops the active heap footprint to 0 bytes per slot instantly, preventing latent memory pooling in the allocator. * On High-Thread configurations (> 2 threads), the system preserves the original behavior, retaining allocation arenas to minimize global heap lock contention across large multi-core thread clusters.- BENEFIT: Eviscerates hidden memory spikes. Encoder Peak RAM drops to ~88-89 MB under Wine/Windows, while Decoder RAM stabilizes at a flat 45-51 MB of pure RSS memory, matching the footprint performance of ancient single-threaded compression tools.--------------------------------------------------------------------------------4. HYBRID PERFORMANCE BENCHMARK (THE ADAPTIVE VERIFICATION CYCLE)--------------------------------------------------------------------------------Summary of chronological cross-platform performance data using a dense stream distribution workload (**352.78 MB input, 694 zlib streams, setup_virtuaverse_1.38_(92629).exe):Used DualCore with Wine 6.0.3ENCODE PHASE (Compression Analysis)Engine Version Environment/Runtime Peak RAM Encode Time Pipeline Architecture & Control---------------------------------------------------------------------------------------------------------------------------Nitro NG v1.03 (precomp_zlib) Windows x64 (Wine) 1183 MB 2m 18.754s First Release / Unbounded Heap AllocationNitro NG v1.05 Windows x64 (Wine) 1647 MB 2m 33.438s Concurrent Multi-thread / Unthrottled Allocation BloatNitro NG v1.09 Windows x64 (Wine) 89 MB 1m 36.744s Adaptive Backpressure & Aggressive Slot ReleaseNitro NG v1.09 Linux Native (Raw) 104 MB 1m 26.949s Linear Streaming / Direct Page Cache WriteXtool 0.8.9 (Razor12911) Windows x64 (Wine) 489 MB 2m 18.000s High-Thread Buffer AggregationPrecomp 0.48* (Christian S.) Windows x64 (Wine) 80 MB 7m 16.821s Monolithic Sequential / Intense Flags*Note: Precomp 0.48 flags: -cn -intense -t-pzgnfjsmb3 | Recompressed streams: 1859/1885 (Nitro & Xtool match ~696)**Note: Precomp 0.48 uses optimized compilation flags for cmakeDECODE PHASE (Restoration & I/O Target Analysis)Tool & Configuration Environment/Runtime Peak RAM Decode Time Multi-Threading & I/O Profile---------------------------------------------------------------------------------------------------------------------------Precomp 0.48 (Classic) Windows x64 (Wine) 33 MB 2m 59.043s Hardcoded Monolithic Sequential Execution (3x Slower)Xtool 0.8.9 (Modern) Windows x64 (Wine) 353 MB 2m 49.000s High-Thread Buffer Aggregation (HDD Thrashing)Nitro NG v1.09 Windows x64 (Wine) 45 MB 2m 31.654s Bounded Queue with 2 MB Core-Aware Disk FlushingNitro NG v1.09 (Real HDD) Linux Native (Raw) 51 MB 2m 00.947s Pure Uncached Mechanical Plato I/O (Dropped Caches)Nitro NG v1.09 (Cached) Linux Native (Raw) 46 MB 0m 46.000s Native Linux Kernel Page-Cache Streaming ModeConclusion: The implementation of runtime-aware backpressure and conditional memory reclamation turns Nitro_NG v1.09 into an elastic architecture. It safely shrinks historical multi-gigabyte memory leaks down to a baseline of 45-89 MB, transforming a flawed multithreading attempt into a hardware-respectful streaming engine.--------------------------------------------------------------------------------5. BUG FIXES & UX IMPROVEMENTS--------------------------------------------------------------------------------* Console Summary Fix- FIXED: Visual bug where "Levels" summary only displayed the last level- NOW: Correctly shows all levels with counts: "Levels: 6 (x693), 7 (x1)"- ROOT CAUSE: String accumulator used = instead of +=, overwriting previous levels* Progress Line Cleanup- IMPROVED: Final cleanup of progress line to prevent visual artifacts- NOW: Explicit CR+LF output before final summary- BENEFIT: Clean rendering on narrow terminals and Windows/Wine environments* Build System Improvements- FAST TOOLCHAIN DETECTION: Replaced slow `find /usr` scans with directcompiler queries ($CXX -print-file-name=libwinpthread.a)- LOCAL ZLIB 1.3.2: All build scripts compile local optimized zlib 1.3.2and link statically (no external zlib dependency)- PATH NORMALIZATION: Added readlink -f for cleaner winpthread path display--------------------------------------------------------------------------------6. VERIFICATION METRICS (v1.09)--------------------------------------------------------------------------------* Integrity: 100% verified, bit-for-bit restoration matches original input(MD5 confirmed)* RAM Footprint: Bounded and deterministic (~112 MB peak on encode, ~63.9 MBon decode for a 352 MB input with 694 streams)* UX Stability: Clean, single-line live progress with hidden console cursorand artifact-free space padding================================================================================FINAL NOTES================================================================================The journey from v1.05 HP to v1.09 represents a complete architecturaltransformation. We did not "polish" the existing code; we rewrote it from theground up, eliminating components that worked well for typical workloads butcould cause unbounded memory growth on extreme cases.The key insight was simple: if you verify streams in streaming mode (inflate →deflate → memcmp), you never need to store the decompressed data. This singlechange allowed us to eliminate the FNV-1a cache, the sharded dictionary cache,and the per-thread buffer pre-allocations, replacing them with a simpler, moreefficient architecture.Version 1.09 is not just faster and more memory-efficient than v1.07 HP; it isfundamentally different in how it approaches the problem.================================================================================* Historical Context & Technical Respect:- Enforced strict acknowledgment of old-guard pillars (Precomp by Christian Schneider) and modern multi-core powerhouses (Xtool by Razor12911). They were the absolute pioneers of the scene. This benchmark explicitly contextualizes algorithmic behavioral shifts on legacy architectures rather than criticizing raw baseline throughput on high-end nodes.================================================================================="And now, no more table talk. Let's get down to business."END OF FILE=================================================================================
Summary of chronological cross-platform performance data using a dense stream distribution workload (**352.78 MB input, 694 zlib streams, setup_virtuaverse_1.38_(92629).exe):Used DualCore with Wine 6.0.3ENCODE PHASE (Compression Analysis)Engine Version Environment/Runtime Peak RAM Encode Time Pipeline Architecture & Control---------------------------------------------------------------------------------------------------------------------------Nitro NG v1.03 (precomp_zlib) Windows x64 (Wine) 1183 MB 2m 18.754s First Release / Unbounded Heap AllocationNitro NG v1.05 Windows x64 (Wine) 1647 MB 2m 33.438s Concurrent Multi-thread / Unthrottled Allocation BloatNitro NG v1.09 Windows x64 (Wine) 89 MB 1m 36.744s Adaptive Backpressure & Aggressive Slot ReleaseNitro NG v1.09 Linux Native (Raw) 104 MB 1m 26.949s Linear Streaming / Direct Page Cache WriteXtool 0.8.9 (Razor12911) Windows x64 (Wine) 489 MB 2m 18.000s High-Thread Buffer AggregationPrecomp 0.48* (Christian S.) Windows x64 (Wine) 80 MB 7m 16.821s Monolithic Sequential / Intense Flags*Note: Precomp 0.48 flags: -cn -intense -t-pzgnfjsmb3 | Recompressed streams: 1859/1885 (Nitro & Xtool match ~696)**Note: Precomp 0.48 uses optimized compilation flags for cmakeDECODE PHASE (Restoration & I/O Target Analysis)Tool & Configuration Environment/Runtime Peak RAM Decode Time Multi-Threading & I/O Profile---------------------------------------------------------------------------------------------------------------------------Precomp 0.48 (Classic) Windows x64 (Wine) 33 MB 2m 59.043s Hardcoded Monolithic Sequential Execution (3x Slower)Xtool 0.8.9 (Modern) Windows x64 (Wine) 353 MB 2m 49.000s High-Thread Buffer Aggregation (HDD Thrashing)Nitro NG v1.09 Windows x64 (Wine) 45 MB 2m 31.654s Bounded Queue with 2 MB Core-Aware Disk FlushingNitro NG v1.09 (Real HDD) Linux Native (Raw) 51 MB 2m 00.947s Pure Uncached Mechanical Plato I/O (Dropped Caches)Nitro NG v1.09 (Cached) Linux Native (Raw) 46 MB 0m 46.000s Native Linux Kernel Page-Cache Streaming ModeConclusion: Runtime-aware backpressure and conditional memory reclamation safely shrink historical multi-gigabyte memory leaks down to a baseline of 45-89 MB, transforming a flawed multithreading attempt into a hardware-respectful streaming engine.
