CVE-2026-120756: 
NixOS vulnerability analysis and mitigation

CVE-2026-120756 is a heap buffer over-read vulnerability in the HDF5 library (versions through 2.2.0) affecting the H5O__chunk_deserialize() function in src/H5Ocache.c. When decoding object header messages, the declared message size is validated against the end of the chunk image before the full message header (flags and optional creation index) has been decoded, allowing a message body to start up to four bytes further into the chunk than the check accounts for. A crafted HDF5 file can exploit this flaw during H5Fopen() to cause denial of service or limited process memory disclosure. It carries a CVSS v3.1 base score of 6.1 (Medium) and a CVSS v4.0 base score of 1.9 (Low) (GitHub Advisory, Red Hat Bugzilla).

The root cause is an out-of-bounds read (CWE-125) in H5O__chunk_deserialize() (src/H5Ocache.c). The size check for an object header message is performed while chunk_image still points at the flags byte — before the flags, reserved bytes (version 1), or creation index (version 2) are consumed — so the actual message body can begin up to four bytes further into the chunk than the check assumes. A maliciously crafted HDF5 file can declare a message size that passes this premature check but causes the message body to extend past the end of the heap-allocated chunk image. The decode callback (e.g., H5O__mtime_new_decode() in src/H5Omtime.c) is then handed a buffer end pointer outside the allocation, resulting in a READ of 4 bytes past the 279-byte heap region as confirmed by AddressSanitizer (HDF5 Issue #6401, HDF5 PR #6589). The fix, merged in PR #6589, moves the size check to after the full message header is decoded in H5O__chunk_deserialize() (HDF5 PR #6589).

Successful exploitation can cause the application processing the HDF5 file to crash (denial of service) or leak a limited amount of process memory contents. The vulnerability is triggered at file-open time via H5Fopen(), meaning any application or pipeline that opens untrusted HDF5 files is at risk. Integrity is not affected, and the memory disclosure is limited in scope; lateral movement or privilege escalation are not directly enabled by this flaw (GitHub Advisory, Red Hat Bugzilla).

A proof-of-concept (PoC) fuzzer crash file (poc.zip) was publicly disclosed in HDF5 GitHub issue #6401 on May 7, 2026, with concrete reproduction steps using AddressSanitizer and libFuzzer (HDF5 Issue #6401). The CVSS v4.0 exploit maturity is rated "Proof of Concept." There is no evidence of in-the-wild exploitation, no known threat actor attribution, and the vulnerability is not listed in the CISA KEV catalog. Exploitation requires user interaction (opening a malicious file) and is local in attack vector, limiting its practical reach (GitHub Advisory).

1. Craft a malicious HDF5 file: Create or use the published PoC HDF5 file with a crafted object header — specifically a version 1 header with a chunk containing an mtime message whose declared body size passes the premature chunk-end check but extends past the actual heap allocation boundary.
2. Deliver the file to the target: Social-engineer a user or automated pipeline into opening the malicious .h5 file, or place it in a location where an application automatically processes HDF5 files.
3. Trigger file open: The target application calls H5Fopen() on the crafted file, initiating the call chain: H5Fopen → H5F_open → H5G_mkroot → H5O_msg_exists → H5AC_protect → H5O__chunk_deserialize.
4. Trigger out-of-bounds read: H5O__chunk_deserialize() performs the size check prematurely, passes it, and invokes the message decode callback (e.g., H5O__mtime_new_decode()), which reads 4 bytes past the end of the 279-byte heap region.
5. Achieve denial of service or memory disclosure: The process crashes (ASan abort or segfault) or, in non-sanitized builds, reads adjacent heap memory that may contain sensitive process data (HDF5 Issue #6401).

• File System: Presence of unexpected or externally sourced .h5 or .hdf5 files in application input directories, particularly files with anomalously small chunk sizes or malformed object headers.
• Process: Application crash or abnormal termination immediately upon opening an HDF5 file; core dumps or crash reports referencing H5O__mtime_new_decode, H5O__chunk_deserialize, or H5Ocache.c/H5Omtime.c in stack traces.
• Logs: Application error logs showing H5Fopen() failures or memory access violations; AddressSanitizer reports indicating heap-buffer-overflow at H5Omtime.c:137.
• Network: Unexpected delivery of .h5/.hdf5 files via email attachments, web downloads, or shared storage from untrusted sources to systems running HDF5-dependent applications (HDF5 Issue #6401).

Upgrade HDF5 to version 2.3.0 or later, which includes the fix merged in PR #6589 that moves the object header message size check to after the full message header is decoded in H5O__chunk_deserialize() (HDF5 PR #6589). As a workaround, avoid opening HDF5 files from untrusted or unverified sources, and implement application-level file validation or sandboxing before processing HDF5 files. Organizations using Red Hat products should monitor the Red Hat Bugzilla entry for downstream package updates (Red Hat Bugzilla).

The HDF5 maintainers triaged the issue promptly after public disclosure on May 7, 2026, with a collaborator noting the root cause was in buffer size calculation rather than the H5_IS_BUFFER_OVERFLOW macro itself (HDF5 Issue #6401). A community contributor submitted a fix (PR #6589) on August 3, 2026, which was reviewed and merged on September 12, 2026 (HDF5 PR #6589). Red Hat opened a tracking bug and assigned medium severity, reflecting the limited exploitability of the flaw (Red Hat Bugzilla). No significant broader media coverage or social media discussion has been identified.