CVE-2026-50538: 
Linux Debian vulnerability analysis and mitigation

CVE-2026-50538 is a heap out-of-bounds write vulnerability in the libvncclient Tight decoder component of libvncserver. A malicious or man-in-the-middle VNC server can force a connecting libvncclient to write attacker-controlled data past the end of its framebuffer allocation, enabling denial of service or remote code execution. The vulnerability affects libvncserver versions 0.9.12 through current master (including 0.9.15+), introduced by commit d7b1462 in January 2018. It carries a CVSS v3.1 base score of 8.8 (High) (GitHub Advisory). As of the time of disclosure, no vendor-supplied patch is available (OSV Debian, OSV Ubuntu).

The root cause is an attacker-controlled heap-based buffer overflow (CWE-787 / CWE-122) in src/libvncclient/tight.c, specifically in the HandleTightBPP() function (compiled as HandleTight8/16/32). The Tight basic-compression zlib decompression loop writes batches of decompressed rows directly into the framebuffer via filterFn (FilterCopyBPP, FilterPaletteBPP, or FilterGradientBPP) but only validates the total row count after the loop completes — meaning a server-controlled zlib payload that decompresses to more rows than the rectangle's declared height (rh) causes writes past the end of the framebuffer allocation. The per-rectangle bounds check in rfbclient.c validates only the declared width and height, not the decompressed row count, so a geometrically valid rectangle (e.g., 16×16) with an over-expanding zlib payload bypasses all checks. Tight encoding is advertised by default when the library is built with zlib and libjpeg (both ON by default), and exploitation requires no authentication since the attacker controls the server (GitHub Advisory).

Exploitation unconditionally crashes any vulnerable libvncclient-based application (denial of service). Because real VNC viewers (SDL, GTK, Qt-based) typically point client->frameBuffer into their own objects (e.g., sdl->pixels), the out-of-bounds write corrupts adjacent live heap memory with attacker-controlled length, contents, and offset. The advisory authors demonstrated overwriting an application callback pointer and redirecting execution to attacker-chosen code under the default configuration, achieving full code execution — though exploiting this for arbitrary code execution on ASLR-enabled binaries additionally requires an information leak not provided by this bug alone. Any application embedding libvncclient is affected, including desktop VNC viewers and services that connect to VNC servers (GitHub Advisory).

A proof-of-concept exploit (including a malicious Python VNC server evil_server.py, a minimal client harness, and a control-flow-hijack demonstration) was developed and attached privately to the GitHub Security Advisory by the reporter. Exploitation requires no authentication and fires from a single FramebufferUpdate packet the moment the victim connects to a hostile server. The vulnerability is exploitable via network (AV:N), with low complexity (AC:L), no privileges required (PR:N), and only requires user interaction in the form of the client initiating a connection. No CISA KEV listing or EPSS score is currently available, and no in-the-wild exploitation has been publicly reported as of disclosure (GitHub Advisory).

1. Set up a malicious VNC server: Deploy evil_server.py (Python standard library only) on a host the victim will connect to, or position as a man-in-the-middle on the victim's VNC connection. The server advertises RFB 3.8 with security type "None".
2. Wait for victim connection: When the libvncclient-based viewer connects, the server completes the RFB handshake without requiring authentication.
3. Send malicious FramebufferUpdate: Immediately after handshake, send a single FramebufferUpdate containing a Tight-encoded rectangle with valid declared geometry (e.g., 16×16 pixels within a 16×16 framebuffer) but a zlib payload that decompresses to far more rows than the declared height.
4. Trigger out-of-bounds write: The HandleTightBPP() loop decompresses the zlib stream and calls filterFn for each batch of rows without clamping to rh, causing writes past the end of the 1024-byte framebuffer allocation (16×16×4 bytes) into adjacent heap memory.
5. Achieve denial of service or code execution: At minimum, the client crashes. For code execution (as demonstrated in evil_rce.py), craft the overflow to overwrite an adjacent application callback pointer (e.g., in SDL/GTK viewers where client->frameBuffer = sdl->pixels) and redirect execution to attacker-controlled code (GitHub Advisory).

• Network: Unexpected or unsolicited VNC server connections (inbound RFB 3.8 traffic on port 5900) from untrusted hosts; VNC sessions where the server sends a FramebufferUpdate immediately after handshake with no prior client request.
• Process: VNC client process (e.g., vncviewer, SDL/GTK-based viewer) crashing unexpectedly or exiting with a segmentation fault or heap corruption error shortly after connecting to a server.
• Logs: AddressSanitizer or similar memory-safety tool output referencing FilterCopy32/HandleTight32/HandleRFBServerMessage in stack traces; application crash logs from libvncclient-based tools with heap-buffer-overflow indicators.
• File System: Core dump files generated by VNC client processes in working directories, potentially containing evidence of heap corruption (GitHub Advisory).

No vendor-supplied patch is available as of the advisory publication date (May 29, 2026), and no fixed version of libvncserver has been released (GitHub Advisory). The advisory suggests a code-level fix: clamp numRows to rh - rowsProcessed inside the decompression loop before calling filterFn, and return FALSE with a log message if exceeded. As a workaround, organizations should avoid connecting libvncclient-based applications to untrusted or unverified VNC servers, enforce network-level controls (firewalls, VPNs) to prevent man-in-the-middle attacks on VNC sessions, and consider disabling Tight encoding support at build time (compile without WITH_ZLIB/WITH_JPEG) if operationally feasible. Monitor the upstream LibVNC/libvncserver repository for patch releases (GitHub Advisory).

The vulnerability was reported by Bas Levering (credited as "Bazskillz") and published by maintainer bk138 on May 29, 2026 as a GitHub Security Advisory. The advisory notes that a recent upstream commit (5b27054, "fix Tight gradient decoding overflow", May 2026) does not address this issue, as it only clamps rectangle width for a gradient stack buffer and does not protect against the decompressed row count overflow (GitHub Advisory). No broader media coverage or notable social media reactions have been identified at this time.