An unpatched OpenSSL server can have up to 131 KB of memory frozen per connection by a malicious TLS handshake message just 11 bytes long, with the consumed memory never returning to the kernel on glibc-based systems until the process restarts. The denial-of-service vulnerability, named HollowByte by Okta’s Red Team that discovered and reported it, was quietly fixed in OpenSSL releases dated June 9 with no CVE identifier, no security advisory, and no changelog entry.
The flaw lies in how OpenSSL handles inbound TLS handshake messages. Each message carries a 4-byte header, three bytes of which declare the expected body length. Older versions allocate a receive buffer matching that declared size the moment the header arrives, before any body data or handshake validation takes place. An attacker sending a stream of headers with inflated length values but never delivering the bodies forces the server to reserve memory that glibc’s allocator fragments rather than reusing.
In Okta’s tests against NGINX, a 1 GB server was killed by the out-of-memory handler after HollowByte locked up 547 MB. Against a 16 GB server, the attack consumed 25 percent of system memory without ever hitting the connection ceiling, leading the researchers to conclude that standard connection-limiting defenses are insufficient.
Fixed releases include OpenSSL 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21. However, downstream distributions that backport patches rather than updating version numbers may report the same vulnerable version string even after applying the fix. System administrators must verify with their vendor or check for pull requests 30792, 30793, and 30794 on the OpenSSL GitHub repository.
Adding to the concern, the OpenSSL security team classified this as a “bug or hardening” fix rather than a vulnerability, meaning scanners and CVE-keyed inventory tools will not flag it. The DTLS path remains unpatched.
