Introduction
More than a decade after Heartbleed changed the way organizations viewed OpenSSL security, another OpenSSL issue has captured the cybersecurity community’s attention. Dubbed HollowByte by Okta’s Red Team, this newly disclosed vulnerability demonstrates how a tiny 11-byte TLS request can gradually consume server memory and eventually deny service to legitimate users.
Unlike Heartbleed, HollowByte is not an information disclosure vulnerability. It does not expose encryption keys, leak sensitive memory, or enable remote code execution. Instead, it exploits the way OpenSSL allocates memory during the early stages of a TLS handshake, making it an efficient resource exhaustion attack.
Executive Summary
HollowByte is an unauthenticated Denial-of-Service (DoS) vulnerability affecting multiple OpenSSL branches prior to June 2026 fixes.
The vulnerability allows an attacker to:
- Send only 11 bytes of crafted TLS traffic
- Force the server to allocate as much as 131 KB of memory
- Repeat the process thousands of times
- Cause persistent memory fragmentation
- Eventually trigger Out-of-Memory (OOM) conditions or service degradation on Linux systems using glibc.
Why Is It Called HollowByte?
The attacker advertises that a TLS message is much larger than what is actually transmitted.
The server reserves memory for a payload that never arrives.
The allocated buffer is effectively “hollow”—empty but occupying valuable memory—hence the name HollowByte.
Technical Root Cause
Every TLS handshake message begins with a 4-byte header.
Three of these bytes specify the expected message length.
Older OpenSSL versions trusted this length immediately and allocated a receive buffer before validating whether the client would actually send the remaining data.
The attacker:
- Opens a TLS connection.
- Sends only an 11-byte partial ClientHello.
- Claims the remaining message is much larger.
- Stops transmitting.
- Disconnects.
OpenSSL allocates the memory, waits for the missing payload, and later frees it when the connection closes. However, on many Linux systems, the glibc memory allocator retains those freed chunks instead of returning them to the operating system. By varying the claimed message size across many connections, attackers can fragment the heap and steadily increase the process’s resident memory footprint.
Why Traditional Defenses May Fail
Unlike Slowloris, HollowByte does not rely on maintaining huge numbers of concurrent connections.
Instead, it exploits memory allocator behavior.
Even after the malicious connections terminate:
- Memory remains reserved by glibc.
- Process memory usage continues to grow.
- Restarting the application is often required to fully reclaim the memory.
Potential Impact
Internet-facing services using vulnerable OpenSSL releases may experience:
- HTTPS service degradation
- Reverse proxy instability
- API gateway outages
- VPN service interruptions
- TLS-enabled application crashes
- Increased risk of OOM kills by the operating system.
Affected Versions
All releases before the following fixed versions are affected:
- OpenSSL 4.0.1
- OpenSSL 3.6.3
- OpenSSL 3.5.7
- OpenSSL 3.4.6
- OpenSSL 3.0.21
Earlier releases in those branches should be upgraded.
The Unusual Disclosure Story
One of the most interesting aspects of HollowByte is how it was handled.
According to public reporting:
- No CVE identifier was assigned.
- No dedicated OpenSSL security advisory was published.
- The fix was included in June 2026 releases without explicitly identifying it as a security vulnerability.
- The issue became widely known only after Okta’s Red Team disclosed its findings and named the vulnerability “HollowByte.”
This highlights that organizations should not rely solely on CVE feeds when assessing the importance of upstream software updates.
Detection Opportunities
Security Operations Centers should monitor for:
- Sudden growth in OpenSSL process memory
- Large numbers of incomplete TLS handshakes
- Unexpected increases in TLS connection failures
- Frequent OOM killer events
- Repeated short-lived TLS sessions from the same or distributed IP addresses
- Long-lived processes with continuously increasing RSS memory usage.
Mitigation
Organizations should:
- Upgrade to patched OpenSSL releases.
- Restart affected services after patching to reclaim fragmented memory.
- Implement TLS connection rate limiting where appropriate.
- Monitor memory utilization on public-facing TLS services.
- Review reverse proxy and load balancer configurations for abnormal handshake behavior.
Final Thoughts
HollowByte may not rival Heartbleed in terms of confidentiality impact, but it is a powerful example of how subtle implementation details in widely deployed cryptographic libraries can have significant operational consequences. A carefully crafted 11-byte TLS request is sufficient to trigger disproportionate resource consumption, making this vulnerability a reminder that availability is as critical a pillar of cybersecurity as confidentiality and integrity.
