Heartbleed — the memory-disclosure classic, for the record
Still finding Heartbleed lurking in embedded/legacy appliances? Patch OpenSSL, rotate the keys, reissue the certs.
01 The Big PictureFor leadership · no jargon
Okay, so this one needs no introduction — but if a scanner has just flagged it on something you own, introductions are due anyway. OpenSSL is the software that puts the padlock on internet traffic: back in 2014 it was encrypting connections for roughly two-thirds of all websites, plus email servers, VPNs and chat. Heartbleed is a bug in that software. The simplest analogy I've heard: it's like a hotel concierge you can ask to "read back the last N words from the notepad". Lie about N, and the concierge cheerfully keeps reading past your message into whatever else is on the pad — other guests' messages, the safe combination, the master key. An attacker could ask a vulnerable server to read out chunks of its own memory, over and over, and the server would just... comply.
What does an attacker actually get, in business terms? The master keys protecting all of your encrypted traffic, plus whatever else happened to be in memory at the time — customer usernames and passwords, active login sessions, private documents. With the master key in hand, they can decrypt traffic they recorded earlier and impersonate your site convincingly. The theft leaves no trace in normal logs, which is the properly nasty part. And this was not theoretical: it was exploited at massive scale within days of disclosure in April 2014. Canada's tax agency lost 900 taxpayer IDs, and a US hospital group lost 4.5 million patient records through an unpatched VPN appliance. Yikes.
Today the urgency is different but real. Any modern server was patched a decade ago; where Heartbleed still shows up is legacy and embedded gear — old VPN concentrators, load balancers, printers and appliances that never received firmware updates. If a scan flags one of yours, treat it as having been exposed since the day it went online.
Who acts: whoever owns the flagged box, immediately. On a normal Linux server the fix is a routine package update plus service restarts — minutes of work, no data loss. The heavier lift comes after: because the master keys may already have leaked, you need to generate new certificates and force everyone to log in again. On an appliance, fixing means applying vendor firmware — or replacing the box if the vendor no longer ships updates.
02 Technical BreakdownFor engineers
Right, the nitty-gritty. CVE-2014-0160 is a missing bounds check in the TLS/DTLS heartbeat extension (RFC 6520), in OpenSSL's tls1_process_heartbeat(). A heartbeat request carries a small payload plus a *claimed* payload length of up to 65,535 bytes. A vulnerable server copies the claimed number of bytes out of its receive buffer into the response without checking that the request actually contained that many bytes — so it happily returns the payload plus whatever heap memory sits after it, up to roughly 64 KiB per request. Repeat the request indefinitely and you get to go for a walk through the heap.
Affected code: OpenSSL 1.0.1 (released March 2012) through 1.0.1f inclusive, and 1.0.2-beta1. The 0.9.8 and 1.0.0 branches never implemented heartbeat at all and are unaffected. The attack vector is any TLS endpoint linked against the vulnerable libssl — HTTPS on 443, SMTP over STARTTLS, OpenVPN-style concentrators, XMPP — reachable without authentication. It also runs in reverse: a malicious server can bleed memory from a vulnerable client, which is why Android 4.1.1 devices were exposed. Exploitation is silent; heartbeat handling lives entirely inside OpenSSL, below application logging. What surfaces in a given 64 KiB chunk is luck of the heap, but repetition pays: during the public Cloudflare challenge, researchers recovered a full private key within hours.
The fix in 1.0.1g adds the missing length check and silently discards malformed heartbeat requests; the supported alternative is recompiling with -DOPENSSL_NO_HEARTBEATS, which removes the extension entirely. One packaging trap to watch for: distributions backported the fix, so Red Hat/CentOS 6 shipped it as openssl-1.0.1e-16.el6_5.7 — the version string still reads 1.0.1e, and openssl version alone will mislead you. Judge by package build or changelog. Finally, patching only stops the bleeding; it does nothing about material already leaked. That's why key regeneration, certificate revocation and session rotation are part of the remediation below, not optional hygiene.
03 Affected
OpenSSL 1.0.1 through 1.0.1f inclusive, and 1.0.2-beta1 — every service, client and appliance linked against those builds. Fixed in 1.0.1g and 1.0.2-beta2. OpenSSL 0.9.8 and 1.0.0 are not affected.
In practice, current exposure lives in legacy and embedded gear: end-of-life VPN concentrators, load balancers, TLS-inspection boxes, printers and network appliances frozen at 2012–2014-era OpenSSL, plus Android 4.1.1 clients. Mind the backport trap from earlier: on RHEL/CentOS the fixed package still reports version 1.0.1e (1.0.1e-16.el6_5.7 and later are patched), and Ubuntu shipped the fix as 1.0.1-4ubuntu5.12. Check the package changelog, not the upstream version letter.
04 Detection
openssl version -a | head -1 Identify the build on legacy gear. 1.0.1 through 1.0.1f is vulnerable, 1.0.1g or later is fine — but see the backport caveat above before trusting the letter.
dpkg -l | grep -E "openssl|libssl" On Debian/Ubuntu, lists installed OpenSSL packages; compare against the distro's fixed build (Ubuntu 14.04: 1.0.1-4ubuntu5.12).
rpm -q --changelog openssl | grep -i "CVE-2014-0160"On Red Hat-family systems, confirms the backported fix is present; expect a changelog line referencing the CVE.
nmap --script ssl-heartbleed -p 443 host Actively probes the service with a malformed heartbeat; nmap reports VULNERABLE or NOT VULNERABLE. The definitive remote check.
sudo lsof -n | grep libssl | grep DELLists processes still mapping the old, deleted library after a patch — daemons you forgot to restart. We've all been there.
05 Remediation
01 Patch the library
First things first: upgrade OpenSSL to the fixed build. The library on disk is what every TLS service loads, so this is the root fix; distro packages pull in the backported patch even when the upstream version letter looks unchanged (see above — the letter lies).
sudo apt-get install --only-upgrade openssl libssl1.0.0Patch the library (Debian/Ubuntu). Confirm the installed build against your distro's advisory afterwards.
sudo yum update openssl Red Hat/CentOS equivalent; expect 1.0.1e-16.el6_5.7 or later.
02 Restart every service linked against libssl
Here's the bit people miss: patching the file on disk does not touch daemons already running — they keep the vulnerable code in memory until restarted. Let's enumerate everything linked against libssl and restart the lot; on an appliance, a full reboot is the blunt but reliable catch-all.
sudo lsof | grep libssl | awk '{print $1}' | sort -uInventory every process using the library; each one needs a restart.
sudo systemctl restart nginxRestart TLS-terminating daemons — web servers, mail, VPN, proxies, not just nginx.
03 Regenerate keys and reissue certificates
Now the less fun part. Assume the old private key leaked — there is no way to prove it didn't. Generate a fresh key and CSR, have your CA issue a new certificate, deploy it, then revoke the old certificate with the CA so it can't be abused to impersonate you.
sudo openssl req -new -newkey rsa:2048 -nodes -keyout server.key -out server.csr Generate a new private key and CSR in one step; submit server.csr to your CA and deploy the returned certificate.
04 Rotate credentials and sessions
Anything that crossed TLS during the exposure window may have been lifted from memory — passwords, session cookies, API tokens. Closing the hole without rotating these leaves stolen sessions valid, which rather defeats the point.
No single command here: invalidate web sessions, revoke OAuth grants and API tokens, and force password resets for accounts that authenticated over TLS while the box was vulnerable. Watch for replayed sessions in the days after.
06 Verification
nmap --script ssl-heartbleed -p 443 host The proof that matters: expect NOT VULNERABLE. Run it against every TLS port, not just 443.
sudo lsof -n | grep libssl | grep DELExpect empty output — no process still maps the pre-patch library.
openssl s_client -connect host:443 -tlsextdebug 2>&1 | grep -i heartbeatShows whether the server advertises the heartbeat extension. Patched 1.0.1g builds still do — presence proves nothing either way; only the nmap probe above is the verdict. Absence means the extension was compiled out.
echo | openssl s_client -connect host:443 -servername host 2>/dev/null | openssl x509 -noout -serial -startdateConfirms the reissued certificate is live: the serial and start date should match the new cert, not the old one.
07 Rollback
Two different things here, only one reversible. If the OpenSSL upgrade itself breaks an application, you can pin back to the previous package build — but that re-opens the CVE, so treat it strictly as a bridge while you fix forward.
sudo apt-get install openssl=<previous-version> libssl1.0.0=<previous-version>Downgrade to the prior build only as a temporary bridge; you are vulnerable again until you re-upgrade.
Key and certificate rotation has no rollback — rotated keys can't be un-rotated, and you wouldn't want them back anyway: the old key material must be considered burned. There is no scenario where rolling back to the old certificate is correct.
08 References
- NVD — CVE-2014-0160
- OpenSSL Security Advisory, 07 Apr 2014
- heartbleed.com — original disclosure write-up
- CISA Known Exploited Vulnerabilities catalog
---
Spotted a mistake, or got a question about a weird appliance? Corrections and questions are always welcome.
Ship the fix. Prove it’s closed.
If this runbook doesn’t fit your stack, send the CVE — an operator writes a verified path, usually within a business day. Or contribute your own and get co-authored credit.