Two weeks after the Heartbleed bug was announced, CloudFlare patched the Heartbleed bug, created a challenge to prove the bug could be used to find private keys (uncovering a second bug in OpenSSL) and turned its entire network into a giant honeypot. This session will discuss the specific steps taken to prevent early disclosure, creating and scaling the first public vulnerability test, how the CloudFlare Heartbleed challenge showed that you can reveal private SSL keys (how a second bug in OpenSSL made this possible) the incredible impact of revoking over 100,000 certificates in a single day, and the results of our honeypot revealing the proportion of attack traffic versus research traffic.
CloudFlare was notified about Heartbleed as soon as it was discovered--ahead its public announcement--and took extreme precaution to not reveal anything about the bug. This required communicating only over secure channels, restricting the visibility of the branch from which we built the workaround, and using secure software deployment methods. After the patch was announced, there was a rush to reverse engineer the bug and create an exploit. The cloudFlare team immediately started working proof of concept, and hosted it on a website allowing others to scan for vulnerable sites. Within minutes, the original site was flooded with requests. CloudFlare’s Nick Sullivan will share this process and the feats pulled off to make sure the site could scale and provide accurate results. He will go into the numbers and technical details of the PoC and speak about its bugs and how they were found. Statistics and anonymized raw data of the 70+ millions of results will be provided, giving an overview of the patching process over time. It was clear soon after the bug was revealed that the number of servers affected by this bug was massive. What wasn’t clear was the scope of data that was vulnerable to attack. In order to determine the risk to private keys from this vulnerability, his team launched the CloudFlare Heartbleed Challenge. They set up a site that was vulnerable to the attack, added logging and created a webpage to submit a signed proof of key ownership. In less than a day, there were several successful submissions. Nick will go over the naive (but successful) strategy used to extract keys and the more advanced technique based on Coppersmith’s Method. Finally he will discuss the *second* OpenSSL bug we discovered that allowed the private key to be extracted via Heartbleed. After the exploits were in the wild, his team added logging to see who was trying to exploit this bug. Nick will reveal the results of this analysis and cross-reference the results with the IPs of the test site. These numbers give new insight into how many people were attempting to maliciously exploit this flaw versus research done to probe vulnerable sites. Once the dust settled and the team worked out the details with our CA, we revoked over 80,000 of CloudFlare’s SSL certificates. This turned into an internet scaling nightmare, resulting in a constant flood of more than 40 gigabits per second of traffic to serve overgrown certificate revocation lists. Since CloudFlare provides caching for its CA, the team bore the brunt of this traffic. Their revocation lists would have DDoSed most sites (and some certificate authorities) off the internet. Nick will talk about caching CRLs, and how the revocation system was not designed for this scale of internet flaw. In conclusion we he will summarize the many ways this coding error revealed some of the deeper flaws in the internet, and discuss ways we can move forward. Nick will share actionable advice and the security strategies used by cloud service companies on how to monitor the way companies store keys internally. Attendees will leave with actionable advice on how to better secure their own systems against the next Heartbleed and the security strategies used by cloud service companies on how to monitor the way companies store keys internally.
Speakers: Nick Sullivan