Cloudflare yesterday faced a brief yet disruptive outage that knocked down numerous websites, including some of the popular ones. The outage was caused by the deployment of bad software, which has now been rolled back.
Cloudflare is one of the largest networks in the world, serving more than 16 million internet properties. Businesses, non-profit organizations, and bloggers use Cloudflare to accelerate and secure their presence on the internet.
The outage lasted for around half an hour yesterday. During the outage, the websites faced downtime, showing the ‘502 Bad Gateway’ error.
Famous websites like Pinterest, BuzzFeed, Coinbase Pro, and business apps like Dropbox faced the bad gateway error, according to Down Detector.
Matthew Prince, Co-founder and CEO of Cloudflare, acknowledged in a tweet that he is aware of the outage and the team is working on getting to the bottom of the issue.
He later updated that the issue causing the outage had been mitigated. Traffic was restored. His team was working on restoring all the services globally.
What actually happened at Cloudflare?
Cloudflare had deployed new rules within its Web Application Firewall (WAF) that could help in blocking the inline JavaScript used in attacks. Unfortunately, one of the rules was misconfigured that caused 100% CPU spike, resulting in 502 errors.
“For about 30 minutes today, visitors to Cloudflare sites received 502 errors caused by a massive spike in CPU utilization on our network. This CPU spike was caused by a bad software deploy that was rolled back. Once rolled back the service returned to normal operation and all domains using Cloudflare returned to normal traffic levels,” wrote John Graham-Cumming, CTO of Cloudflare, in a blog post.
Following reviews, Cloudflare rolled back the misconfigured rule and tested the changes to ensure that the issue was fixed correctly.
