"When your website goes down, it forces you to put your money where your mouth was" -- Buzzfeed's Ben Smith on last night's outage
Just before 7pm Eastern last night, editor-in-chief Ben Smith noticed that one section of Buzzfeed.com was getting a lot more traffic than usual: the error page. Something was clearly wrong, so he messaged Mark Wilkie, Buzzfeed's Director of Technology. A few minutes later, Buzzfeed's staff had lost access to the entire back end of the site. They finally learned the reason for the outage, when a text arrived from an official at the datacenter housing Buzzfeed's servers:
"Basement flooded, fuel pump off line - we got people working on it now. Five feet of water now.”
Although portions of Buzzfeed.com were cached and viewable to the public, the staff was unable to publish new content. This would be a crisis on any night, but especially last night as Hurricane Sandy was making landfall. So Smith decreed that his editors would shift their focus to each of Buzzfeed's Tumblr pages. And If a vertical page didn't have a Tumblr yet, they'd create one.
Switching to social platforms wasn't terribly difficult for Buzzfeed, says Smith, who prefers to think of stories less as static items that live on a particular site and more like atomized pieces of content that are distributed and transported on social platforms.
"The logical conlcusion of our way of looking at the media universe is that you wouldn't even have a website," says Smith. "And then when your website goes down, it forces you to put your money where your mouth was."
The transition wasn't entirely painless. At one point, social media editor Mike Hayes lost power, and so Smith had to take over the feeds in a pinch. But ultimately it was a great night and morning for Buzzfeed stories. In one article, Jack Stuef managed to uncover the identity of @ComfortablySmug, a Twitter user who spent last night spreading false rumors about the storm while others were trying to deliver helpful and accurate information. Another great story by John Herrman was about how Twitter, despite its capacity for spreading misinformation, was in fact a "truth machine." It's been shared by TechCrunch, Techmeme, and Poynter.
"I like our posting interface better," Smith said, "but content is content."
To a point, anyway. Writers were filing stories again, but there was still the problem of getting the site back online. After it became clear that the power wasn't coming back to the datacenter anytime soon, Buzzfeed decided to rebuild its site from scratch and host it in the cloud using Amazon Web Services.
Just three developers worked throughout most the night to get Buzzfeed.com back up and running. One of them, Eugene Ventimiglia kept working even after a tree fell fell through the roof of his home in North New Jersey.
"it took years to build (Buzzfeed) and they rebuilt it in six hours," Smith said.
Buzzfeed's story is significant for a couple reasons. One, it's a validation of the core principle driving Buzzfeed, which is that news doesn't need a homepage to survive, it only needs platforms on which it can be shared and consumed. The second is that, with a bit of technological creativity and a ton of hard work, news sites can weather even storms as catastrophic as Sandy. Which is a good thing, because it's during natural disasters like Sandy that people need the news the most.