Welcome to DU! The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards. Join the community: Create a free account Support DU (and get rid of ads!): Become a Star Member All Forums Issue Forums Culture Forums Alliance Forums Region Forums Support Forums Help & Search
 

Klaralven

(7,510 posts)
Thu Oct 7, 2021, 09:41 AM Oct 2021

Facebook rendered spineless by buggy audit code that missed catastrophic network config error

In a write-up by infrastructure veep Santosh Janardhan, titled "More details about the October 4 outage," the outrage-monetization giant confirmed early analyses that Facebook yesterday withdrew the border gateway protocol (BGP) routing to its own DNS servers, causing its domain names to fail to resolve.

That led to its websites disappearing, apps stopping, and internal tools and services needed by staff to remedy the situation breaking down as well.

But this DNS and BGP borkage turns out to have been the consequence of other errors. Janardhan explained that it operates two classes of data center.

One type was described as "massive buildings that house millions of machines," performing core computation and storage tasks. The other bit barns are "smaller facilities that connect our backbone network to the broader internet and the people using our platforms."

Users of Facebook's services first touch one of those smaller facilities, which then send traffic over Facebook's backbone to a larger data center. Like any complex system, that backbone is not set-and-forget – it requires maintenance. Facebook stuffed that up.

"During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network," Janardhan revealed.

https://www.theregister.com/2021/10/06/facebook_outage_explained_in_detail/

9 replies = new reply since forum marked as read
Highlight: NoneDon't highlight anything 5 newestHighlight 5 most recent replies
Facebook rendered spineless by buggy audit code that missed catastrophic network config error (Original Post) Klaralven Oct 2021 OP
Facebook has always been spineless. lagomorph777 Oct 2021 #1
This guy should be fired. lapfog_1 Oct 2021 #2
It is odd that they don't have any oob access, aye? Sapient Donkey Oct 2021 #3
I think he should be nominated for a Nobel Peace Prize. lagomorph777 Oct 2021 #4
yes, there is that lapfog_1 Oct 2021 #6
I used Arpanet in 1979 to download code for an industrial automation system. lagomorph777 Oct 2021 #7
DEC-10 with fanfold paper tape lapfog_1 Oct 2021 #8
I used the punch cards in the early 1980s. Kind of annoyed with that at the time. lagomorph777 Oct 2021 #9
16 billion must not be enough to buy out-of-band system management? Klaralven Oct 2021 #5

lapfog_1

(29,166 posts)
2. This guy should be fired.
Thu Oct 7, 2021, 09:56 AM
Oct 2021

"It was not possible to access our data centers through our normal means because their networks were down, and … the total loss of DNS broke many of the internal tools we'd normally use to investigate and resolve outages like this," Janardhan stated.

Idiots.

I design and build large data centers for the likes of NASA and now a major chip maker.

I ALWAYS design in a "backdoor" that depends on NOTHING being available other than standard TCP/IP routers and switches. No DNS, no BGP. Nothing fancy. And I build automation to restore services just in case something like this happens.

Sigh.

lagomorph777

(30,613 posts)
4. I think he should be nominated for a Nobel Peace Prize.
Thu Oct 7, 2021, 11:10 AM
Oct 2021

He gave us a few hours less of the hate machine known as Facebook.

lapfog_1

(29,166 posts)
6. yes, there is that
Thu Oct 7, 2021, 12:52 PM
Oct 2021

I have a facebook account, and a twitter account, and a snapchat, myspace, and god knows what else.

some old friends do post things to facebook that I get notifications on.

I NEVER post to social media. nope, not ever... if my old friends post something that I feel motivated to comment on, I send them an email.
DU is the only social media I ever post to or visit to read.

I have the accounts because I don't want anyone impersonating me online.

But I am their worst user. They get nothing from me for their advertisements. Unfortunately, the pandemic has forced me to use Instacart and Amazon to order things... so now many websites are littered with ads for things I already purchased.

And I've been using the "Internet" since 1977 (back when it was Arpanet and the complete "map" showed around 50 or 60 systems) plus I wrote some of the translating code for the BBN IMPs.

lagomorph777

(30,613 posts)
7. I used Arpanet in 1979 to download code for an industrial automation system.
Thu Oct 7, 2021, 01:18 PM
Oct 2021

It was very high-tech. I downloaded the code onto a paper tape, which I then read into the "ladder" controller.

lapfog_1

(29,166 posts)
8. DEC-10 with fanfold paper tape
Thu Oct 7, 2021, 04:56 PM
Oct 2021


or teletype with the paper tape reel



I used both... started out with IBM 026 punch cards writing Fortran in 1970





lagomorph777

(30,613 posts)
9. I used the punch cards in the early 1980s. Kind of annoyed with that at the time.
Fri Oct 8, 2021, 09:08 AM
Oct 2021

But looking back, I'm glad my school afforded me the "old school" experience.

The paper tape I used was om a spool; I am pretty sure was the same machine as you photo (Teletype?)

Latest Discussions»General Discussion»Facebook rendered spinele...