Welcome to DU! The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards. Join the community: Create a free account Support DU (and get rid of ads!): Become a Star Member All Forums Issue Forums Culture Forums Alliance Forums Region Forums Support Forums Help & Search

ChairmanAgnostic

(28,017 posts)
Tue Aug 9, 2016, 10:18 AM Aug 2016

Delta Meltdown

So, one of the best run, profitable, and on-time airlines built its own internal computing system to track flights, issue tickets, handle scheduling, billing, and more.

It is a complex, robust, effective computer system, far better than the 1890's technology that Southwest Air has been trying to replace.

Can someone explain to me how a power failure in Atlanta caused the airline to collapse? Even my home office computer has a backup power system, letting me work even if the office is getting no power. Do they really expect us to believe that with all its programing and computer successes, Delta did not have a backup power system set up to prevent such a meltdown?

20 replies = new reply since forum marked as read
Highlight: NoneDon't highlight anything 5 newestHighlight 5 most recent replies
Delta Meltdown (Original Post) ChairmanAgnostic Aug 2016 OP
Okay no airline has 1890's tech... Just saying. Agschmid Aug 2016 #1
Amish Airways. AngryAmish Aug 2016 #2
He probably meant 1990. Massacure Aug 2016 #3
No typo. ChairmanAgnostic Aug 2016 #11
Fair enough. Agschmid Aug 2016 #16
Icarus Airways does. 1890s BC. JHB Aug 2016 #9
Lol. Agschmid Aug 2016 #15
Easy to have a shutdown DustyJoe Aug 2016 #4
Power switch zipplewrath Aug 2016 #6
Almost every large company has what they call a Business Continuity Plan Massacure Aug 2016 #5
Probably something silly... Suburban Warrior Aug 2016 #7
HAHAHA trotsky Aug 2016 #13
*snort* smirkymonkey Aug 2016 #18
Had to be a hack. sarcasmo Aug 2016 #8
Apparently no, but cyberattack was certainly my first wonder Hortensis Aug 2016 #10
They most likely run the data center themselves. MohRokTah Aug 2016 #12
Delta probably ran some sort of risk/cost analysis MineralMan Aug 2016 #14
here is what I heard Skittles Aug 2016 #17
DELTA = Doesn't Ever Leave The Airport davidn3600 Aug 2016 #19
The news of Clinton leading in Georgia caused a meltdown... Phentex Aug 2016 #20

DustyJoe

(849 posts)
4. Easy to have a shutdown
Tue Aug 9, 2016, 11:20 AM
Aug 2016

I was IT director for one of the largest commuter airlines for 12 years with hubs coast to coast. The HQ where the primary server was located had redundant power, redundant data storage, even a redundant server,but the bottleneck was the data carrier. One time a fiber cut 150 miles away cut the data lifeline and a days outage resulted. The data carrier had no redundant links to the server. Nowdays the use of satellite, fiber, landline, wireless makes that type of loss rare, but in the 80's-90's wasn't available. You can have a backup generator in place, but if your data carrier does not, you are looking at your adequately powered system displaying 'no network access'. Murphy is alive and well in the IT world.

zipplewrath

(16,646 posts)
6. Power switch
Tue Aug 9, 2016, 12:00 PM
Aug 2016

The press reported once that it was a "power switch" that was being tested and failed. If it was the "power switch" that controlled where the power came from in the first place (including any alternates/backups) that could easily have been an unrealized single point failure.

I guess I was a tad surprised that in this day of "clouds" that the systems aren't a big more distributed than this. But I guess that just goes with the fact that the systems are old and resistant to change.

Massacure

(7,498 posts)
5. Almost every large company has what they call a Business Continuity Plan
Tue Aug 9, 2016, 11:52 AM
Aug 2016

I work in IT. At my previous employment, a company with about four billion dollars in revenue, one of my lunch buddies was the data center supervisor and so I got a little bit of insight into this company's Business Continuity Plan from an IT perspective.

This company had two data centers. The primary one was in the basement of the corporate office. The secondary one was located in a building they leased several miles away. This particular company held disaster recovery exercises about every six months. It didn't matter how bumpy the exercises went, the Chief Information Officer always declared them a success much to the chagrin of the people actually executing it.

There was a thunderstorm one summer that knocked out power to a good portion of the city. The data centers have what is called a UPS, an Uninterruptible Power Supply. It's basically a huge set of batteries that automatically power the data center for a period of time after grid power is lost and while the generator is coming online. During this thunderstorm a switch of some sort in that UPS failed, and the data center went dark. It happened so quickly they couldn't even cut over to the secondary data center. The company I was working for at the time was pretty pissed off at the vendor responsible for maintaining the UPS equipment.

I cannot speak to what happened in Delta's case. But to answer your question, Delta probably did have a backup system of some sort, and their means of cutting over to it probably failed just as spectacularly as their primary systems. It's unfortunate, but it happens.

 

MohRokTah

(15,429 posts)
12. They most likely run the data center themselves.
Tue Aug 9, 2016, 01:47 PM
Aug 2016

Take it from me, almost no company is capable of running a full fledged high availability data center themselves. Most companies, Delta included, would be better off outsourcing to several hosting organizations strategically located globally. You can insource the actual management of the systems and software, but you outsource the hosting and hands on requirements. This minimizes risk due to power failure etc. because the hosting facility hosts so many customers they can afford state of the art generators and backup power systems for the entire facility.

MineralMan

(146,192 posts)
14. Delta probably ran some sort of risk/cost analysis
Tue Aug 9, 2016, 01:54 PM
Aug 2016

on the impact of a catastrophic data outage like this. When they weighed the cost of enough redundancy to prevent it, though, they calculated that it would cost them less to just eat the loss when it happened. Corporations are always doing stuff like that to justify taking chances on things. So, how much did this crash cost Delta? I'm betting it cost less than setting up and maintaining a fully redundant system.

That's my guess.

Skittles

(152,967 posts)
17. here is what I heard
Tue Aug 9, 2016, 08:28 PM
Aug 2016

"Supposedly one of the 59 older Cisco servers had a "soft" hardware failure, which did not send a fail message. Eventually they took individual servers off line one at a time and found the problem."

Latest Discussions»General Discussion»Delta Meltdown