FAA ground stop cause: old routers choked during an upgrade

The failure happened in the network that moves information between FAA facilities, not on the planes or at the airports. At 3:36 p.m. Eastern time, Duffy said, the agency detected instability in data feeding its high-altitude control centers and its TRACONs, the radar rooms that guide planes into and out of busy metro areas. Controllers started seeing radar and communication data that didn’t match.

The FAA never lost radar or radio completely, Duffy said. Even so, at 3:54 p.m. ET it ordered every departing flight in the country to stay put, citing safety and extraordinary circumstances. The order was lifted at 8:15 p.m. ET.

Several reports linked the trouble to the FAA Telecommunications Infrastructure, known as FTI, the private network that ties the air traffic control system together. Duffy described the piece that failed as a legacy system. Part of it still uses TDM links, an older style of telecom connection designed for analog-era phone lines rather than modern internet-style data.

Homeland Security Secretary Markwayne Mullin gave the most specific account. He said the problems surfaced while crews were adding patches and installing new equipment, and that the newer technology swamped the older network it was connected to. By Mullin’s account, then, the new gear being added overloaded the old network, and his takeaway was that patchwork fixes don’t work.

Officials say they see no sign of one, but they haven’t closed the door. Mullin said there was no indication of anything nefarious and that no lives or flights were ever at risk. He also said his department was still checking and couldn’t be completely certain yet.

Two other suspects were cleared quickly. Microsoft said it had no indication the grounding was tied to the outage that knocked Outlook and other services offline the day before. Air Space Intelligence, the AI startup that supplies the FAA’s new SMART air traffic software, said its product played no part, and Duffy said SMART is completely separate from the equipment that broke.

The National Weather Service also spoke up after false claims spread online that its radar was down too. It said its radar and website were working normally.

Nationwide ground stops are rare. The last one came in January 2023, when a key FAA computer system broke down. Here is the scale of Friday’s shutdown:

• Duffy said over 3,500 planes were in the air when the stop began; tracking service Flightradar24 counted 4,042 flights bound for US destinations airborne at that moment.
• Cancellation totals differ by tracker and time. FlightAware showed more than 6,300 flights canceled and 9,500 delayed as the stop ended, while another count passed 7,000 cancellations.
• A Dartmouth aviation logistics professor estimated 300,000 to 400,000 passengers were stranded in the first hours.
• Europe’s air traffic agency held US-bound departures from 5 p.m. Eastern until midnight, and Air Canada scrapped all of its US flights for the rest of Friday evening.
• Flights inside Alaska and between Hawaii’s islands kept running, and some international departures leaving US airspace were cleared.

Freight and mail flights were hit as well. Getting back to normal is slow because aircraft and crews ended up in the wrong cities, and some staff ran into on-duty shift limits.

The FAA is partway through a multiyear rebuild of FTI that replaces decades-old copper wiring with fiber-optic cable. The agency says the new network will mean fewer outages and fewer outage-related delays, and will cope better with newer aircraft like drones and air taxis.

Government contractor L3Harris Technologies runs the legacy network until the new one is ready. L3Harris said in May that it was halfway through its modernization work, and earlier this month said seven in ten FAA sites now run on fiber. The FAA’s own count says 69% of copper wires have been replaced. L3Harris didn’t respond to a request for comment after the outage.

In August the FAA chose AT&T to build an entirely new version of FTI, taking over a role that Verizon had earlier been picked for. That contract starts at $73 million and is expected to grow into the billions over the years.

Other pieces are moving too. This month the FAA began retiring the floppy disks some control facilities still use to load weather and other key information. The replacement display system, called E-IDS, takes over from four systems dating back to the 1980s and is due at 435 sites by the end of 2028. That’s also the year the administration has set for finishing the wider overhaul.

Watchdogs have doubts about the pace. A September Government Accountability Office report warned that the rush might favor speed over cost control and smooth operations. It also found the FAA juggling 11,389 separate project schedules rather than one master plan.

Congress approved $12.5 billion last year to upgrade air traffic control. Hours before the ground stop, Duffy sent congressional leaders a letter asking for $30.1 billion more. By Friday night he had raised the request to $50 billion, and Mullin said that’s roughly what a complete overhaul would cost.

House Minority Leader Hakeem Jeffries called the day an unprecedented breakdown and blamed the administration’s leadership. Duffy, for his part, blamed Congress for not funding the upgrades, though he didn’t directly answer whether Republicans, who control both chambers, share that blame.

Because an FAA ground stop counts as outside an airline’s control, like bad weather, your rights are narrower than many travelers expect. Under US Department of Transportation rules:

• If the airline canceled your flight and you decide not to travel, you’re owed a refund, even on a nonrefundable ticket. You don’t have to take a voucher.
• If a domestic flight was delayed at least three hours (six for international) and you choose not to fly, you can also claim a refund.
• Refunds go back to your original payment method, generally within seven business days for credit cards and 20 calendar days for other payment types.
• Airlines don’t have to cover hotels, meals or cash compensation for this kind of disruption, though agents can hand out vouchers case by case.

Airlines have also posted fee waivers:

• Delta: flights on Nov. 13 and 14 can be canceled for credit or rebooked for travel before Nov. 21.
• United: flights on those same two days can be refunded in full or changed free for travel by Nov. 20.
• Southwest: trips booked for Nov. 13 through Nov. 16 can be rebooked without fees or flown standby within 5 days of the original departure.
• American: flights to, from or through select cities on Nov. 13 and 14 can be changed without fees for travel by Nov. 17.
• JetBlue: Nov. 13 flights can be changed without fees for travel by Nov. 18, and canceled flights qualify for refunds.

One practical tip: if your flight in the next few days hasn’t been canceled, don’t cancel it yourself. You’d probably get only a credit, while an airline cancellation entitles you to your money back. Rebook in the airline’s app instead of waiting in the gate line. Duffy said he expects flights to be back to normal before the Thanksgiving rush.