When the Systems Go Down: Continuity Planning Lessons From Sea-Tac’s Whiteboards

When the Port of Seattle’s systems went down last month, airport workers wrote flight information on dry-erase boards and airlines routed bags with pen and paper. That is business continuity at its most basic: a way to keep serving people while the systems you depend on are unavailable. This post walks through how to build that kind of fallback for a manufacturer, distributor, medical office or professional services firm with 20 to 800 employees.

Two recent disruptions, two different causes

On September 13, The Record reported that the Port of Seattle would not pay the Rhysida ransomware gang after an August attack. The attack encrypted some systems and disrupted baggage handling, check-in kiosks and ticketing at Seattle-Tacoma International Airport. Manual workarounds kept operations going, though the port warned that stolen data could be leaked. Executive Director Steve Metruck said paying “would not reflect Port values or our pledge to be a good steward of taxpayer dollars.”

The second disruption had nothing to do with criminals. After the July CrowdStrike outage, Microsoft hosted a summit on Sept. 10 with endpoint security vendors and government officials. On September 12 it published the key themes, including safer ways to deploy updates. Microsoft’s David Weston wrote that the incident “underscored the responsibility security vendors have to drive both resiliency and agile, adaptive protection.”

A continuity plan that only imagines storms and hackers will miss the familiar vendor whose update takes every laptop offline at once. Your plan should list the vendors whose failure would stop work, and say what you would do in the first day without each of them.

Six steps to a plan that works

  1. Rank what has to keep running. List the functions that bring in revenue or meet obligations: taking orders, shipping, billing, patient scheduling, payroll. For each one, estimate how long you could go without it before customers, patients or regulators feel it.
  2. Write the manual fallback in advance. For each critical function, decide how work continues with the system down: printed order forms, a phone tree, a paper log, a spare card terminal. Staff shouldn’t have to invent workarounds under pressure.
  3. Assign roles, with backups. Name who declares a disruption, who talks to customers, who works with IT and vendors on recovery, and who can approve emergency spending. Give each role a second person, because the first may be on vacation or locked out of the same systems.
  4. Protect what you’ll need to restart. Keep automated backups with an offsite copy, a spare for any single point of failure such as an internet connection, and contact details for critical vendors stored somewhere other than the systems that might fail.
  5. Practice. Once or twice a year, walk managers through a scenario: ransomware on the file server, a bad update on every computer, a building without power. Gaps found in a conference room cost far less than gaps found during an outage.
  6. Revise after every exercise and every real event. Ask staff, and customers where it makes sense, what broke and what held up, then change the plan.

Bring department heads in early

The people who run each area know which workarounds are realistic and which only look good on paper. Their input makes the plan usable, and taking part means they know it exists and where to find it. Share every revision with them, not only the first version.

How EUS can help

EUS assesses your security posture and recovery readiness against the requirements that apply to your business, then helps close the gaps with controls and system updates, including backups and network connections. We also coordinate incident response and can help you test your plan with your team. If you’d like to walk through your current plan with us, we’re glad to.

Sources