AT&T is the largest telephony company in the USA. In 1990, a single switch failure at one of AT&T’s 114 switch centres failed. It reset itself and set an automated message to the other centres which caused them to fail, shutdown and reboot.
Despite reports of hackers during the incident, it was reported that the error lay in one line of code that had been added during a software upgrade.
Whatever the cause, the impact on customers were that 75million phone calls were lost. It was further claimed that American Airlines estimated they lost 200,000 reservations as a result.
What can business continuity planners learn from the collapse?
- Simple errors can cause massive incidents. Don’t assume that something obvious, intentional or significant has to cause a major incident. Seemingly innocuous errors can cause problems months later, and be hard to identify during an incident.
- Don’t assume the cause is recent but do ask “what’s changed”? Changes made some time ago can suddenly become problems, but it’s still sensible to ask “what’s changed” and investigate that first. We’ve come across cases where the incident manager is told of a massive recent change but told, “but it’s not possible for X to have caused this outage” causing delays as X wasn’t immediately investigated even though it was, in fact, the cause.
- Scenario-based business continuity exercises are useful to project managers. As well as the usual benefits, it can help projects imagine or practice with the realities of their work before it is rolled out.
- You may not always be able to identify the cause during an incident. This can make resolving the problem very difficult. Some organisations have to suffer a particular circumstance several times before they are able to connect the dots tot the source. Consider conducting rigorous reviews for all significant incidents, not just the ones that impact the customer in the name of prevention.
Was this article useful? Share it below, or comment to join the conversation.
Subscribe - weekly news and a free course!