Perhaps a catchy title for a post, but a truly challenging question to address and resolve; for this reason, I decided to write this post and share the experience gained over time.
Over the past two decades, the business sector has changed in almost a radical manner. Among all the changes we have experienced as a society, none is as fundamentally profound as the dependence on information technology systems in supporting critical business infrastructure. The performance of data centers has become a monetizable commodity for all major sectors such as banking, retail, aerospace, telecommunications, internet service providers, and others.
This unprecedented dependence on IT systems has resulted in an even stronger connection between data center availability and business alignment for companies. Just a downtime event in a data center signifies a significant and direct impact on both profitability and business viability.
The primary function of data centers is to provide constant uptime for mission-critical applications they host. However, interruptions in service delivery can occur unexpectedly, and operators must be proactive in finding ways to prevent and correct them. Understanding the causes of data center interruptions, quantifying the losses incurred by companies, and finding ways to prevent them are crucial to avoid business interruption turning into million-dollar losses, customer losses, and damage to corporate reputation.
Real cases of interruptions in data centers and their enormous economic cost to the companies involved.
British Airways, estimated loss of approximately $200 million, over 3 days.
On May 27, 2017, British Airways (BA) experienced a data center interruption that forced the cancellation of more than 400 flights, stranding over 75,000 passengers.
A BA spokesperson told the UK publication IT PRO: "There was a loss of power in the UK data center, which was compounded by the uncontrolled return of power, causing a power surge that knocked out our IT systems. So we know what happened; we just have to figure out why. It was not an IT failure and had nothing to do with IT outsourcing; it was an electrical power source interruption." Eighteen months later, BA decided to sue the property specialist CBRE, who was the operator of the data center, for the damages caused during the interruption.
A BA spokesperson told the UK publication IT PRO: "There was a power loss in the UK Data Center, exacerbated by the uncontrolled return of power, which caused a power surge that knocked out our IT systems. So we know what happened; we just have to find out why. It was not an IT failure and had nothing to do with IT outsourcing; it was an electrical power source interruption." Eighteen months later, BA decided to sue the property specialist CBRE, who was the operator of the Data Center, for the damages caused during the interruption.
Delta Airlines, estimated loss of approximately $150 million, over 5 days
Delta Airlines has experienced several incidents in its data centers, with the first major one resulting in the cancellation of over 2,000 flights over 5 days in August 2016, with losses of approximately $150 million and more than 250,000 passengers stranded at airports.
An electrical component at their Atlanta headquarters failed, leading to the shutdown of the transformer providing power to the airline's data center.
BlackBerry, estimated loss of approximately $350 million, over 4 days.
The interruption extended over four days and five continents, affecting over half of its 70 million users worldwide and leaving millions of critical emails for its customers unsent. Loss estimates ranged from a single UK estimate of only $26 million to a retrospective global estimate of $350 million in total. For BlackBerry, it meant a massive loss of corporate and mobile phone users, in addition to the company's overall setback. It was a lucky break for competitors like Apple, Google, and Samsung.
The estimated cost of infrastructure vulnerability.
According to the Electric Power Research Institute (EPRI), 98% of power outages last less than 10 seconds. However, imagine how much it can cost companies if their mission-critical applications fail for just 10 seconds. The 2016 Ponemon Institute survey showed that the overall average cost of a data center outage is $750,500.
According to studies by the Uptime Institute, the three primary causes of interruptions in Data Centers are as follows:
1. Failure in Uninterruptible Power Systems (UPS)
2. Cybercrime (DDoS)
3. Human error
These three are responsible for over 50% of all interruptions in Data Centers
The monetary costs of failures in these three primary causes are compared in the following table, expressed in $1,000:
Types of costs associated with Data Center interruptions.
Within Data Centers, the costs associated with service interruptions can encompass a wide range of variables. The most common cost centers and activities can be grouped into the following categories:
Detection Cost: Activities associated with the initial discovery and subsequent investigation of the partial or complete interruption incident.
Containment Cost: Activities and associated costs that reasonably prevent the interruption from spreading, worsening, or causing further disruption.
Recovery Cost: Activities and costs related to bringing the company's core networks and systems back to an available state
Response Cost: All incidental costs following the interruption incident and business recovery.
Equipment Cost: The cost of purchasing and repairing new equipment, including refurbishment.
IT Productivity Loss Cost: Lost time and related expenses associated with IT personnel.
User Productivity Loss Cost: Lost time and related expenses associated with end-users.
Third-Party Cost: The cost of contractors, consultants, auditors, and other specialists engaged to assist in resolving unplanned interruptions.
Revenue Loss Cost: Total loss of income from customers and potential clients due to their inability to access core systems during the interruption period.
Business Interruption Cost (Consequences): Total economic loss from the interruption, including damage to reputation, customer loss, and missed business opportunities. .
Legal Process Cost: Legal processes and lawsuits associated with service delivery contract breaches.
DCIM as an Ally in Service Delivery.
Data Center Infrastructure Management (DCIM) software allows Data Center administrators to prevent unplanned downtime, which can cost hundreds of thousands of dollars per interruption and wreak havoc on their business. Some of the ways to avoid human errors and maximize uptime with DCIM include:
· Temperature and humidity management.
· Ensuring power redundancy.
· Maintenance activity planning.
· Clean room management.
· "What If" scenario analysis for interruption scenarios.
· Health surveys for facilities and IT equipment.
· Real-time monitoring of facilities and IT equipmenten.
· Data collection for post-mortem analysis.
Conclusion
Data Center infrastructure is critical to business operations today. Interruptions in Data Centers have already caused million-dollar losses to various companies in different sectors of the economy. The various factors that interact within the ecosystem of data management and storage, and the critical nature of their proper management, make it essential to use the right tools to ensure timely service delivery and maintain vital business operations.
