There have been several different fires in various data centers in recent times. These fires include the Los Angeles-based web hosting provider, WebNX, and an OVHcloud data center in Strasbourg, France. While data center and cloud providers have well-thought-out systems in place, because of the recent incidents many people are now questioning how reliable these particular systems are. These incidents can be caused by many different factors, and this needs to be addressed and discussed. What can we learn from these disasters?
Data centers are well-oiled machines that are used by almost all modern industries, but just like anything else, things can happen. It’s how we learn from the situations and incidents that help people, businesses, and industries altogether grow. Although the exact details of these incidents haven’t been reported just yet; there are several takeaways we can learn from.
While data centers are run with the very best technology available and best practices in mind, data center fires aren’t always completely avoidable. Data center fires have long been a risk and a part of using any type of data center service. The Uptime Institute has stated there have been 25 data fires in recent history, which includes 11 data center fires that were not publicly reported. These recent incidents should make users aware of the risk that goes into using every type of data center service. Including the one, you are already currently using. This even applies to Google Drive, Amazon Web Services, and iCloud. Data storage systems always come with some risk. Knowing this is important, but finding a trusted data center provider can help you and your business navigate through any of these disasters.
There are many aspects of a data center that have the potential of causing a fire. This includes the digital equipment, cables, raised floors, wires, HVAC equipment, and suspended ceilings, and more. Because investigations and outage reports can take time, the details of the OVHcloud disaster will not be disclosed until next year. Nonetheless, there are several theories about what caused the fire.
It’s reported that extensive work had been done on at least one of the backup power or UPS (Uninterruptible Power Supply) earlier that day. It could have been a maintenance issue or a configuration issue. Since the fire, some people are now questioning the tower design with convection-cooling-based auto-ventilation being used. As cool air comes in, it passes through a heat exchange for the direct liquid cooling system, the warm air then rises through the tower into inter of the building. Some are believing this could form a chimney-like situation. But the leading theory of what caused the fire is that OVHcloud didn’t have a VESDA system (very early smoke detection apparatus) or a fire suppression system.
Fire suppression systems are designed to detect a fire as soon as it starts and suppresses it immediately. The goal is to prevent the fire from spreading to other areas. If the system detects heat and flames, the system is set off and it begins the process of suppressing the fire.
There are several different types of fire suppression systems and these various systems can vary completely from each other. All of these different systems fall under two main categories. This includes engineered fire suppression systems and pre-engineered fire suppression systems.
Engineered fire suppression systems work by flooding the room with what is called a “clean agent”, which is made up of gases that can suppress the fire without harming the equipment, and more importantly, the people working in the data center. Some of these engineered systems’ clean agents are stored as a liquid but are released as gas to not damage the equipment. A pre-engineered fire suppression system doesn’t flood the whole room with a clean agent like engineered systems do. Pre-engineered fire suppression systems protect smaller areas like engine compartments, electrical panels, and even CNC and EDM machines.
There are also different methods of fire detection which include an active detection system and a non-electric fire detection system. Active detection constantly and continuously monitors the location for heat and/or smoke. This method works well at sensing fires and automatically triggers the fire suppression system. The downside of this method is it needs electrical power. This means if the power source being used goes down—the system also goes down with it. The other process is called a non-electric fire detection system, which as the name states, doesn’t need electricity to function. Instead, fire detection tubes are installed. As soon as the tubes are introduced to heat and/or fire, the tube bursts releasing the clean agent to suppress the fire.
All of these various methods of fire detection and types of fire suppression systems can be beneficial for all data center operations. All of these various systems have an end goal of stopping the fire as promptly as possible, so it doesn’t grow and affect other areas.
One of the major takeaways regarding these data center fires should be how the industry as a whole handle these situations. There have been more, much needed, discussions about data center fires because of the recent complete loss of two data centers. These data center disasters only make the news when the fire is reported. The data center industry needs to be more transparent about these instances and the risk that every data center and cloud providers have. Even the highest-rated tier (Tier 4), which is required to include the highest level of redundant power, hardware, fault-tolerant components, and cooling still has fire as one of the two single-points-of-failure that are allowed to cause downtime. The only other one is an emergency power off.
The risk of fire has always been there. Only time will tell if the industry takes the proper steps to help spread this awareness. What we do know is data centers need scheduled and proper maintenance along with a good data center management team. Also, fire suppression systems are crucial to data center operations. This includes a VESDA system and a water or gas fire suppression system. These incidents may or may not have been avoidable, but data center fire suppression systems could potentially help save an entire data center from facing the same fate as the two recent disasters.