At around the time that a series of earthquakes hit the Tiberias area a few weeks ago, the State Comptroller’s Office released a report on earthquake preparedness. The report addressed significant lapses in planning for disaster recovery for schools, hospitals and the distribution systems for water and natural gas. According to the report, the lack of preparedness puts thousands of lives at risk.
What about Israeli businesses? How prepared are they to cope with a disaster? Would ATMs continue to supply cash? Would the stock market continue to operate? Would Tnuva’s fleet continue to deliver dairy products the morning after the “big one?”
>> Opinion: Coming soon: A devastating earthquake
All experts agree that Israel lags behind the rest of the Western world when it comes to information technology business continuity, the ability of businesses to continue operating after extreme disruptions to their computer systems.
Ronnie Sadeh, the CEO of MedOne, which operates Israel’s biggest data center, said, “The company is now dealing with business and functional continuity in the context of the earthquake report, but that’s just one of the threats. When you talk about extreme situations throughout the world, you are talking about natural disasters — wildfires, earthquakes, storms, volcanic eruptions — and widespread cyberattack. In Israel there is also the threat of missiles and terror attacks, proximity to the Syrian-African fault and the possibility of a mass call-up for reserve duty.”
- Spate of earthquakes leaves Tiberias feeling vulnerable
- Israeli hotels look to vamp up emergency plans for future earthquakes
- Which would cause more deaths in Israel, a missile attack or an earthquake?
According to Sadeh, Israeli business isn’t really ready for an extreme event of this nature. “There’s a big gap between the need and what companies actually do. It’s not just a matter of technology. You have to build what’s called a BCP, a business continuity plan. It’s a what-if exercise, a map of what assets you want to preserve in the event of a disaster and a contingency plan that explains how to make that happen. If I have a call center with 100 employees and one night the building stops functioning for one reason or another, what do I do? Maybe the decision is to keep 10 employees at a different site? Or maybe we have to find a different solution?”
There have already been instances in which large Israeli businesses couldn’t supply services because of disasters or malfunctions. In November 2008, for example, Bank Hapoalim suffered a shutdown of its computer system for two days, causing the bank millions of shekels in damage. It was reported at the time that the entire computer system had gone down, including its ATMs, its online systems and all the systems in its branches. For two days branch staffers had to conduct transactions by hand. The bank canceled fees for customers who tried to conduct transactions during the shutdown.
Cellular operators have suffered periodic service shutdowns. In December 2010, Cellcom’s system was down for an entire day. The event cost the company 66 million shekels ($18.6 million at the time), and led to several class action against it. Pelephone has had two similar shutdowns, one of which was caused by a fire in the company’s power room. Maccabi Health Services a few months ago had half a day without functioning computers. And these are just the incidents that made the news.
There are similar stories from all over the world. In May 2017, British Airways suffered a major technical malfunction that forced it to cancel all its flights from London and caused chaos at the airport; in 2013 the computers crashed at The New York Times and in 2017 Microsoft’s Outlook email system was shut down for a time. Recently the Taiwanese chip maker TSMC was attacked by a computer virus that seriously disabled some of the company’s plants. Five days after the incident, the company had still only managed to restore 80% of its activity. The company suffered an estimated $180 million in damage.
There is an international standard for preparedness for extreme situations, ISO 22301, that is recognized by the Israel Standards Institution. But only 24 Israeli firms have adopted it so far, and none of them are on the TA-35 index of the 35 largest companies on the Tel Aviv Stock Exchange’s TA-125 index.
Alon Rozen, previously the director general of the now-defunct Home Front Defense Ministry, is the CEO of Elements Homeland Security, a consulting and project management firm in the field of business continuity.
“Israel is to a great extent a microcosm of the world in terms of all the possible threats — tsunami, earthquakes, terror — but it’s also an economy characterized by collective bargaining agreements and the threat of strikes,” Rosen says. “Still, unfortunately the state has built its emergency systems in a patchwork fashion and not as a unified whole. The business sector doesn’t even understand which threats to prepare for, for the simple reason that we haven’t suffered an earthquake in the modern era, and since 1973 we haven’t experienced a war that was nationwide in scope.
“The slow adoption of ISO 22301 shows the depth of the lack of understanding in the market, and the only companies that are relating seriously to business continuity are either companies that are subject to very strict regulation or international companies like Coca-Cola and Microsoft.” Of all the Israeli banks, only Union Bank (Igud) meets the standard.
Duplicating your computer room
Ofer Gadish, CEO of the business continuity startup CloudEndure, says that “In Israel they belittle the issue of business continuity and many times they get confused; they do a backup and they call that DR [disaster recovery]. If you have pictures of your children on your home computer and you want to back them up, you take an external hard disk, copy them and put it in a safe. That’s data backup, but that’s not DR. Real DR means having an additional computer next to you, with the data on both of them identical.”
In the business world, creating disaster recovery solutions requires a substantial investment in infrastructure. A corporation that wants to be prepared for a disaster, malfunction or cyberattack must take its entire computer room and duplicate it, including all the servers, organizational software and so on. This room must be located in a different facility from the company headquarters and it must be maintained — you have to pay for electricity, for duplicate software licenses and so on. When a server is replaced at one location, it must be replaced at the second location as well. And all this is for servers that will operate only in an emergency.
According to Gadish, one computer room could cost tens of millions of dollars. There’s also a cultural problem; even at firms that have a disaster recovery site, the information technology manager is often reluctant to test it, lest the company’s routine work get interrupted. “Those responsible are afraid to do transition testing,” he says. Gadish argues that a good disaster recovery solution also includes a response to a cyberattack. “A good solution allows one to go back in time to the point before the incident, which takes the sting out of ransomware attacks that lock the organization’s computers.”
Gadish’s arguments are backed up by data. In July 2016, the Finance Ministr’s Capital Markets, Insurance and Savings Division issued an auditor’s report on business continuity planning in the wake of exercises on the issue conducted by institutional investors and insurance companies.
“In some of the institutional investors we found that no examination had been conducted in recent years of the feasibility of restoring information from the hard backup copies,” the report said. “In some of the institutional bodies there had been no survey of critical infrastructures against the reference scenarios, like the ability of the structures and the water and electricity infrastructures to withstand an earthquake.”
Regarding cyberthreats, the report said, “In some of the institutional bodies, the procedures for coping with cyber events involve merely disconnecting the information systems, and do not specify any additional actions to cope with such events.” This is the situation in entities that are subject to regulation on this issue and that examine themselves.
In fact, the insurance sector is the only one in Israel that is required by regulations to maintain a third copy — that is, a third computer room. There is no such requirement in any other sector, not even the banking sector. The relevant regulation for insurance companies reads, “An institutional body must back up its data with another copy, to assure the recovery of information in the event that the information is undermined at the main site and at the alternate site at the same time.”
The law doesn’t require it, but companies can maintain the third copy in a different country, so that it could serve as a backup in the event of a major catastrophe in Israel. At least two insurance companies already keep copies of their computer rooms abroad; Phoenix, which maintains a disaster recover site in Cyprus, with a direct link via undersea cable, and Clal Insurance, which maintains a third site in Greece linked by fiber-optic cable. According to the Insurance Supervisor, there are insurance companies that don’t maintain a third copy at all, despite the instruction.
Hard backup or the cloud?
Until recently, Sadeh of MedOne and Gadish of CloudEndure were rivals, offering two competing products. Sadeh was selling “floor space” for physical backup, while Gadish sold a cloud-based software solution for disaster recovery. In Sadeh’s case, large companies would set up their second-copy computer room in his server farm underground, with a double electrical feed. This is his primary business.
The solution offered by Gadish is less expensive and focuses on backing up data in the cloud.
“We sell modern disaster recovery solutions,” he says. “We know how to transfer all organizational applications to any public cloud infrastructure — Amazon, Azure (Microsoft’s cloud services platform) and so on. We routinely duplicate the data into the cloud all the time, and in real time we supply recovery time of two-three minutes. This solution is 50% to 80% cheaper than traditional DR. The players in the public cloud have lots of unused computing power, and for companies this is an ideal solution.”
The two companies recently began a cooperative venture, in which Gadish’s CloudEndure provides its recovery solution from disasters, malfunctions and attacks on MedOne’s own public cloud servers. According to Gadish, “We chose the most durable site in the country that can support the most customers.”