The statistics and fears associated with ransomware receive a lot of attention from the press and in the media alike. However, to fully appreciate the devastating impact that a ransomware attack may have on an organization, it helps to speak to someone who survived one. Recently, I had that opportunity. That conversation provided a sobering look into the speed of ransomware attacks and their increasingly insidious nature.
I spoke to the COO at a large professional services firm based in the southeastern United States. He came up through the IT ranks at this firm and has been with them for 20+ years. As such, he had a solid grasp on his companyâ€™s IT environment to include its backup strategy and disaster preparedness.
The company runs the Nutanix Enterprise Cloud platform and hosts most of its applications on Nutanix, using AHV. The professional services firm handles many medical cases making those stored records subject to HIPAA regulations. It supports more than 50 servers, 10+ million files, and 30+ TB of data.
To protect these data and applications, the professional services firm takes a two-fold approach. On the front end, it uses cybersecurity software to detect and prevent ransomware attacks. For backup it uses HYCU for Nutanix. HYCU backs up the firmâ€™s VMs by doing daily incremental backups and full weekly backups.
The firm also replicates all its data to a second data center located more than 100 miles away. This architecture ensures the firm may quickly failover and recover both locally and remotely.
The COO thought that he was well-prepared to respond to any natural disaster or attack, including a ransomware attack. Little did he realize how well ransomware could penetrate these defenses and even use these defense mechanisms against his environment.
The Warning Signs
The ransomware attack on the firm displayed few early warning signs until its detonation. Alerts first showed up in their logs the Friday before the attack started. The cybersecurity software the firm had in place showed signs of suspicious activity. However, none of these alerts were raised to the level that it caught anyoneâ€™s attention or prompted a response.
Later investigations would later reveal the root cause of the attack. A Word file attached to an email entered the firm at the end of the work week. Embedded in this Word file was a macro. This Word file, when opened, introduced the RYUK malware virus into the environment that started a two-pronged attack.
The first phase of the attack spread the malware throughout the companyâ€™s network by infesting file and Windows servers. This infestation planted an executable that would detonate during the second phase. Once planted, the original malware virus deleted itself. This made it difficult to later diagnose the source of the attack and helped hide the virusâ€™ origins and identity.
All Hell Breaks Loose â€¦ on a Sunday
The COOâ€™s world began to become undone around 11 am on a Sunday morning as the RYUK virus sprung to life. While no one was in the office it executed assuming the same security permissions as the user on each device. This gave the virus broad access to that systemâ€™s resources as well as the network.
The attack continued by first detecting and encrypting the network-attached and locally attached drives to which it had access. It then proceeded to encrypt all the files and data on the local server or PC. On each infected device, that ransomware only left one readable file: an HTML file. This file included a link to the hackerâ€™s email address that the victim could contact to negotiate a ransom payment.
No Good Path Forward
By the time the COOâ€™s pager went off, it already appeared too late. All the Windows PCs, laptops, servers and network files were encrypted. Aggravating the situation, the processes the firm had in place to replicate data to its DR site had already executed. These processes copied encrypted data to the firm’s DR site preventing the firm from performing a remote recovery.
Then, to add insult to injury, the ransomware encrypted all the backup files that were stored on shared network drives. Despite feeling increasingly hopeless, the COO began by exploring the only four paths he identified as available to him.
- He contacted the hackerâ€™s email address with his own anonymous email address to ascertain the requested ransom amount.
- He checked his backup tapes to determine what, if any, data he could recover.
- He engaged a contractor that claimed to help companies recover from ransomware by helping to decrypt their data.
- He contacted his backup provider, HYCU, to see if it could help his firm recover its data.
Paths to Nowhere
The feedback on the first three of these paths forward only confirmed his worst fears. The hackers responded to his email with a ransom request of 92 bitcoins (approximately $1M US dollars.) While the COO was almost prepared to pay something, $1M was not an option.
Recovering from the backup tapes offered some hope, but not much. The firm only kept about 60% of its data on tapes and it only moved data to tape monthly. Further, it would take weeks to recover the data from tape and reconstruct the applications. The COO viewed using tape as a source for recovery only as a last resort. Even then, he was unsure if a recovery from tape would work.
The response from the contractor provided no comfort and raised other red flags. Minimally the fee would be $10,000 to decrypt the data assuming the contractorâ€™s decryption software worked. However, further investigation revealed the contractor may well be a front for the hackers who encrypted his data. So, he dismissed this as an option.
At this point, the COO began to prepare his firm for what appeared to be the inevitable. A painstaking process to rebuild all the servers from scratch from the data they had on tape. His firm would also need to revert to paper-based processes for an indeterminable time.
HYCU: The First Glimmer of Hope
The first glimmer of hope that the COO saw came late Sunday when his team found an overlooked, unencrypted file on the HYCU VM. The ransomware had encrypted all the backups stored on the network filers. However, HYCU kept a single system-generated file residing on its host VM. Since HYCU runs on a Linux-based CentOS VM the ransomware did not infect or encrypt this file.
The COO and his team contacted HYCU shortly after detecting this unencrypted file on the HYCU VM. They wanted to find out if there was any way they could use this file for recovery.
A Hard Dayâ€™s Night
Once contacted, the HYCU support team prioritized the issue. The challenge was the file stored on the HYCU VM was not initially intended or formatted for recovery.
This file essentially functions as a cache for backup data. The HYCU backup software uses this to cache prior to creating the backup files stored on the network server. Though the ransomware did not encrypt this file, it was unclear initially if HYCU could access and use it for recovery.
As the HYCU support staff worked the issue, minutes turned into hours leading the COOâ€™s hopes to dim. In the background, HYCU passed the issue from support team to support team around the globe. In this way, fresh eyes were always working the problem. HYCU also pulled in members of its engineering group to diagnose the issue and work the problem.
By the time Monday came around, HYCUâ€™s collective global support team had diagnosed and resolved the issue. HYCU managed to unpack this file, access the data in it, and restore all the firmâ€™s VMs.
The COO could not begin to express how eternally grateful he was. HYCU had pulled the proverbial rabbit out of the hat and helped to recover his firmâ€™s data. He acknowledged how vital it was that HYCU had made that file available to him for recovery. Otherwise, it would have taken months for his firm to completely recover and rebuild its IT environment. While the firm would have survived, it would have been a significant setback. He could not even begin to put numbers to the firmâ€™s potential losses in terms of money and time.
HYCU could have stopped at any point and told him it could not retrieve the data from that file. However, the COO never saw the HYCU support team admit defeat. Rather, he witnessed their determination to work and live the problem with his team until it was resolved. By the time HYCU finished, the COO acknowledged, â€œIt was a miracle to have all our systems and data back.â€
The firm is now back up and running having learned many lessons from the experience. For instance, the COO no longer will only keep backups on network drives. He plans to always keep a backup copy offline that his firm can use for recovery should another ransomware attack occur.
He also displayed a sense of humor in the aftermath of this attack. During the days immediately following the attack, everyone had to help recover data and restore operations. The COO noted, â€œBy the time we finished recovering, even my personal assistant knew how to re-image a PC.â€
HYCU is a DCIG client. Both HYCU and the professional services firm had the opportunity to provide input on the content published in this blog entry for accuracy.
All information contained in this blog is correct to the best of DCIGâ€™s knowledge. Neither HYCU nor its client paid DCIG any fee to create or publish this blog entry. It is the collective hope of DCIG, HYCU, and the firm that readers of this content will become more aware of the threat that ransomware poses and take the appropriate actions to protect themselves from it.