Disgruntled Former Employees Take Revenge

How it Started

It all started on a Friday morning when employees of a major staffing agency were locked out of an app on their mobile devices that allowed them to clock in and out. This app would track the hours they worked, allowing employees to get paid. The agency would also know how many hours to bill their customers. It’s a simple and paperless solution that works very well…when the technology cooperates.

Because the company was well-established, they also had a paper backup system in place ready to go just for scenarios like this one. However, this system was neither convenient for the hundreds of individuals involved nor guaranteed to be in compliance with different state laws regarding breaks, overtime, etc.

 

The Vendor Refuses to Help

With their client temporarily “fixing” the issue by switching to a paper backup system, the 1Path team started analyzing what had happened in order to fix the problem. Their credentials weren’t working and their first call was to the vendor of the app. They were not helpful. 1Path owned all the licenses for this application, and the vendor essentially said (with a straight face) that they couldn’t be sure that 1Path hadn’t themselves made the changes responsible for the problems affecting their client.

What had happened was that all user profiles had been deleted and only one (newly created) administrator account remained, [email protected]. Given the situation, Armon and Patrick were flabbergasted at the vendor’s attitude. (Especially given the name of the new administrator account, which would indicate an author with bad intent.) While 1Path was trying to solve a problem for one of their own end-users; they also very much considered themselves as an end-user in relation to their vendor. 1Path was definitely not feeling the love, especially given all the business they had done with this vendor over the years.

Given this temporary dead end, 1Path continued to follow its normal policies and procedures for crisis management. They determined that no current, or former, 1Path employee had access to the environment. This allowed them to create a roadmap to help them get the client up and running again.

While 1Path started the process of getting new, properly provisioned, mobile devices shipped out to hundreds of employees all around the country, they were still working with their vendor to find a faster solution.

The vendor finally agreed that they would be willing to release some information if:

  • An email was sent from a personal email address (not a company one)
  • With a signed letter from the CFO of the company

1Path waited for three days then called to follow up. It turned out that there was something in the letter that the vendor didn’t like and they hadn’t bothered to call to tell 1path what it was. 1Path jumped through some hoops to satisfy the vendor and finally got access to their account. They then found emails in the archive from years ago, when the client was first created, and with that, the vendor was finally able to reset 1Path’s account.

 

The Culprits

It turns out that the entire attack was orchestrated by two former employees of the client in question. Some time prior, 1Path had exposed the fact that these employees were spying on the email of the CEO, this had led to their dismissal. They then decided to get their revenge on 1Path and cost their former employer a lot of time and money, and their former colleagues a lot of hassle.

They had probably retained access to, or had a copy of, an administrative password that hadn’t been changed and didn’t require multi-factor authentication.

 

Lessons Learned

The first lesson that 1Path learned was directly related to this security hole. As a result, they implemented a forced password policy (making a user change the password upon login) combined with multi-factor authentication. They didn’t just do this with the client who had suffered the attack, they rolled it out across all their clients.

The password policy change was part of a larger conversation regarding collaboration with customers. Thus explaining the desire to be true partners, which meant abiding by standards that everyone could see would lead to better security. The conversation was also framed in the context that the instigators of the attack seemed to have a collaborative relationship, but “the good guys” didn’t have any collaborative plans in place.

1Path also learned that despite having a “relationship” with their vendor, when the chips were down, that vendor couldn’t be relied upon. The vendor had long contracts written by expensive lawyers ensuring that their liability, in situations like the one above, was almost nonexistent. That has led to 1Path building systems with more backups and redundancies, to make sure that they don’t ever have to rely on a vendor again the way they did in this particular case.

As we said at the beginning of this article, no one expects an IT provider to be perfect. Customers know that at some point there will be a crisis, and it will be in that crisis that a partnership will be tested. 1Path came out of this crisis stronger. They realized where they could improve and adjusted their expectations and policies accordingly.

If the worst happens, you’re going to want the best financial, legal, and technical support to get you back up and running again. With cyber insurance from Datastream, we find the most comprehensive insurance coverage on the market alongside critical post-incident customer support.

 

Click here to learn more about how we can help secure your business data!

www.thecybercrimelab.com

Surviving and Learning From Kaseya Cyberattack

When cyberattacks happen, most of us only hear reports from the media about what the FBI might be doing or how the company that was attacked is coping with it. We don’t often get a chance to hear from those on the front lines—from the businesses who were affected or from those who helped those businesses get back up and running.

Luckily, we had just such an opportunity recently, when Jay Tipton, CEO and Owner of Technology Specialists, appeared as a guest on the Cyber Crime Lab Podcast. Jay was one of the 50 managed service providers (MSPs) who were affected and he and his team worked day and night to clean workstations and servers and get his clients back in business.

To better understand what Jay shared, we need to know the facts of the case first.

 

The Kaseya VSA Ransomware Attack

Even those familiar with the basics of technology might not know what Kaseya or VSA mean.

Kaseya is a software company headquartered in Dublin that offers a framework for maintaining and managing IT infrastructure. The products it offers, including one called VSA, are used by MSPs around the world.

Kaseya VSA is a remote monitoring and management (RMM), endpoint management, and network monitoring solution.

Ransomware is a type of malicious software designed to block access to a computer system until a sum of money is paid. In the case of this attack, $70M in BTC was demanded by the attackers.

This particular ransomware attack was probably initiated by a gang known as REvil, which injected code into VSA.

 

What it Was Like on the Ground

Jay was at a client’s, working on a laptop, and saw a couple of Microsoft products close themselves before he signed off. He thought it might be a standard program bug. But as he headed back to the office, he spoke to one of his team who told him that multiple client calls coming in to say that their computers were down.

When he got back to the office, Jay saw ransomware on one of the computers and went straight into Technology Specialists’ network operation center (NOC) and literally started pulling plugs and turning things off until he could figure out what was going on.

Over the next few hours, it became clear that all his clients had their data encrypted as part of the attack and he had to fend off angry customers who wanted to hold him accountable.

“You go from blaming yourself, to thinking of blaming others, to taking full responsibility, being totally numb, and not being able to do anything,” Jay said. He worked for almost two days straight before collapsing onto one of the company couches. He and his top engineer logged almost 500 hours each in the four weeks that followed.

During this time, two things happened that Jay and his customers had no say in:

  1. Kaseya refused to pay the ransom
  2. The FBI acquired a decryption key that it refused to share with Kaseya

The purpose of this article isn’t to critique either Kaseya or the FBI for their actions (that’s already been done) but to give context to what Jay and his team had to do. In the absence of the decrypt key, Jay offered a simple clean and restore of all the workstations and servers affected. This was an active move, as it meant not waiting for a decrypt key—which would take everything “back to normal”—but instead got companies off on the best foot they could manage with whatever backups they had in place.

Since Technology Specialists was itself affected by the attack, they had to find an old server that had contact information for clients to start making appointments to get the hardware fixed.

When they got started, Jay was overwhelmed by the support from clients and team members who pitched in with help—be it time or food. He even had ex-business partners and employees come in to help.

It took six weeks after the first day of the initial attack for all of Jay’s clients to be fully functional.

 

Prevention?

Jay notes that despite using industry best practices for his clients including two-factor authentication (2FA) on numerous applications, a vulnerability in software was still exploited.  That’s something we have to become increasingly aware of: that despite our best efforts and security measures, it’s likely to be a question not of if but when we deal with a cyberattack that affects us or our businesses.

With that inevitability in mind, Jay and his team have been putting together services that can respond more robustly to future attacks. Jay found that at some points during the attack there was so much information coming in and so little frame of reference to make the right decision, that he simply froze. Unable to make decisions, he wasn’t able to help anyone.

This situation will be remedied in the future. Veterans of the Kaseya attack will fly out by helicopter, if necessary, to more remote clients to help them with the decision-making process that Jay had to struggle with in July and August 2021. As Jay learned, “winning” in this scenario wasn’t about waiting for the authorities to “do something” but about finding a way to communicate with his clients and get a plan of action in place. It was that “can-do” attitude that ensured that Jay kept all but one of his 50 clients, some of whom had been there from the very beginning, in 1998, when Jay started the company. His actions under pressure are a helpful guide for anyone navigating a business crisis, particularly one as traumatic as a cyberattack.

If the worst happens, you’re going to want the best financial, legal, and technical support to get you back up and running again. With cyber insurance from DataStream, we find the most comprehensive insurance coverage on the market alongside critical post-incident customer support.

 

Click here to learn more about how we can help secure your business data!

www.thecybercrimelab.com

Learning More About the Realities of Cyber Attacks

While many people know that cyber attacks happen, they are often less familiar with “what happens next,” or the probabilities of such an attack happening to them or their organizations.

That’s why we started the Cyber Crime Lab Podcast, to shine a much-needed light on cyber crime through the stories of those people who’ve endured what is probably the worst day of their professional lives. We will also hear from the people and organizations who help victims recover from these attacks, and offer practical advice for those who don’t want to be the next chapter in this story.

Join Andy Anderson, CEO and Founder of Datastream Insurance, as he uses his decade-long experience in tech and insurance to break down these issues so you don’t need special qualifications to understand.

We look forward to educating and protecting you!