How it Started

It all started on a Friday morning when employees of a major staffing agency were locked out of an app on their mobile devices that allowed them to clock in and out. This app would track the hours they worked, allowing employees to get paid. The agency would also know how many hours to bill their customers. It’s a simple and paperless solution that works very well…when the technology cooperates.

Because the company was well-established, they also had a paper backup system in place ready to go just for scenarios like this one. However, this system was neither convenient for the hundreds of individuals involved nor guaranteed to be in compliance with different state laws regarding breaks, overtime, etc.

The Vendor Refuses to Help

With their client temporarily “fixing” the issue by switching to a paper backup system, the 1Path team started analyzing what had happened in order to fix the problem. Their credentials weren’t working and their first call was to the vendor of the app. They were not helpful. 1Path owned all the licenses for this application, and the vendor essentially said (with a straight face) that they couldn’t be sure that 1Path hadn’t themselves made the changes responsible for the problems affecting their client.

What had happened was that all user profiles had been deleted and only one (newly created) administrator account remained, [email protected]. Given the situation, Armon and Patrick were flabbergasted at the vendor’s attitude. (Especially given the name of the new administrator account, which would indicate an author with bad intent.) While 1Path was trying to solve a problem for one of their own end-users; they also very much considered themselves as an end-user in relation to their vendor. 1Path was definitely not feeling the love, especially given all the business they had done with this vendor over the years.

Given this temporary dead end, 1Path continued to follow its normal policies and procedures for crisis management. They determined that no current, or former, 1Path employee had access to the environment. This allowed them to create a roadmap to help them get the client up and running again.

While 1Path started the process of getting new, properly provisioned, mobile devices shipped out to hundreds of employees all around the country, they were still working with their vendor to find a faster solution.

The vendor finally agreed that they would be willing to release some information if:

• An email was sent from a personal email address (not a company one)
• With a signed letter from the CFO of the company

1Path waited for three days then called to follow up. It turned out that there was something in the letter that the vendor didn’t like and they hadn’t bothered to call to tell 1path what it was. 1Path jumped through some hoops to satisfy the vendor and finally got access to their account. They then found emails in the archive from years ago, when the client was first created, and with that, the vendor was finally able to reset 1Path’s account.

The Culprits

It turns out that the entire attack was orchestrated by two former employees of the client in question. Some time prior, 1Path had exposed the fact that these employees were spying on the email of the CEO, this had led to their dismissal. They then decided to get their revenge on 1Path and cost their former employer a lot of time and money, and their former colleagues a lot of hassle.

They had probably retained access to, or had a copy of, an administrative password that hadn’t been changed and didn’t require multi-factor authentication.

Lessons Learned

The first lesson that 1Path learned was directly related to this security hole. As a result, they implemented a forced password policy (making a user change the password upon login) combined with multi-factor authentication. They didn’t just do this with the client who had suffered the attack, they rolled it out across all their clients.

The password policy change was part of a larger conversation regarding collaboration with customers. Thus explaining the desire to be true partners, which meant abiding by standards that everyone could see would lead to better security. The conversation was also framed in the context that the instigators of the attack seemed to have a collaborative relationship, but “the good guys” didn’t have any collaborative plans in place.

1Path also learned that despite having a “relationship” with their vendor, when the chips were down, that vendor couldn’t be relied upon. The vendor had long contracts written by expensive lawyers ensuring that their liability, in situations like the one above, was almost nonexistent. That has led to 1Path building systems with more backups and redundancies, to make sure that they don’t ever have to rely on a vendor again the way they did in this particular case.

As we said at the beginning of this article, no one expects an IT provider to be perfect. Customers know that at some point there will be a crisis, and it will be in that crisis that a partnership will be tested. 1Path came out of this crisis stronger. They realized where they could improve and adjusted their expectations and policies accordingly.

If the worst happens, you’re going to want the best financial, legal, and technical support to get you back up and running again. With cyber insurance from Datastream, we find the most comprehensive insurance coverage on the market alongside critical post-incident customer support.

Click here to learn more about how we can help secure your business data!