Makes you wonder how good your response plan is to various levels of cyber attacks.
The company has confirmed that work at the UK plant has been halted alongside a suspension of other operations in North America, Turkey, Italy and Japan.
However, it added that it hoped some of the affected sites would go back online this afternoon or later this week.
Imagine if you had to tell your boss, it might not be resolved to later this week - and it’s only Tuesday.
You don’t have to be a big company either. We’re relatively small and one of our techs just proved that restoring our system from backup takes 4 days during which Epicor can’t run because everything is in one place.
So, we’re splitting everything up, but still, our data grew from tens of Gb to several Tb in a few short, well, years I guess, but still - it’s a test everyone should be doing routinely these days. In fact, small businesses are particularly vulnerable because they usually have limited IT expertise and zero redundancy.
Even your backups have to be air gapped or offsite and air gapped. We got hit in 2018 and the ransomware sent a dump your backups command and cleared the primary device which then replicated that command to the offsite DR and that was the ballgame. It was not 4 days it was 21 really long days to build from the ground up a new infrastructure. Now we can spin up and have Epicor running offsite in under 15 minutes and the complete domain in 4 hours.
Wow, that’s wild. Would you be able to share your topology?
This is why we automate.
Backups should always be in 3 (or more) copies, ALWAYS. One copy on-premise for quick restores, another copy on offline tape on-premise, one more copy in the cloud. It’s a good idea to also keep a snapshot of the virtual machines of the servers before and after each update or install.
It used to be 2 copies (one tape on-site, and one tape off-site or at the bank) was good enough, but with these ransomwares the more redundancy you have the better. Also, do not be too quick to rush to come back online. Before you bring the servers back online, you have to make sure you have identified and corrected the vulnerability. If you cannot figure out how anyone got in, DO NOT restore the server backup, “nuke it from orbit”, rebuild it from a base image and apply all patches and updates, then restore your databases only.
User executables and archive files and (if using an on-premise email server) all email attachments should be wiped on restore, and only restored on specific demand, and ideally all email messages on the on-premise email server should be converted to text only. This is where a web or SaaS setup shines. You don’t have to run around all the client PCs, you can concentrate your efforts on just a few servers instead.
That’s a good point. I hadn’t thought of that until I read the book you recommended, The Phoenix Project. I’d love to be able to build the production environment from scratch automatically, because then restoring the DB would be a breeze.
It goes without saying - yet still needs to be said - ALL user accounts should be disabled / have their passwords reset.
Work for a Tier 1 supplier to Honda. A lot of their stuff is still down, but they’re starting to recover and are still ordering plenty of parts.