I just heard about one three-man on-prem team who has to manually restart about one thousand machines this weekend and next week with offsite people.
3 Likes
The “reboot 15 times” fix? Yes, I facepalmed at that suggestion. 
3 Likes
honest question, why would this affect an on-prem setup?
Or did I just give away a vulnerability about mine by asking?
We are on prem, and we had servers and workstations down this morning. With a few of us, we were able to get them all patched and running again in 4 hours.
If you have a lot of machines and are remote, it becomes a nightmare because the system isn’t loaded to run your remote administration software. So you have to rely on end users to help you out…
1 Like
If you used Crowdstrike on-prem you could of still been affected. Your servers may be on sight and under your control, but they’ll need to get/send something over the internet eventually so you have some vulnerability.
1 Like
Apparently if you have bitlocker you may as well just cry in a corner. Cause you can’t just restart and delete the files
You have to unlock the drives using a BL recovery key which is likely sitting in a management server that itself is also… Crowd"Striked"
I cannot for the life of me not fathom that Windows in this day in age doesn’t have a simple recovery that says hey if you can’t boot X times in a row cause a particular driver fails just… STOP LOADING THAT DAMN DRIVER
Even Outlook and Visual Studio tell you after so many times when an extension is being dickish that they’ll stop loading it so is not like Microsoft doesn’t know this is an option.
6 Likes
Remember “revert to last known good configuration” in safe mode? Yeah the planets all had to be aligned for that to even work 
1 Like
I spent many many hours manually doing BitLocker recovery today. It was painful.
5 Likes
After 2weeks of printing bugs and a UAT that almost failed for printing, I pick the RIGHT weekend to go off grid for some camping.
So, what’d I miss? 
6 Likes
Yep. Bother works for state IT, all the gov computers have bit locker. He had Friday off, but instead he worked Friday, Saturday and Sunday
I’m with you… took Friday off (first day off in a while) assuming I’d have a busy week this week after the 2024.1 roll-out to production.
I asked this morning how Friday went, everybody said it was fine. No issues. Based on posts I read on here over the weekend, I assumed all cloud customers were hit, but somehow we skirted through. Lucky I guess.
1 Like
An epic 
post.
1 Like
We were actually fine on Friday as well, it was just confusing because of the Azure DC outage immediately preceding on Thursday night which DID bring us down. I don’t understand why some were affected by crowdstrike on Friday and others weren’t. Different DCs? Or maybe it was just a question of the order in which systems were restored?
2 Likes
Interesting. We generally have a skeleton crew on 2nd & 3rd shifts and I hadn’t heard of any issues. But I just had a coworker tell me they had an issue Thursday night working on a PO while working from home. Couldn’t even reach the Kinetic log-in screen.
So, I think we were in the same boat. Had an issue Thursday evening, but then no problem on Friday.
Us too! Epicor was fine Friday morning for us when business opened at 7AM. There were some 3rd party integrations, web site, EFTs, supplier portals, etc that were our main headaches that day which we could only wait for the vendor to fix.
We were mildly affect by the Thursday night issue on 2nd shift but they work of printed Job Travelers so no production was slowed they just couldn’t clock out end of shift.
We are all used to such things from anything running on Windows environment. It was under control when software updates were not pushed automatically.
Stating the obvious but what better controls are required for applications with this level of access.
Signature and hash verification ffs.
Crowdstrike:
Verify patch works. Sign and hash.
Deploy to stage, verify hash etc, deploy.
Windows end:
Download, verify hash etc, install.
The basics. They tested this patch in house, it worked fine.
Somewhere between them and the deployment server, the file became all nulls.
Obviously the hash and sig wouldn’t match, but there was no checks… anywhere.
3 Likes
Apparently, Azure outage it unrelated to CrowdStrike outage
https://www.reddit.com/r/AZURE/comments/1e6qles/comment/ldvvdc7/
Root cause was a botched decommissioning of legacy storage services. Product group deleted the wrong thing which took the entire region down.
Source: I was on P1 breakout w/MS PG engineers.
4 Likes