Sudden drop Epicor speed caused by Microsoft Updates -> (KB4601319, KB5000802, KB5000808,KB4601382,KB4601383) +

Update

Microsoft confirmed the performance fix affecting ERP will be released on July 20 as KB5004296. For more information see "KB0108621 February and March 2021 Windows

EDIT Current list of updates that have been seen to slow the performance of Epicor are as follows.

W10 build 1909 KB5000808
W10 build 2004&20H2 KB4601319 & KB5000802 & KB4601382
Server 2019 Build 1809 KB4601383 &KB5000822

Original Topic
As far as I know nothing has changed in the environment except maybe the typical windows updates that get processed from time to time, but as of 2/11, maybe a day or so earlier, loading data into a form in Epicor has drastically slowed down.

Queries donā€™t seem to be effected, but when you open job entry, for example, open the search dialog search for any job, which results load quick, and then select it, the loading of that job into the entry screen is painfully slow, it takes at least 8-10 seconds (for me, longer for others) for the form to fully load and be usable. If you click on the clear form button it takes about the same amount of time to clear the form and you will get a (Not Responding) in the title bar for a few of those seconds.

Iā€™ve power cycled pretty much everything in our environment, Server, Domain controller, switches, Firewallā€¦ Iā€™ve also double checked to ensure there were no pending updates that were installed but waiting for a reboot on the Epicor server. No other network traffic seems to be hindered like this and if you are using Epicor on the server locally itā€™s a 1 second form load time, nearly instant.

Running a quick ā€œLan Speed Testā€ application and writing a 500mb file to the Epicor server from my machine Iā€™m seeing speeds of 728.5Mbps write and 860.4Mbps read, as well as pingā€™s are all <1ms. Most of the network machines in my environment are only 1 hop away from the server, there is a small handful that are maybe 3 hops away, but all of the backbones between those switches are at min 1Gbps full duplex links. Iā€™m also not seeing high network port usage on the switches at all, they typically are all reporting 0% load on all ports, even the uplinks.

Any insight or areas I can check will be greatly appreciated.

14 Likes

We experienced the exact same problem last week on the 10th/11th, but mainly for my remote offices in the EU. Everyone here in the building with the servers (or working from home in the area) seems to be fine.

We have noticed in the last year or so, that ā€˜pendingā€™ Microsoft updates cause issues. Seems stupid, but weā€™ve noticed on the PCā€™s that any pending update will drop the PCā€™s performance drastically. Iā€™m no conspiracy theorist, but this is just too coincidental. Once the updates are processed, everything is fine. this only applies if you have it set to download and prompt, rather than just prompt or auto-install.

Over the weekend, I went through all the Epicor servers and installed the patches, rebooted, cleared logs, etc. and am waiting for feedback from the users now.

I know thatā€™s not a great answer, but Iā€™m glad that Iā€™m not the only one experiencing an issue. Sadly for me itā€™s not limited to remote users. I feel like at home when I was on the VPN the speeds were the same as they are right now when Iā€™m in the office and thatā€™s obnoxious.

Also as you noted, yes I have updates install automatically and then reboot on a schedule and Iā€™ve seen that process really slow down a box until itā€™s rebooted and updates are fully applied.

Just for other info, pretty much every one of our machines is currently at 2004 W10 patch level. That has been rolling out in the past few weeks, but I have not seen the slowness be related to that directly. Iā€™ve had some machines on 2004 for a while now with no speed issues.

The thing that really bugs me is the fact that the speed is normal when running on the serverā€¦ That points directly to a network issue, but I canā€™t replicate any slowness in that area on the server or the network as a whole. Thatā€™s what has me really confused.

Better to check SQL Monitor if any query is stuck or recycle all App pools, it will clear if anything stuck.

If that were the case would it not perform the same at every location including locally on the server? As noted Iā€™ve done multiple server restarts, including App server pool recycles and nothing has changed. I would assume that a server restart would kill any and all queryā€™s that could/would be stuck.

With SQL Activity monitor open and opening jobs locally and on my machine, Iā€™m barely seeing any activity on the overview graphsā€¦ nothing is being even remotely stressedā€¦ Never over 0.8MB/Sec Database I/O and not even 5% on the processor time graph.

I am glad I found this. The exact same issue here as of around the 10/11th of this month. Job Entry and Order Entry screen load is very slow. I thought it may be with the app server or the sql server, but after restarting everything and reindexing the SQL database the issue is still very noticeable. My desktop is completely up to date on windows updates, but last week we did update desktops to Windows most current update and this is when the issue began.

1 Like

Weā€™re also finding this as our win10 machines progressively get their 20H2 updates. We had stalled them since the 2004 update after we figured out an SMB change in the update broke our network share to an old windows server hosting our legacy Vantage ERP.

users have reported the same on my environment, done all the tricks but no luck

Has anyone done any network analysis in these environments, or running a new windows performance test against your benchmarks?

good idea, have you got a link to explain how to do that ?

There is a performance tuning document on Epicweb that goes into the detail if I recall. It talks about using windows Performance Monitor and the PAL tool.

To do an initial check use the resource monitor on both the Server (if it has updates applied at the same time as the Client PCs) and locally on one of the windows machines that also had the update. Resource monitor is accessable from Task Manager or from the command line with %windir%\system32\perfmon.exe /res Take a look at the network tab.

On other really useful tool is the Windows Admin Center, which you can download and install. You can use it to view and monitor both On-Prem and Cloud environments (Azure).

You can download it here. Windows Admin Center Overview | Microsoft Learn

Of course the other place to look at network traffic is on your switching infrastructure, it really depends on what sort of hardware you have and what sort of monitoring your Network Admiins have setup, that might help put you in the right direction.

The other fall back is using the performance diagnostic tool on the local pc and test the network with that. Of course if you have not done it prior to the problem it might not be much help.

In Summary:
Look at the simple tools first (like task monitor/resource monitor), look at high CPU and disk processes on the server and on the PC., then drill down into the more detailed monitoring tools. Making sure you keep benchmarks for the next time you have issues.

Unless you have data to prove there is a performance issue it can be difficult to refute claims by users when they say ā€œEpicor is Slowā€.

I have seen in the past where Windows Updates do things like re-enable Windows search, or reset the indexing to defaults which sends the PC into a spin, itā€™s noticed in particular when the PCā€™s are on the edge of the Epicor System requirements.

Apologies for the vague and long post, but diagnosing these sorts of issues, can be tricky as there can be so many different variables at play.

Hereā€™s a Microsoft Doc on the PAL tool. Using the Performance Analysis of Logs (PAL) Tool - BizTalk Server | Microsoft Learn

2 Likes

many thanks @Hally, not at all, it is crystal clear to me, i appreciate that mate

Update:

I have found that 2 of the 2/9/2021 updates (Microsoft Update (KB4586876) and Security Update (KB4598242)) were the cause of the slowness in Epicor. I have uninstalled these updates on my machine and the result is night and day when it comes to the performance increase with Epicor.

4 Likes

I just came to a similar conclusion, but different updatesā€¦ I removed KB4601319 which made the biggest difference. It was installed on 2/10/21, in addition to that I removed KB4601050, but I did not notice a change in the system when removing that update.

Jon_Hall, which version of W10 are you on? The update that changed it for me was specific to 2004 and 20H2 versions of 10. I also did not show those same updates on my machineā€¦ Only after uninstalling KB4601319 did KB4598242 appear and shows an install date of 2/17/2021, and currently my experience is back to normal.

Iā€™m going to chock it up to KB4601319 for my environment.

1 Like

Wowā€¦ the process to hid a damn update now is MISERABLE! Thanks Microsoftā€¦

After removing and re-installing KB4601319 and the performance dropping like a rock again and then returning, I have now hidden that update using the Show or hide update tool that I had to download from Major Geeks since every link from Microsoftā€™s download center was broken. After removing that update, it reverts back to having the KB4598242 update installed which Iā€™m assuming is the 01/2021 security update. That update is not proving to hider performance any. Also KB4601050 Update for .NET Framework is not the culprit either.

I marked this as the solution since this update removal has fixed my issue.

8 Likes

I was checking all of my network statistics and as noted in my earlier posts, none of my switches were even flinching in the least when requesting this data. This is a small company with about 120 machines total and 40-50 power users. I have Cisco SG350 switches as my core and access switches and they are hardly tasked in the least with our throughput. This is why I was wondering if anyone else noticed this change, because if they had then it definitely points to a software change somewhere since my hardware hasnā€™t changed for over 2yrs.

More Problemos

I donā€™t see how that is related to this? Also a majority of my machines are on SP 2004, not 1809 or 1909

Good Point. I am 2004 and my screen lags, I am updating to 20H now see if it helps :slight_smile: