I’m wondering if anyone has had similar experience to our slowness (especially at a remote site) issues.
We have been having a lot of Epicor slowness for the last year+ and seem to notice it more in our Iowa facility vs our Connecticut facility (where our hardware resides).
We did some very thorough testing this week with our networking consultant and Silverpeak (our networking hardware vendor) to see if there were any network related issues that is causing the Epicor slowness. Below is a synopsis of what we found (from the words of Sean, our networking consultant). Essentially, we are seeing large packets being sent from traffic within Connecticut network. However, those same processes done in Iowa are not seeing those packets being sent from Iowa to Connecticut, or back. These tests included logging into Epicor and opening Part Tracker. We tested both sides equally, at the same time and with the same process. What are these large packets being sent, and why are the not appearing in Iowa traffic?
Essentially, we are several packets that have a length of 1514 coming from the Utility server, higher than that standard Maximum Transmission Unit (MTU) of 1500 when we connect in CT.
Those oversized packets aren’t present in anything going to Iowa, in fact, they don’t seem to be going much of anywhere beyond the local network. Since the packets are going through locally, there shouldn’t be any issue since the main switches all seem to have an MTU of 9000 (Jumbo Frames enabled) and the packets are working locally. I will review the switch configs to verify those MTU packets are looking good on the majow switches.
We need to find out what those packets are and if they could be affecting the connection to Iowa. If so, can we dial those packet sizes back so they are the standard 1500 MTU length?
There is very little latency and pretty much no drops in packets going to Iowa as far as they can see. What goes in to CT is coming out in IA and vice versa, with the notable exception of those packets that we are seeing on Rob’s pcap.
No idea about the packet size question, but the networking gurus on this site will probably want to know what protocol/binding settings you’re using (NET.TCP vs HTTP, HTTPS).
As far as general slowness across all sites, the most common culprits I tend to see are 1) database indexes not being rebuilt regularly, and 2) inefficient BPMs (often firing multiple times unnecessarily)
There was also a fiasco earlier this year where Windows updates were causing severe client side slowness. Last I heard that was supposed to be resolved, but if you are behind on workstation patching maybe that issue is still in play for you.
MTU and packet size are the domain of the ethernet layer, not TCP… Therefore it’s most likely a switch or router configuration that’s wrong. Epicor running on IIS has no way of setting or even knowing the MTU, this stuff happens way below the application layer…
Besides, at worst having the wrong MTU config might cause some fragmentation, but that really shouldn’t affect performance unless the link is already saturated.
Maybe check and make sure no one is downloading torrents on the site’s internet connection, or uploading backups, or anything silly like that… Epicor does require a fair amount of bandwidth to transfer datasets back and forth…
We did get the Windows update issue resolved a while back. The indexes are being rebuilt every week as well as definitions being updated. The slowness we see is as basic as logging into Epicor and opening Part Tracker. That process in Iowa takes over 1 minute. In CT, where our servers are, it takes roughly 25-30 seconds. Essentially all of our processes take twice as long at our remote facility. I would expect some delay, but not to this extent. Epicor basically goes into a Not Responding state when anything is done within any of the menus.
Where is the physical server located? Is it being hosted in Connecticut, is it on the cloud, are you using a VPN, are the Epicor clients configured to use the net.tcp endpoint, or the https one?
All questions that need answers before anyone can really help you…
Also have both sites run a Speed Test to make sure the issue isn’t poor internet connection…
The physical server is located in Connecticut and we have an SD-WAN solution that we are using for connection between the sites. Our remote employees are using VPN to connect internally for Epicor. Speed has been really good, and we do not have high traffic utilization that would be a bottleneck. Epicor clients are configured using the net.tcp endpoint.
It does not appear to be network related. When looking in our Silverpeak, the packets are being sent and returned in milliseconds. Speed test in both locations 80+mbps upload and download. Also, here are my results for Network Diagnostics from the PDT tool, ran from my machine in Iowa to the CT application server.
Have you tried using HTTPS instead of NET.TCP? I don’t know a ton about it, but the Epicor System Admin tech reference guide literally says that https is “designed to facilitate communication between clients across Wide Area Networks and the internet”. Whereas net.tcp “is designed to facilitate communication between servers that reside in the same data center”.
That’s really just a question of routing, you can’t route net.tcp across the internet, but across a VPN it shouldn’t matter. Net.tcp will always perform better than https…
If the network performance is definitely not the culprit, then next I would look at DNS issues. Try using IP addresses in config files instead of host and domain names.
Net.tcp in Epicor has been very sensitive to connectivity. Enough of a delay and we would get disconnected and have to restart the client. For us, the slight performance benefits of Net.tcp were dwarfed by the number of restarts. Https on the other hand, we could have full line drops for seconds at a time and the client would happily pick up where it left off without the user noticing anything.
And as mentioned in another post, Net.tcp is WCF and that is a technology that Microsoft is giving no more development time. And of course, the browser version of Epicor is all https.
It’s all WCF… net.tcp vs https is just the endpoint protocol. And even if WCF is being deprecated in .NET 5, that is just because .NET 5 is moving to a Core/Standard base platform. It will still be production-ready and supported for decades. WCF is scheduled to be replaced by… CoreWCF! lol
(Yes I know Microsoft is pushing gRPC, but that’s not going to take unless users adopt it, and that’s not going to happen until the expertise develops around it…)
Yes, WCF will continue to exist but the setup of https is far more complicated than the preferred ASP.Net Core WebAPI (soon to be minimal APIs in .Net 6.)
More specifically, why stay with Net.tcp? It only works if both ends are Net.tcp. Bart said the speed difference was getting quite small.
From experience, a WCF service running on a net.tcp endpoint is far, FAR faster than any REST service, and a little faster than binary SOAP services. It’s also far more efficient, since everything is passed as binary objects, and there is no XML or JSON wrapping, and so no serialization or deserialization.
REST was never about performance, it’s about usability and portability.
They compare serialization here