I’d say perform the tests there first and see if you get the same disconnect / performance issues
We already tried this and we had the same performance issue
Did you set up a domain service account to run the app pool? Maybe its getting locked out?
Yes, we setup domain service account to run the app pool. Can you please clarify what do you mean by locked out?
Too many bad password attempts, like a normal user lock out.
When a user password is expired in AD, if you are using SSO, Epicor can throw the attempting to re-connect to the server error.
The account is okay. There is no issue with the password or locked out.
What kind of client PCs do you have? I’ve found that there is a lot happening at the client level that is never captured in traces. Everything can look peachy keen in traces and server config and things still be dog ass slow. When the UI hangs it doesn’t seem to impact timing of executions the way you would expect.
We have different type of laptops DELL, HP or ASUS. Most of the laptops are either core i5/7 with min of 8GB memory. I don’t think it is issue with the PC because this slowness is happening for all the users randomly.
Coming in a bit late on this but it could be related to the protocol you using between the App server and SQL server. Are you using TCP/IP or Named Pipes?
I would suggest that you use TCP/IP only with a fixed port. When MS SQL is on the same server as the App server then this is not too much of an issue but when the DB server is split this can cause problems.
Here is a screenshot of the configuration I am referring to:
1 Like
I’m also coming into this a bit late…
We have a pretty large Epicor environment completely virtualized (included SQL DB) on a Hyper-V 3 node cluster with a Compellent SAN. We have 1 SQL server, 1 SSRS server, 8 Epicor App servers, and 6 Microsoft RDS servers… also a few more VMs for Bartender and DocStar. We have 7 remote location that have 100MB connections back to the central location where this Epicor environment is located… around 70% of our users connect remotely through the RDS servers and a few Finance people also connect remotely using a local client. We have around 100-120 full client users and 200-240 MES users connected at any given time, licensed for 140/275. Over all we are definitely happy with our performance.
I didn’t notice if you said what size your environment is but there are a lot of factors that come into play here which will depend on how many users you have.
I know you said you have SSD for storage but what raid are you running across how many drives… have you run SQLIO test on your SQL Server and is it within Epicor’s requirements? Have you verified you SQL recommended settings, like have you set a Maximum memory that SQL can use so it does not starve the OS of memory, SQL will try to take every last bit if it can without this.
This next thing can be a big issue depending on how many active clients you have. On your app server look in the Task Manager and see how much memory the IIS Worker Process is using. If you are over 9GB of memory then you should consider adding another App server… if you get close to 10GB of memory usage on this IIS Worker Process you will start seeing slowness and even freezes in any clients connected to that app server. The more clients you have connected to the app server the more memory this IIS Work Process will use… so the number of active client connections you have is going to determine how many app servers you will need. With the number of users we have we are averaging about 6GB across all 8 of our app servers. Another thing, I would consider running the Task Agent on an App server that is dedicated just to that Task Agent (no clients) especially if you have a larger environment… because when some processes are running, like MRP, EDI, or Postings, the Task Agent can use up a lot of this memory in the IIS Worker Process while these processes are running, and when clients are also connected to the same App pool it can quickly hit that 10GB mark.
What does your Hyper-V setup look like. Are you running this on a single Hyper-V host? I would recommend that you have both the App server and SQL server on the same virtual switch… and if you are running a cluster make sure that virtual switch is attached to a 10BG NIC for the fastest connectivity possible between the App server and SQL server. Epicor can be sensitive to latency between the App server and SQL server.
I see in your Performance diag there is issues related to power setting… make sure your power settings are set to high performance in the control panel.
For the Hyper-V host box have you reviewed the BIOS settings? There are things in the BIOS that will definitely effect performance if not set properly… since BIOS settings can very a lot I really can’t give you specifics. What processors do you have in the Host box/es? If you don’t have procs at least around 3GHz in speed Epicor is going to perform very slow.
5 Likes
Hi Jeff Martinson thanks for the details explanation. Our environment is really very small. Here are the details about our infrastructure.
Epicor Version: 10.2.200.27
License: 20 full license and 6 MES license
We have two offices within 300m distance directly connected with Wi-Fi (1Gbps)
We have two VMs one for Epicor Apps and one for SQL database. We are using Hyper-V on Windows server 2016. Both VMs are on the same Hyper-V Windows server 2016
Processor: Xeon 2.3GHz
Each VM has 48GM memory and 4 core of CPU
We are using RAID 1
Average memory utilization is 20% on Apps server and 80% on SQL server
We have checked the BOIS and server hardware and there is no issues found. We have verified the server hardware with DELL and they did not find any issue on server.
Latency between Apps and SQL server is very good but latency from user PC to server is 1ms and sometimes it goes up little bit.
Disk I/O on both servers are very good.
Hi Mohammad Sazzad Hossain,
The 2 things that stand out with your environment is processor speed and storage.
First, the processor you have at 2.3GHz is slow for Epicor, even Epicor’s hardware guide says this. Regardless of environment size Epicor (SQL and App servers) really needs fast processors… your environment size will determine the core count you need, not processor speed. Like I said before to have good performance you need to have a processor that is around 3Ghz depending on the generation of Xeon. If you are running the latest generation of Xeon then a 2.8GHz (with a 3.4 - 3.5GHz turbo) will have good performance but if you have like a v2 or v3 you will want at least a 3GHz. Regardless, the faster the processor the better… it is kinda a juggling match to find a processor with high enough speed for Epicor and enough cores for virtualization. Also, only 4 vCPU cores on the VM’s seems low even for a smaller environment like yours… you have to remember the OS and other services will also need a couple vCPU cores so you are basically only leaving 2 vCPU’s for SQL and IIS. I would recommend at least 6-8 vCPU cores for each VM’s… SQL is multi-threaded and will take advantage of more threads.
I’ll give you an example of processor speed difference. We used to have a site with it’s own E10 environment. This site had it’s own hyper-v cluster that hosted VM’s for E10 and other servers at that site. This cluster had 2 nodes running processors at 2 different speeds… one node had a Xeon E7-8891 v2 at 3.2 GHz and the other had a Xeon E7-4830 v2 at 2.2GHz. When we would live migrate the E10 VM’s between these two Nodes Epicor ran at least 50% faster on the Xeon E7-8891 v2 at 3.2Ghz.
Our current corporate E10 servers are running 10 core Xeon Processor E7-8891 v4 at 2.8GHz with 3.5GHz Turbo… yes the Turbo makes a difference because our processors are never pegged across all cores but we needed the core count for the number of VM’s we have.
For your storage, even though you are running SSD’s having only raid 1 is limiting your performance. With raid 1 you are really only getting redundancy and not much for performance gains. You should really consider running more smaller drives with raid 10 (you need at least 4 drives) than just 2 larger drives in raid 1. You do not need anything large like a 24 disk raid 10 array for your size but like a 6 disk raid 10 SSD array will be a big improvement over raid 1. Another thing is if you are running multiple raid 1 arrays, like a different drive for each VM, you will probably be better off performance wise with 1 large raid 10 array than multiple raid 1 arrays.
2 Likes