Epicor 10.1.500 Performance Issues

Epicor ERP 10
1

Good afternoon everyone,

We are attempting to install a new Epicor 10.1.500.13 environment and we are running into some pretty horrific performance issues.

The servers are configured as such. The SQL server is a physical box with SSD direct attached storage and 96Gb memory with 8-core. There is currently a load balanced NIC team with 2 ports.
The application server is a on a VM Host with it being the only guest. It has been configured as to the sizing guide for 50 users as provided by Epicor. The host is ESXi 6 with all of the latest NIC and storage drivers up to date with 2 virtual switches, 1 production and 1 VM management. This is also all SSD direct attached storage with the same specs as the SQL physical box. These 2 machines are connected to a new Cisco switch with new cables.

The initial configuration and setup on both servers was completed by Epicor consulting and at that time the server was 10.0.400 and over the past year or so the applications team has been importing data and upgrading to 10.1.500.13. These performance issues were not seen until some moderate testing and Import activity had started being performed in the last 5 months or so.

We noticed using the DMT tool the records would be going in at a reasonable pace then come to stop for about period 3-10 minutes or longer and then start again. Is this common occurrence with DMT?
The load times for opening Order Entry or Customer Tracker is often times longer than 1 minute. Even switching between records could have 15-20 second delay.

The primary indicator of trouble is the ‘Network Diagnostics Tool’ shows poor performance on the network times after the 9th test in a series of 10.

Here are a list of things we have checked and confirmed to be correct:
BIOS Settings on both boxes and in VMWare – all performance levels are at max
Driver updates and all OS updates
SQL memory usage setting
Several network testing utilities (PSPing, Measure-Copy in Powershell) – all within acceptable ranges as per our consultants.

Any ideas would be greatly appreciated

Thank you

Don Kollmann

2

What are you DMT’ing in? We have orders with near 1,000 releases and a
DMT slows down after a few hundred releases. Same with customers with a
ton of Ship To’s or Contacts.

3

Just the base files for PartRev – Company, PartNum, RevNum, Plant .
4,500 records – 67 Records per minute
The 1st 300 records go in just fine at a 400 records per minute approx. and then hit the wall and it slows considerably to 67 records per minute.

I don’t believe it just DMT as we are seeing the performance issue in the application when opening screens it can take 6 minutes to open the Sales Order.

Don

4

Probably not related, but I had to post anyhow. When you look at your server and you aren’t even doing anything, is the CPU sitting at like 30-40% usage?

I had this in upgraded 10.1.500.12 environment. I narrowed it down to only happening while the Task Agent Service was running, but the system monitor claimed nothing was running (I even deleted all my scheduled tasks from saytem agent). I turned on Server Trace/Logging on the app server, and saw this exception being thrown over and over, quickly filling up log files:

<Op Utc="2017-02-15T20:57:33.0795118Z" act="Ice:Lib:RunTask/RunTaskSvcContract/RunSystemTasks" dur="1.0687" cli="fe80::84fc:9621:aaf:db56%12:53886" usr="NT01NL\EpicorAdmin" machine="SVICEERP03" pid="2600" tid="44">
  <Exception><![CDATA[
Unit of measure is not active for Class:Other  UOM:FT (BusinessObjectException) 
... etc etc...

Although I’m still not 100% sure why it was happening, I figured out that in my particular scenario I could get it to stop happening by re-enabling the UOM in that particular UOM Class. Like I said, I highly doubt this is your exact scenario, but maybe it will give you some ideas on some more stuff to check.

5

I would check more into the disk usage statistics on the SQL server. Specifically average queue length, we were having issues with average queue length getting to high during a migration and it was being caused by the database autogrowing every 10GB.

You can see the latest autogrow statistics for a DB in SSMS under the Disk Usage report just below the graphs. The disk usage report can be found here;

Pasted image756×372 27.4 KB

I haven’t had the pleasure of using the DMT tool, so there may be an issue with that instead or maybe as well as a disk IO issue.

2 Likes
6

Adam,
I looked into your idea and the CPU is sitting at 3% usage.

Thank you

7

Have you sized your SQL databases correctly? If they are doing an autogrow, it may be contributing to the delay.

8

A post was split to a new topic: Add UD fields to RDD for Job Travler (Crystal Rpts)

9

Have you ran the Performance Diagnostic Tool to done a check config?
It does sound strange behaviour perhaps install onto another box like a local laptop run the same DMT import and see if you get the same result as a test.

Is there any ANti Virus involved?

You could try enabling the trace flags and sql profiling if you can recreate.