Virtualized Epicor Server: Now it is slower

So…we virtualized the server! And, when just using basic Epicor, runs normal. However, when logging in, running print routing, our pulling up large customers its taking FOREVER! I was told they have given significant resources to E10 already that match what we had previously, but something is slowing us down. As an example, when I remote into the E10 server not only doe sit take a long time to get in, but even clicking the icons takes much longer than normal.

Anyone else experience this? Any potential fixes?

@Will79 when you say you virtualized your server. Did you do a PtoV or did you create the VM from scratch?

The other question I’d ask is did you have a baseline using the Epicor performance and diagnostic tool before the PtoV and after.

Is your virtual machine host located in the same infrastructure as your original Physical machine?

1 Like
  1. Validate the host(s) configuration (BIOS). Disable Power Saving options and C-States in BIOS. Enable Turbo Boost.
  2. vCPU must be reserved. Create a full (100%) virtual machine reserve to ensure that the virtual machine has dedicated access to physical CPU cores.
  3. RAM must be reserved. Select Static memory when you allocate memory to the virtual machine.
  4. Run the Performance Diagnostic Tool.
3 Likes

We run all of our Epicor infrastructure virtualized and don’t see these issues. So it’s something with your VMs and/or host infrastructure.

  1. vmWare or Hyper-V
  2. SSD or HDD? This will make a big difference if your host is HDD as the VMs will be waiting to read/write from disk.
  3. Memory - Needs to be reserved as @Alexandre_Pothier said
  4. vCPU - Same thing @Alexandre_Pothier already asked
  5. Networking - How many virtual switches/VMs per
  6. What else is taking up resources on the host
  7. New VMs or P2V? Would be better to do new VMs.
2 Likes

Thank you all! i will get with my IT team and see what we come up with.

@Hally VM from scratch, is my understanding. Our IT team did it, and I wasn’t over it. Unfortunately, we did not have an baseline. Yes, it is hosted on site.

More Questions:

  1. What type of CPU, How Many? More Threads or More Ghz Power?
  2. SAN/NAS?
  3. Hyper-V or VMWare?
  4. Cisco, Dell, HP or ?
  5. SQL Standard or SQL Enterprise?

Run PDT Tool, do you pass the CPU Test and SQL Test?


Some Old Meeting Minutes I had when I worked with Epicor

9:10AM: 1 Issue, 8 Sockets on VMWare - Standard SQL can't see more than 4 sockets. (Sockets vs Cores)
Keep the Sockets under 4. (Issue, needs fix)

9:11AM: Launched vSphere for Sherman to show us what he means by Socket Configuration
- CPU tab (1 socket, 8 cores) - Change it to 4 sockets and 2 Cores.

9:15AM: Looked at the Network Adapter - VMXNet 3 - The CM Servers should be ParaVirtual SCSI (some are not, Issue)

9:22: There is some latency there, it will affect how the client protocol performs on the LAN - However E10 uses NET.TCP w/ good compression.
- Downloaded tcping.exe to run TCP Pings
- tcping.exe -t 192.1.10.106 445

PLUS: The Pings 40ms are really consistent, which is very helpful

9:32: Pinging SQL Server
  - tcping.exe -t SERVER-SQL1 1433
  Looking for majority of the pings to be about 0.5ms, we are looking good! averaging 0.5ms as expected.

9:35AM: Run 32-bit Performance and Diagnostics Tool
  - We should be receiving 1/2 second of time to retrieve Packet 0.5 -- we are at about 0.8
  - Sherman: "This could indicate that CPU is not running at full speed"
  - We are looking at <300ms yet its 554ms - BIOS Settings are not set to MAX Performance
   - Lets proof this by going to vSphere looking at the Power Schema at the Host Level

  We are currently running on "Balanced". LETS Change it to "High Performance" (Lets do this also to CPS Later!) - It will BOOST our CPU Performance to meet right Metrics!
  You have to change this at UCS BIOS Setting really. (Cisco UCS Manager at 25.25.254.100)

Bios Policies... 
     C1E Disabled It!, 
     C3, 
     C7 and on CPU Performance set "Enterprise" it will give us the Faster peformance
     [ Will Require a Later Reboot ]

  Power Technology, leave that are Performance and Energy Performance leave that at Performance too.
  P-STATE Coordination: Let's to "SOFTWARE" all and we will set it in vSphere to "High Performance"

  Package C State: C0
  Memory RAS Config: maximum-performance

  VT For Directed IO: Enabled
  NUMA: Enabled

- Disabled SpeedStep which underclocks the CPU
   Short for Enhanced Intel SpeedStep Technology, EIST allows Intel processors to run at a lower speed, which reduces the overall power consumption. This feature is very useful in portable computers that use battery power.

Our Issue was resolved after 1 year of tinkering… Updated Cisco Firmware viola! :slight_smile:

2 Likes

I ran the diagnostic tool and it came back with a CPU Speed warning but no other info.

What CPU is the host running?

HyperV; 64 gigs dedicated; 24 virtual processors; SSD; Intel Xeon Gold 5118 CPU @ 2.30ghz

What server hardware? What OS is the host? Hyper-V on Server Core or something?

Any visible performance difference when running the client directly on the App server?

We’ve been virtual for years. One time IT “moved some stuff around” and we started having issues. They insisted that the virtualization was just moved to a different host, and it couldn’t be anything they did. Turned out that that a switch the new host was connected to had issues, giving us intermittent connectivity issues.

Run tracert to the App server from a client workstation.

2 Likes

If this helps:

  1. Hyper-V
  2. SSDs
  3. 64GB reserved static RAM
  4. 24 Cores, I just set them to 100% reserve now.
  5. 1 Virtual Switch, 1 VM. This is the only VM running on the brand new host we installed.
  6. Nothing
  7. This was a Microsoft Conversion to migrate the physical machine to a VM.

When ran from the server where the app server is, it runs slower. Which I find very weird.

Is SQL being hosted in the same virtualization as the App Server?

If not, are the virtualizations for each on the same host?

Yes, they are on the same virtualization.

  1. Hyper-V
  2. SSDs
  3. 64GB reserved static RAM
  4. 24 Cores, I just set them to 100% reserve now.
  5. 1 Virtual Switch, 1 VM. This is the only VM running on the brand new host we installed.
  6. Nothing
  7. This was a Microsoft Conversion to migrate the physical machine to a VM.

@aidacra Any input on this by chance?

Direct or NAS like QNAP?

Also can you give us more info on the Host? Dell, HP, Cisco? Blade System?

1 Like

Just curious … What’s a Virtual Switch? The host has many ethernet ports, and treats them like a switch before passing the info along to the virtualizations on the host?

And your App to SQL communications aren’t going through this, are they?

In Hyper-V is a virtual switch is how VMs connect to the physical network. You create a virtual switch and bind it to a physical port.

1 Like

How many VMs? 1 for SQL, 1 for App, 1 for task?