I’ve been working with a customer on and off for years, and there’s always been some background ‘noise’ about performance. It was annoying when needing somewhere to test changes, but never enough to justify dedicated time to fix. Until recently. Dev and Test have app servers on the same machine, databases on the same SQL instance. MRP in Dev takes 50% longer than Test. ‘Sometimes’. The users do say sometimes they see Dev runs almost the same as Test. ‘Sometimes’.
I checked BPM, admin console, webconfig, client and server installs, task agent, system agent. I captured server logs, trace logs, checked event viewer and PDT. Nothing. Dev is still slow. On the SQL server I moved and corrected names of db files, rebuilt indexes, shrinkenified log files. All purposeful and satisfying, but Test and Dev were roughly the same. And Dev is still slow.
-- Here a lot of head scratching happened, until–
Dev and Test run the IIS app pools under different service accounts and write their server and client logs to slightly different places. I looked at the event viewer again, checking Windows Logs–>Security. And when running MRP with logging turned on in Dev, there were ~1000 security audit events per second. Only for that service account, only when logging was being used. Hence the ‘sometimes’ nature of the issue in Dev. Solved!
If this saves anyone as much time as I lost on this, it’s worth the post.
Interesting
We run MRP with logging (always) in dev and prod. I’ve never noticed the security logs. Going to have to check now.
We don’t notice a performance difference either though. What version of ERP/Kinetic?
This ones a bit older - I want to say 10.2.600 but don’t hold me to that.
I haven’t had time to investigate what needs to be configured to prevent the audit events being created, but worth checking your event viewer(s). But check it when MRP is running. These guys had a limit of ~22,000 events and that fills up and overwrites very fast. Even at quiet times there are 5 or 10 events per second.
If I hadn’t looked when MRP was running, it’s likely I’d have missed it. And what happens with user tracing logs? Or server logs, or Mfg Lead Time logs etc.
Likely need someone better than me on Active Directory and security audit configuration. Can we exclude by folder, by user, by user login type?
On 2024.2 I don’t see the flood of security audit events while MRP is running. I do see a floor of them from some other process. But not during MRP.
Epicor has significantly overhauled MRP in later versions of Kinetic. It’s far better than it’s ever been. I’m wondering if they fixed the messy logging. Looks like other processes aren’t cleaned up though.
It’s possible. In this case, it didn’t affect one environment because it didn’t affect the service account that is writing those MRP logs. One account can apparently write the MRP log files without creating masses and masses of audit events, but the other account - which we thought was ‘identical’ - is somehow different. I’ll post some screenshots later - I’ll just anonymise them first.
If you’re not seeing those audit events, that’s good! But it’s good to KNOW that’s the case.
This is a very current investigation, so I’ll update as I find out more. Right now, I don’t even know what makes the two accounts different.
Screen shots of the messages. I got ~150,000 events in about 2 minutes, well over my 1000 per second guess, and it seems to get slow when it reaches the “Maximum Log Size” and has to start overwriting the old events.
Resource Monitor shows the disk activity is…a lot. I’m pretty sure the read/write hit ludicrous speed for a few seconds there. Active time on C:\ hit 99% frequently, and the disk queue length got to ~1.5 or more several times. And most of that means very little to me. But when MRP stops, it’s like the million voices that were crying out are suddenly silenced.
Running MRP without logging means the disk activity barely registers, but network traffic is higher…likely a good sign.
That’s very interesting about using the local path instead of the ‘Share’ path. I hadn’t considered the configuration of the System Agent data paths.
Maybe it is not the fact that the two environments use different service accounts, but a difference in the System Agent configuration in each environment.
"if auditing must stay on, writing MRP logs to a local path instead of through the share name would avoid the SMB share audit path entirely.
Instead of (UNC/share)
\SERVERNAME\EpicorData\Companies\GSGR\Log\MRP.log
Use (local)
C:\EpicorData\Companies\GSGR\Log\MRP.log"
Yep, spot on! Thank you so much - it isn’t the Service Accounts, it is accessing the “Server Data Directory” using a UNC/Share path instead of a local path.
And this might apply to the Live/Production environment, which would be a good win!
I wouldn’t want to have to track down where different logs are. We have multiple task agents per app server cluster. So we use central log directory to keep things clean. The local path would be all over different servers if we didn’t use UNC paths.
For us, all Windows logs get sucked into a SIEM and then parsed and handled by the SIEM. So noisy audit logs about SMB are discarded and irrelevant for us.
@Hally Think we are all agreeing on the same thing. If all in one use local. If you have mulitple servers use UNC. If using UNC turn off the auditing if the event log writting is an issue.
