So we had an interesting situation arise today that I haven’t run into before , I am wondering if anyone has seem something similar or if there’s some rational explanation for it.
All day long our users have been bombarding us with Epicor performance issues particular AR / Cash Recepipt everything was taking “for ever”
We were able to verify this, going into a Cash Receipt group was taking upwards of 45 seconds and then just clicking around the group or selecting invoices could take minutes and even over an hour before we aborted and bailed.
I looked at the SQL Server load and it was/is nominal, same for all app servers. Regardless we recycled all the app servers to see if things would improve. (It didn’t)
I did some tracing and saw there was a huge spike on the time call for CashRec.CashRecGetInvoices
After much digging and digging we found nothing. Out of curiosity backed up live and restored it into Test and just like magic the performance issue was gone in test. That means it wasn’t a data / index issue it had to be some sort of appserver issue but we had recycled the appservers.
Out of frustration I decided to hookup the test instance to a debugger to see if I could see anything weird, and I saw that there was a BPM being called on CashRecGetInvoices. This BPM has been in place for over 3 years without issue.
However as a hail mary, I disabled this BPM in live and BOOOOOOOOM everything sped right up, the weird bit is that I then re-enabled the BPM and BOOOOOOOM everything is running like a dream.
The act of enabling / disabling the BPM somehow “unstuck” whatever was stuck in the appservers and everything is running okay now. I know BPMs generate DLLs which get loaded dynamically into the appserver so it would make sense how disabling / enabling the BPM would have caused something to get unstuck.
However I can’t explain what, how could a BPM that has been in place for years all of the sudden go “corrupted” and how does even recycling the app server not help.
Any ideas? I’m trying hard to come up with an explanation that I can be okay with…
I’m out of ideas.