BPM runtime caught an unexpected in simple ChangeLog BPM

victorcarabez · November 8, 2023, 11:38pm

I am testing to apply an upgrade to Kinetic 2023.2.5, I have this error/exception in each table where I have a simple ChangeLog data directive, this occurs only in my “production” database, in the demo database the BPM works correctly :

## System Information ##
==================

AppServer Connection: https://myservername.mydomainname/KNTCLive
Form Name: AR Invoice Entry
Customization Name: csgInvoiceEntry
Menu ID: CSG121
Software Version: 11.2.400.0

============

Server Side Exception

BPM runtime caught an unexpected exception of 'BLException' type.
See more info in the Inner Exception section of Exception Details.

Exception caught in: Epicor.ServiceModel

## Error Detail ##
============

##!Correlation ID:##!  288481bb-6bae-45cd-a7c1-24a62658222b
##!Description:##!  BPM runtime caught an unexpected exception of 'BLException' type.
See more info in the Inner Exception section of Exception Details.
##!Inner Exception:##!  Error accessing the database: Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
Operation cancelled by user.
The statement has been terminated.
Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
Operation cancelled by user.
The statement has been terminated.
The wait operation timed out.
##!Program:##!  Epicor.Ice.dll
##!Method:##!  UpdateActivity
##!Line Number:##!  162
##!Column Number:##!  9
##!Server Trace Stack:##!     at Ice.Tracking.ActivityTracker.UpdateActivity(ActivityEvent activityEvent, SysActivityTracker sysActivityTracker) in C:\_releases\ICE\ICE4.2.400.5\Source\Server\Framework\Epicor.Ice\Tracking\ActivityTracker.cs:line 162
   at Epicor.Customization.Bpm.BpmActivityTracker.Log(DirectiveDescription directiveDesc, String userId, String activityId) in C:\_releases\ICE\ICE4.2.400.0FW\Source\Server\Internal\Lib\Epicor.Customization.Bpm\BpmActivityTracker.cs:line 29

Inner Trace:
Error accessing the database: Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
Operation cancelled by user.
The statement has been terminated.
:   at Ice.Lib.SolutionTracker.ExecSqlCommand(SqlCommand cmd, SqlConnection sqlCnx) in C:\_releases\ICE\ICE4.2.400.0FW\Source\Server\Internal\Lib\SolutionTracker\SolutionTracker.cs:line 83
   at Ice.Triggers.Default.WriteTrigger`2.Write(TRec newRecord, TRec oldRecord) in C:\_releases\ICE\ICE4.2.400.0FW\Source\Server\Db\Triggers\Default\Write.cs:line 22
   at Ice.DataTrigger`2.Ice.ILinqTrigger.Write(LinqRow newRecord, LinqRow oldRecord) in C:\_releases\ICE\ICE4.2.400.5\Source\Server\Framework\Epicor.Ice\Triggers\TriggerBase.cs:line 21
   at Ice.Triggers.TriggerQueue.ExecuteWriteTrigger(IceDataContext context, LinqRow modifiedRecord, LinqRow originalRecord) in C:\_releases\ICE\ICE4.2.400.0FW\Source\Server\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 305
   at Ice.Triggers.TriggerQueue.RunWriteTriggerInNewLevel(IceDataContext context, LinqRow modifiedRecord, LinqRow originalRecord, Boolean forAddedRow) in C:\_releases\ICE\ICE4.2.400.0FW\Source\Server\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 180
   at Ice.Triggers.TriggerQueue.<>c__DisplayClass9_0.<RunWriteTrigger>b__1() in C:\_releases\ICE\ICE4.2.400.0FW\Source\Server\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 147
   at Ice.Triggers.TriggerQueue.RunAtNewLevel(Func`1 buildTriggerRunState, Action action) in C:\_releases\ICE\ICE4.2.400.0FW\Source\Server\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 508
   at Ice.Triggers.TriggerQueue.RunTriggers(IceDataContext context) in C:\_releases\ICE\ICE4.2.400.0FW\Source\Server\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 76
   at Ice.IceDataContext.RunUntilAllTriggersHaveExecuted() in C:\_releases\ICE\ICE4.2.400.0FW\Source\Server\Framework\Epicor.System\Data\IceDataContext.cs:line 527
   at Ice.Triggers.TriggerQueue.RunAtNewLevel(Func`1 buildTriggerRunState, Action action) in C:\_releases\ICE\ICE4.2.400.0FW\Source\Server\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 508
   at Ice.IceDataContext.Validate[TLinqRow](TLinqRow row) in C:\_releases\ICE\ICE4.2.400.0FW\Source\Server\Framework\Epicor.System\Data\IceDataContext.cs:line 292
   at Ice.Tracking.ActivityTracker.UpdateActivity(ActivityEvent activityEvent, SysActivityTracker sysActivityTracker) in C:\_releases\ICE\ICE4.2.400.5\Source\Server\Framework\Epicor.Ice\Tracking\ActivityTracker.cs:line 162

Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
Operation cancelled by user.
The statement has been terminated.
:   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite, String methodName)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Ice.Lib.SolutionTracker.ExecSqlCommand(SqlCommand cmd, SqlConnection sqlCnx) in C:\_releases\ICE\ICE4.2.400.0FW\Source\Server\Internal\Lib\SolutionTracker\SolutionTracker.cs:line 83

The wait operation timed out.
:

 

## Client Stack Trace ##
==================
   at Ice.Cloud.ProxyBase`1.CallWithCommunicationFailureRetry(String methodName, ProxyValuesIn valuesIn, ProxyValuesOut valuesOut, RestRpcValueSerializer serializer)
   at Ice.Cloud.ProxyBase`1.CallWithMultistepBpmHandling(String methodName, ProxyValuesIn valuesIn, ProxyValuesOut valuesOut, Boolean useSparseCopy)
   at Ice.Cloud.ProxyBase`1.Call(String methodName, ProxyValuesIn valuesIn, ProxyValuesOut valuesOut, Boolean useSparseCopy)
   at Erp.Proxy.BO.ARInvoiceImpl.UpdateMaster(ARInvoiceDataSet ds, String cGroupID, String cTableName, Boolean runCheckAmortSched, Boolean runChkLtrOfCredit, Boolean& genAmortSched, Boolean lRASchedExists, Int32 ipInvoiceNum, Int32 ipInvoiceLine, String cARLOCID, Boolean suppressUserPrompts, Decimal dTotalCharges, Boolean lGetDefaultAccount, Decimal& grpTotalInvAmt, String& opGenMessage, String& opLtrCrdMsg, Boolean& lUpdateRan)
   at Erp.Adapters.ARInvoiceAdapter.UpdateMaster(String cGroupID, String cTableName, Boolean runCheckAmortSched, Boolean runChkLtrOfCredit, Boolean& genAmortSched, Boolean lRASchedExists, Int32 ipInvoiceNum, Int32 ipInvoiceLine, String cARLOCID, Boolean suppressUserPrompts, Decimal dTotalCharges, Boolean lGetDefaultAccount, Decimal& grpTotalInvAmt, String& opGenMessage, String& opLtrCrdMsg, Boolean& lUpdateRan)
   at Erp.Adapters.ARInvoiceAdapter.UpdateMasterUsingPasteInsert(String cGroupID, String cTableName, Boolean runCheckAmortSched, Boolean runChkLtrOfCredit, Boolean& genAmortSched, Boolean lRASchedExists, Int32 ipInvoiceNum, Int32 ipInvoiceLine, String cARLOCID, Boolean suppressUserPrompts, Decimal dTotalCharges, Boolean lGetDefaultAccount, Decimal& grpTotalInvAmt, String& opGenMessage, String& opLtrCrdMsg, Boolean& lUpdateRan, String pasteInsertColumnsToValidate)
   at Erp.UI.App.ARInvoiceEntry.Transaction.Update(Boolean refreshGroup)

## Inner Exception ##
===============
Unable to update activity with type: '1'. ActivityID: 'BPM' Key1: '00373518-19f7-4067-933e-5132d7841d16'

Fernando_Montoya · December 1, 2023, 8:20pm

same here, any solution found?

arrobatony · December 1, 2023, 8:27pm

Could be related with the size of your log data, with a timeout error and what you describe as your BPM only failing in live, I believe your SQL connection times out due to data transfer. Have you tried extending the time out parameters?

Fernando_Montoya · December 1, 2023, 8:34pm

Im on cloud!

Olga · December 1, 2023, 8:47pm

Did you report this? Do you have Activity Tracking on? you may try to switch it off for BPM

victorcarabez · December 1, 2023, 9:29pm

This happend me twice, in two test environment, same error, also in both cases the log file are less than 1 Gb.
I report the case to support, but currently is unsolved.

victorcarabez · December 1, 2023, 10:03pm

Yes, the Activity Tracking is enable in Company Maintenance, how I can disable it only for BPMs?

victorcarabez · December 1, 2023, 10:08pm

Not yet, sorry.

madeleinesheehan · December 5, 2023, 4:30am

The ‘System Activities’ flag in Company Maintenance controls the System Activity Log.

Your stack trace indicates the issue may originate from the new Activity Tracking feature.
Configured here:
System Management > Activity Tracking > Activity Type Maintenance

Are you able to navigate to that form and confirm the Activity with ID ‘BPM’ is enabled?
If so you should be able to edit that record and disable tracking for that specific activity type.

victorcarabez · December 5, 2023, 4:27pm

Thanks Madeleine, I will share this solution in the case that I previously generated with Epicor support and I will give you all the credit.

Evan_Purdy · December 11, 2023, 2:28pm

We are having this issue with a specific user as well. It seems like disabling a log is kind of a work around and not a solution. Is there a known bug being worked on?

madeleinesheehan · December 13, 2023, 4:20pm

Yes, this is actively being triaged by engineering.

Rick_Bird · January 23, 2024, 3:17pm

Recently completed Cloud Pilot upgrade to 2023.1.10.
A single user is getting the same type of error in Order Entry, other users do not.
I will open an Epicor Case on this… but it’s seems odd it’s user specific.
I am still investigating.

bahmpi · July 15, 2024, 5:16pm

We just upgraded from 2022.2.15 to 2023.2.14 and are experiencing the same timeout error referencing the BPM Activity Tracker. The error is affecting only two specific users on specific fields. Is a resolution still in the works at Epicor or is there a known fix? I couldn’t find anything in the EpiCare Knowledge Base on this issue.

bahmpi · July 15, 2024, 7:33pm

FYI - I put a case in for this and was informed that it is a known issue in the version we just upgraded to and it has been resolved in 2024.1. I was advised to disable the Activity BPM Tracker to solve the problem and that there is now issue with leaving it disabled. Of course, the only way to fix it is to upgrade again…

Rick_Bird · April 3, 2025, 7:32pm

Just upgraded to 2024.2 and now some users are getting these errors related to BPM Activity Tracking being enabled.
So not fixed?

llv · November 24, 2025, 9:46am

Try selecting the blank Solution Tracking.