Transaction deadlock issues

kfierce · March 9, 2021, 7:56pm

We are seeing a number of users receiving server error and when checking the server log there are Transaction deadlock errors. This is our busy season and it is frustrating to our users and if there are anything we can do to minimize these types of errors let me know. We are on prem 10.2.600.9.

Any seen or have similar issues or have any suggestions to what/where I should go next with this one.

We have seen this in Job Entry, PO Receipts, Sales Order Entry lately. The users can redo their transaction after getting the server error and move on, however this is frustrating for everyone.

<![CDATA[System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while reading from the store provider’s data reader. See the inner exception for details. —> System.Data.SqlClient.SqlException: Transaction (Process ID 241) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

Jonathan · March 9, 2021, 9:08pm

From the server error stack trace can you see if this comes from a single point, like a shared library or is it triggered from multiple libraries/business objects?

I would advise to start a SQL Profile session to capture the lock events and get more information as to what tables are being locked and by which process.

When these errors happen are you also executing long running tasks like MRP or WIP/COS?

kfierce · March 9, 2021, 9:46pm

Is it triggered from multiple BOs - If you mean do I see the error at the same time from multiple BOs - yes. For example today was BO.Job/Entry/JobEntrySvcContract/Update and at the same time with milliseconds difference BO.SalesOrder/SalesOrderSvcContract/MasterUpdate

MRP and COS/WIP are not running when we see the deadlocks.

we have started a SQL Profile sessions looking for deadlocks.

Jonathan · March 10, 2021, 12:30am

Did this start recently? Anything else running in the background that might cause blocking?

Do these transactions have taxes?

Can you share the entire stack trace or logs for the errors? Knowing the exact point or call made before the error would help narrow the source.

Hally · March 10, 2021, 1:10am

Hello @kfierce, sounds like @Jonathan is out there to help you, but I thought I’d throw in my 2c and tackle it from the sql end of things.

You mention it’s your busy season.
You didn’t mention if you are running split SQL or all on the same server.
You did mention that there are milliseconds between the requests that are hitting the same table.

I am guessing is that you might have unusually high disk queue lengths in your SQL server as a result of increased load. It may be related to various things, example disk performance, how you have your data and log files stored, memory usage (virtual disk size and location). Possibly even network latency.

The other area might be worth looking at are the indexes, table size, statistics and fragmentation on the related tables.

The other thing that ii might be worth checking is if there are any BPMs on the first method request when the deadlock occurs. There may be some latency introduced there that might be causing deadlocks when the system is under load.

Hope that helps.

Henry_Burke · March 10, 2021, 2:07am

Hi Kristine,
We’ve had similar experiences with SQL performance. In our case, the issue was largely the result of ‘excessive’ use of change logging… essentially too many fields logged on key tables with standard and UD fields. One revelation our consultant shared is that for a single user level ‘transaction’, we see in SQL both the standard transactions but often additional method and data directives firing so the single user transaction really invokes any number of changes on fields being tracked - in some cases it is a 10:1 or 13:1 ratio of record and field access. Our blocking/locking rate in SQL went through the roof.
We depend on DMT, customizations on Allocations, inventory transactions, receiving transactions, picking, custom QA processes, and packing transactions all touching the part lot level and it was simply a train wreck. Does this have any applicability in your context?

The most interesting aspect is that in testing (at low volume) we saw none of this and frankly were not looking either. It was only at actual production rates of transactions that the train wrecked. Lesson learned on all accounts - change logs cost a lot, especially with emphasis or high dependency on key tables like Part Lot, and there is no substitute for “at expected volume/rate” testing (hard as it is to actually accomplish with busy users).

GaryWarn · March 10, 2021, 12:41pm

Hi Kristine,
Rather than use SQL profiler you can used ‘Extended Events’ to record instances of Deadlocks - actually there is a ‘default’ extended event that already does this but doesn’t necessarily give you enough detail. If you’re able to find the SQL behind the deadlocks you could run in SSMS and show the execution plan, which may in-turn reveal that something is amiss - new index required or running of update statistics,
Extended Events can be found in SQL Server Management Studio under the ‘Management’ folder.

hkeric.wci · March 10, 2021, 12:43pm

In our case it was UD Columns existing on JobAsmbl, JobMtl and the page-fill factor for _UD was way different than regular table, once we matched the page-fill factor after many emails with Epicor’s Raj Tapde, we eliminated many deadlocks.

kfierce · March 10, 2021, 7:02pm

Here is the complete log for the two deadlocks from yesterday.
We have had some errors before. We have increased order and production levels in our busy season so we are seeing this error more frequently than before and it is causing more frustration when you are busy and want to get thing done.

I don’t think the sales order would have taxes, most our customers are tax exempt we do have Avalara and it’s used for only 2-3 customer accounts. This users does not normally enter any taxable sales.

<Op Utc="2021-03-09T17:58:22.9283652Z" act="Erp:BO:JobEntry/JobEntrySvcContract/Update" correlationId="073af7aa-cbbf-4a98-b737-341ddbc85aaf" dur="93789.9916" cli="xxxxxxxxx" usr="ipostma" machine="EPIAPP-A" pid="8812" tid="117">
  <Sql queries="102" cacheHits="9" time="91280.0005" qryTypeCount="49"/>
  <DBExpressionCompiler msg="Query failed to run&#xD;&#xA;System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while reading from the store provider's data reader. See the inner exception for details. ---&gt; System.Data.SqlClient.SqlException: Transaction (Process ID 171) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.&#xD;&#xA;   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)&#xD;&#xA;   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)&#xD;&#xA;   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean&amp; dataReady)&#xD;&#xA;   at System.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean&amp; moreRows)&#xD;&#xA;   at System.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean&amp; more)&#xD;&#xA;   at System.Data.SqlClient.SqlDataReader.Read()&#xD;&#xA;   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.StoreRead()&#xD;&#xA;   --- End of inner exception stack trace ---&#xD;&#xA;   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.HandleReaderException(Exception e)&#xD;&#xA;   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.StoreRead()&#xD;&#xA;   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.SimpleEnumerator.MoveNext()&#xD;&#xA;   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)&#xD;&#xA;   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)&#xD;&#xA;   at Epicor.Data.DBExpressionCompiler.GetResult[TDataContext,TQuery,TResult](Func`3 executeQuery, Cache cacheSetting, TDataContext dataContext, TQuery query) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs:line 429&#xD;&#xA;   at Epicor.Data.DBExpressionCompiler.InvokeList[TDataContext,TQuery,TResult](Expression expression, Cache currentCacheSetting, Boolean cacheQuery, TDataContext dataContext, Func`2 getDataCacheKey, Func`2 compileQuery, Func`3 executeQuery) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs:line 155&#xD;&#xA;(ctx, company_ex, classifier_ex, jobNum_ex) =&gt; ctx.PartSug.With(UpdLock).Where(row =&gt; (((row.Company == company_ex) AndAlso (row.Classifier == classifier_ex)) AndAlso (row.JobNum == jobNum_ex)))" />
  <BpmCustomization Source="BO" BpMethodCode="Erp.JobEntry.Update" Type="BO Customization" Duration="0">
    <BpmDirective Type="1" ID="c46a0ccc-f142-48e0-b1a3-8985696c83bb" Name="JobProdCodeVerification" VisibilityScope="0" Duration="0" />
  </BpmCustomization>
  <Exception><![CDATA[System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while reading from the store provider's data reader. See the inner exception for details. ---> System.Data.SqlClient.SqlException: Transaction (Process ID 171) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean& moreRows)
   at System.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more)
   at System.Data.SqlClient.SqlDataReader.Read()
   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.StoreRead()
   --- End of inner exception stack trace ---
   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.HandleReaderException(Exception e)
   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.StoreRead()
   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.SimpleEnumerator.MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Epicor.Data.DBExpressionCompiler.GetResult[TDataContext,TQuery,TResult](Func`3 executeQuery, Cache cacheSetting, TDataContext dataContext, TQuery query) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs:line 429
   at Epicor.Data.DBExpressionCompiler.InvokeList[TDataContext,TQuery,TResult](Expression expression, Cache currentCacheSetting, Boolean cacheQuery, TDataContext dataContext, Func`2 getDataCacheKey, Func`2 compileQuery, Func`3 executeQuery) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs:line 160
   at Epicor.Data.DBExpressionCompiler.<>c__DisplayClass3_0`5.<Compile>b__0(TContext context, TArg1 arg1, TArg2 arg2, TArg3 arg3) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Data\DBExpressionCompiler.Generated.cs:line 111
   at Erp.Internal.JC.JcSynjob.SelectPartSugWithUpdLock(String company, String classifier, String jobNum) in C:\_Releases\ERP\RL10.2.600.0\Source\Server\Internal\JC\JCSynJob\JCSynJob.Queries.cs:line 516
   at Erp.Internal.JC.JcSynjob._JcSynjob(String p_oldjob, String p_newjob) in C:\_Releases\ERP\RL10.2.600.0\Source\Server\Internal\JC\JCSynJob\JCSynJob.cs:line 81
   at Erp.Services.BO.JobEntrySvc.JobHeadBeforeUpdate() in C:\_Releases\ERP\RL10.2.600.0\Source\Server\Services\BO\JobEntry\JobEntry.cs:line 22992
   at Ice.Services.Trace.TablesetProfilingCollector.DoRowEventTrace(String tableName, String methodName, Int32 rowCount, Action action) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.Ice\Services\TablesetProfilingCollector.cs:line 140
   at Ice.TablesetBound`3.UpdateRow(IceDataContext dataContext, Int32 tableNum, IIceTable table, IceRow updatedRow, IceRow originalRow, IColumnUncensor uncensor, TablesetProfilingCollector parentTraceCollector) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.Ice\Services\TablesetBound.cs:line 1283
   at Ice.TablesetBound`3.WriteTable(IceDataContext dataContext, Int32 tableIndex, IIceTable table, TablesetProfilingCollector parentTraceCollector) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.Ice\Services\TablesetBound.cs:line 980
   at Ice.TablesetBound`3.InnerUpdate[TUpdater](IceDataContext dataContext, TFullTableset tableset) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.Ice\Services\TablesetBound.cs:line 872
   at Erp.Services.BO.JobEntrySvc.Update(JobEntryTableset& ds) in C:\_Releases\ERP\RL10.2.600.0\Source\Server\Services\BO\JobEntry\JobEntry.Designer.cs:line 7443
   at Epicor.Customization.Bpm.MethodCustomizationBase2`3.RunDirectives(TParam parameters) in C:\_Releases\ICE\RL10.2.600.0FW\Source\Server\Internal\Lib\Epicor.Customization.Bpm\MethodCustomizationBase2.cs:line 202
   at Epicor.Customization.Bpm.CustomizationBase2`3.Execute(TParam parameters) in C:\_Releases\ICE\RL10.2.600.0FW\Source\Server\Internal\Lib\Epicor.Customization.Bpm\CustomizationBase2.cs:line 87
   at Epicor.Customization.Bpm.BO6124EAF971D044B7A46C98214F058C7D.JobEntrySvcCustomization.Update(JobEntryTableset& ds)
   at Erp.Services.BO.JobEntrySvcFacade.Update(JobEntryTableset& ds) in C:\_Releases\ERP\RL10.2.600.0\Source\Server\Services\BO\JobEntry\JobEntrySvcFacade.cs:line 9858
   at SyncInvokeUpdate(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at Epicor.Hosting.OperationBoundInvoker.InnerInvoke(Object instance, Func`2 func) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Hosting\OperationBoundInvoker.cs:line 59
   at Epicor.Hosting.OperationBoundInvoker.Invoke(Object instance, Func`2 func) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Hosting\OperationBoundInvoker.cs:line 47
   at Epicor.Hosting.Wcf.EpiOperationInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Hosting\Wcf\EpiOperationInvoker.cs:line 23
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)]]></Exception>
</Op>


<Op Utc="2021-03-09T17:58:19.9439924Z" act="Erp:BO:SalesOrder/SalesOrderSvcContract/MasterUpdate" correlationId="4be24916-266b-4ac6-9091-5e3dafe6331c" dur="96802.4775" cli="xxxxxxx" usr="mpovah" machine="EPIAPP-A" pid="8812" tid="169">
  <Sql queries="8" cacheHits="4" time="8.2279" qryTypeCount="5"/>
  <DBExpressionCompiler msg="Query failed to run&#xD;&#xA;System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while reading from the store provider's data reader. See the inner exception for details. ---&gt; System.Data.SqlClient.SqlException: Transaction (Process ID 241) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.&#xD;&#xA;   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)&#xD;&#xA;   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)&#xD;&#xA;   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean&amp; dataReady)&#xD;&#xA;   at System.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean&amp; moreRows)&#xD;&#xA;   at System.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean&amp; more)&#xD;&#xA;   at System.Data.SqlClient.SqlDataReader.Read()&#xD;&#xA;   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.StoreRead()&#xD;&#xA;   --- End of inner exception stack trace ---&#xD;&#xA;   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.HandleReaderException(Exception e)&#xD;&#xA;   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.StoreRead()&#xD;&#xA;   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.SimpleEnumerator.MoveNext()&#xD;&#xA;   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)&#xD;&#xA;   at System.Data.Entity.Core.Objects.CompiledQuery.ExecuteQuery[TResult](ObjectContext context, Object[] parameterValues)&#xD;&#xA;   at System.Data.Entity.Core.Objects.CompiledQuery.Invoke[TArg0,TArg1,TResult](TArg0 arg0, TArg1 arg1)&#xD;&#xA;   at Epicor.Data.DBExpressionCompiler.&lt;&gt;c__DisplayClass33_1`3.&lt;Compile&gt;b__3(TContext c, Func`3 query) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Data\DBExpressionCompiler.Generated.cs:line 1089&#xD;&#xA;   at Epicor.Data.DBExpressionCompiler.GetResult[TDataContext,TQuery,TResult](Func`3 executeQuery, Cache cacheSetting, TDataContext dataContext, TQuery query) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs:line 445&#xD;&#xA;   at Epicor.Data.DBExpressionCompiler.InvokeSingle[TDataContext,TQuery,TResult](Expression expression, Cache currentCacheSetting, Boolean cacheQuery, TDataContext dataContext, Func`2 getDataCacheKey, Func`2 compileQuery, Func`3 executeQuery) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs:line 301&#xD;&#xA;(context, sysRowID_ex) =&gt; context.OrderHed.With(UpdLock).Where(row =&gt; (row.SysRowID == sysRowID_ex)).FirstOrDefault()" />
  <BpmCustomization Source="BO" BpMethodCode="Erp.SalesOrder.Update" Type="BO Customization" Duration="189">
    <BpmDirective Type="1" ID="daf51a3e-bdef-abac-e411-88cd8b8fa46f" Name="ShipToCountryCheck" VisibilityScope="0" Duration="187" />
    <BpmDirective Type="1" ID="f5306fea-1e06-b99c-e611-3000a48cb3eb" Name="ProtectDealerCenterOrder" VisibilityScope="0" Duration="2" />
    <BpmDirective Type="1" ID="c297352b-c61d-442d-a49e-45f983d7ad68" Name="SetAGLListPrice" VisibilityScope="1" Duration="0" />
  </BpmCustomization>
  <BpmCustomization Source="BO" BpMethodCode="Erp.SalesOrder.MasterUpdate" Type="BO Customization" Duration="53">
    <BpmDirective Type="1" ID="90319283-5a56-c6a6-e311-a6a0b320c355" Name="Restrict Sales by ShipTo Country" VisibilityScope="0" Duration="0" />
    <BpmDirective Type="1" ID="9b79dd0a-6bb5-0d95-e411-48a3cb3a27f7" Name="CheckDealerAgreement" VisibilityScope="0" Duration="30" />
    <BpmDirective Type="1" ID="abed5be7-35c8-5d95-e511-05fd320e7426" Name="ProtectDealerCenterOrders" VisibilityScope="0" Duration="2" />
    <BpmDirective Type="1" ID="62c24bdb-ed65-4642-b722-852eea6a7ed8" Name="NewLeaderPriceCheck" VisibilityScope="0" Duration="4" />
    <BpmDirective Type="1" ID="1377194d-ea66-44c5-8913-766ad441293b" Name="PartSaleRestriction" VisibilityScope="0" Duration="0" />
    <BpmDirective Type="1" ID="545192ce-0754-4ade-84f3-876fabc33010" Name="SetPriority" VisibilityScope="0" Duration="0" />
    <BpmDirective Type="1" ID="28fba5a9-1312-46f8-9614-f2dbae06e08d" Name="CounterSaleCheckNeedByShipBy" VisibilityScope="0" Duration="0" />
    <BpmDirective Type="1" ID="4196989e-0e8e-4ee2-a70a-1b590a90be67" Name="OffHoldMessages" VisibilityScope="0" Duration="15" />
    <BpmDirective Type="1" ID="9205e765-85fe-41f7-9d76-416ddcffbb17" Name="SyncOrderHedDates" VisibilityScope="0" Duration="0" />
  </BpmCustomization>
  <Exception><![CDATA[System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while reading from the store provider's data reader. See the inner exception for details. ---> System.Data.SqlClient.SqlException: Transaction (Process ID 241) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean& moreRows)
   at System.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more)
   at System.Data.SqlClient.SqlDataReader.Read()
   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.StoreRead()
   --- End of inner exception stack trace ---
   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.HandleReaderException(Exception e)
   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.StoreRead()
   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.SimpleEnumerator.MoveNext()
   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
   at System.Data.Entity.Core.Objects.CompiledQuery.ExecuteQuery[TResult](ObjectContext context, Object[] parameterValues)
   at System.Data.Entity.Core.Objects.CompiledQuery.Invoke[TArg0,TArg1,TResult](TArg0 arg0, TArg1 arg1)
   at Epicor.Data.DBExpressionCompiler.<>c__DisplayClass33_1`3.<Compile>b__3(TContext c, Func`3 query) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Data\DBExpressionCompiler.Generated.cs:line 1089
   at Epicor.Data.DBExpressionCompiler.GetResult[TDataContext,TQuery,TResult](Func`3 executeQuery, Cache cacheSetting, TDataContext dataContext, TQuery query) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs:line 445
   at Epicor.Data.DBExpressionCompiler.InvokeSingle[TDataContext,TQuery,TResult](Expression expression, Cache currentCacheSetting, Boolean cacheQuery, TDataContext dataContext, Func`2 getDataCacheKey, Func`2 compileQuery, Func`3 executeQuery) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs:line 306
   at Epicor.Data.DBExpressionCompiler.<>c__DisplayClass33_0`3.<Compile>b__0(TContext context, TArg1 arg1) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Data\DBExpressionCompiler.Generated.cs:line 1082
   at Erp.Tables.OrderHed.FindFirstBySysRowIDWithUpdLock(ErpContext dataContext, Guid sysRowID) in C:\_Releases\ERP\RL10.2.600.0\Source\Server\Db\Erp.Data.910100\ErpContext.Model.cs:line 1780866
   at Erp.Services.BO.SalesOrderSvc.GetDBRowWithUpdLock(Int32 tableNumber, Guid rowID) in C:\_Releases\ERP\UD10.2.600.9\Source\Server\Services\BO\SalesOrder\SalesOrder.Designer.cs:line 407
   at Ice.TablesetBound`3.UpdateRow(IceDataContext dataContext, Int32 tableNum, IIceTable table, IceRow updatedRow, IceRow originalRow, IColumnUncensor uncensor, TablesetProfilingCollector parentTraceCollector) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.Ice\Services\TablesetBound.cs:line 1256
   at Ice.TablesetBound`3.WriteTable(IceDataContext dataContext, Int32 tableIndex, IIceTable table, TablesetProfilingCollector parentTraceCollector) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.Ice\Services\TablesetBound.cs:line 980
   at Ice.TablesetBound`3.InnerUpdate[TUpdater](IceDataContext dataContext, TFullTableset tableset) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.Ice\Services\TablesetBound.cs:line 872
   at Erp.Services.BO.SalesOrderSvc.Update(SalesOrderTableset& ds) in C:\_Releases\ERP\UD10.2.600.9\Source\Server\Services\BO\SalesOrder\SalesOrder.Designer.cs:line 3852
   at Epicor.Customization.Bpm.MethodCustomizationBase2`3.RunDirectives(TParam parameters) in C:\_Releases\ICE\RL10.2.600.0FW\Source\Server\Internal\Lib\Epicor.Customization.Bpm\MethodCustomizationBase2.cs:line 202
   at Epicor.Customization.Bpm.CustomizationBase2`3.Execute(TParam parameters) in C:\_Releases\ICE\RL10.2.600.0FW\Source\Server\Internal\Lib\Epicor.Customization.Bpm\CustomizationBase2.cs:line 87
   at Epicor.Customization.Bpm.BO33DBB11AE5174058BB5DDD3C9BA1CFD3.SalesOrderSvcCustomization.Update(SalesOrderTableset& ds)
   at Erp.Services.BO.SalesOrderSvcFacade.Update(SalesOrderTableset& ds) in C:\_Releases\ERP\UD10.2.600.9\Source\Server\Services\BO\SalesOrder\SalesOrderSvcFacade.cs:line 10205
   at Erp.Services.BO.SalesOrderSvc.MasterUpdate(Boolean lCheckForOrderChangedMsg, Boolean lcheckForResponse, String cTableName, Int32 iCustNum, Int32 iOrderNum, Boolean lweLicensed, Boolean& lContinue, String& cResponseMsg, String& cCreditShipAction, String& cDisplayMsg, String& cCompliantMsg, String& cResponseMsgOrdRel, String& cAgingMessage, SalesOrderTableset& ds) in C:\_Releases\ERP\UD10.2.600.9\Source\Server\Services\BO\SalesOrder\SalesOrder.cs:line 18777
   at Epicor.Customization.Bpm.BOE8F77E8B77DE433082B5990871F3D0F4.MasterUpdateImpl.ExecuteBase(MasterUpdateParameters parameters)
   at Epicor.Customization.Bpm.MethodCustomizationBase2`3.RunDirectives(TParam parameters) in C:\_Releases\ICE\RL10.2.600.0FW\Source\Server\Internal\Lib\Epicor.Customization.Bpm\MethodCustomizationBase2.cs:line 202
   at Epicor.Customization.Bpm.CustomizationBase2`3.Execute(TParam parameters) in C:\_Releases\ICE\RL10.2.600.0FW\Source\Server\Internal\Lib\Epicor.Customization.Bpm\CustomizationBase2.cs:line 87
   at Epicor.Customization.Bpm.BOE8F77E8B77DE433082B5990871F3D0F4.SalesOrderSvcCustomization.MasterUpdate(Boolean lCheckForOrderChangedMsg, Boolean lcheckForResponse, String cTableName, Int32 iCustNum, Int32 iOrderNum, Boolean lweLicensed, Boolean& lContinue, String& cResponseMsg, String& cCreditShipAction, String& cDisplayMsg, String& cCompliantMsg, String& cResponseMsgOrdRel, String& cAgingMessage, SalesOrderTableset& ds)
   at Erp.Services.BO.SalesOrderSvcFacade.MasterUpdate(Boolean lCheckForOrderChangedMsg, Boolean lcheckForResponse, String cTableName, Int32 iCustNum, Int32 iOrderNum, Boolean lweLicensed, Boolean& lContinue, String& cResponseMsg, String& cCreditShipAction, String& cDisplayMsg, String& cCompliantMsg, String& cResponseMsgOrdRel, String& cAgingMessage, SalesOrderTableset& ds) in C:\_Releases\ERP\UD10.2.600.9\Source\Server\Services\BO\SalesOrder\SalesOrderSvcFacade.cs:line 6185
   at SyncInvokeMasterUpdate(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at Epicor.Hosting.OperationBoundInvoker.InnerInvoke(Object instance, Func`2 func) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Hosting\OperationBoundInvoker.cs:line 59
   at Epicor.Hosting.OperationBoundInvoker.Invoke(Object instance, Func`2 func) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Hosting\OperationBoundInvoker.cs:line 47
   at Epicor.Hosting.Wcf.EpiOperationInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs) in C:\_Releases\ICE\UD10.2.600.9FW\Source\Framework\Epicor.System\Hosting\Wcf\EpiOperationInvoker.cs:line 23
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)]]></Exception>
</Op>

kfierce · March 10, 2021, 7:07pm

Hi @Hally! We have one SQL server. Thanks for the suggestions. I am asking the people that knows more about SQL than me. I added data from our server log in another response with the hope someone can decipher better than me what it means.

Jonathan · March 10, 2021, 7:23pm

I see some BPMs are executed, are you triggering Business Objects Updates from within the BPMs?

kfierce · March 10, 2021, 9:37pm

@Jonathan - yes we do in multiple BPMs. All the BPMs will fire, some will execute all the logic, some will execute none based off of conditions

Jonathan · March 10, 2021, 11:25pm

I would say that somehow those BPMs are causing conflicts triggering additional Updates.