Update: There is now a PRB. Anybody else having this issue please log your cases and ask support to link them: PRB0321393 have been updated: PD System Number have been updated to: ERPS-314961
Its a clear cut EF Core compatibility issue and I have provided the full stack trace but Epicor support insists its not reproducible and refuses to even check with development about whether the compatibility issues will be addressed, and what version will have the patch. Has anybody else managed to get this one through to development (and has a PRB for it) so I can stop banging my head against the wall?
aidacra
(Epicor Employee: Nathan your friendly neighborhood Support Engineer)
4
Based on the information available so far, we have not been able to replicate the issue in our testing. However, that does not mean we are dismissing the concern.
If you can update the case with the additional details that were requested, we will continue working with you to investigate the behavior and better understand the conditions under which it occurs. The assigned Support Analyst remains engaged and is ready to assist in moving the investigation forward.
I shouldn’t have to run an hours long process again when the full stack trace has already been provided, how will that help? There are AI analyzers that you can run the code through and it will detect EF core compatibility issues. In fact, Epicor ran such an analyzer on our custom code. Not sure why it couldn’t be done against Epicor’s own base code as well?
Program Ice.Services.Lib.RunTask when executing task 277943 raised an unexpected exception with the following message: RunTask:
Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ThrowAggregateUpdateConcurrencyException(RelationalDataReader reader, Int32 commandIndex, Int32 expectedRowsAffected, Int32 rowsAffected)
at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeResultSet(Int32 startCommandIndex, RelationalDataReader reader)
at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.Consume(RelationalDataReader reader)
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.Execute(IRelationalConnection connection)
at Microsoft.EntityFrameworkCore.SqlServer.Update.Internal.SqlServerModificationCommandBatch.Execute(IRelationalConnection connection)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.Execute(IEnumerable`1 commandBatches, IRelationalConnection connection)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(IList`1 entriesToSave)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(StateManager stateManager, Boolean acceptAllChangesOnSuccess)
at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(Boolean acceptAllChangesOnSuccess)
at Microsoft.EntityFrameworkCore.DbContext.SaveChanges(Boolean acceptAllChangesOnSuccess)
at Ice.IceDataContext.SaveChanges(Boolean acceptAllChangesOnSuccess) in C:\_releases\ICE\ICE5.2.100.7\Source\Server\Framework\Epicor.System\Data\EntityFramework\EFCore\IceDataContext.cs:line 599
at Ice.IceDataContext.SaveChanges[TRow](TRow row) in C:\_releases\ICE\ICE5.2.100.7\Source\Server\Framework\Epicor.System\Data\EntityFramework\EFCore\IceDataContext.cs:line 538
at Erp.Internal.MR.MfgLeadTimeProc.calcMfgLeadTime() in C:\_releases\ERP\ERP12.2.100.0\Source\Server\Internal\MR\MfgLeadTimeProc\MfgLeadTimeProc.cs:line 1383
at Erp.Internal.MR.MfgLeadTimeProc.RunProcess(Int64 instanceTaskNum, String outputFileName) in C:\_releases\ERP\ERP12.2.100.0\Source\Server\Internal\MR\MfgLeadTimeProc\MfgLeadTimeProc.cs:line 571
at Ice.Hosting.TaskCaller.StartProcess(IceDataContext dataContext, Type myType, Operation operation) in C:\_releases\ICE\ICE5.2.100.7\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 123
at Ice.Hosting.TaskCaller.InnerExecuteTask(IceDataContext dataContext) in C:\_releases\ICE\ICE5.2.100.7\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 112
at Ice.Hosting.TaskCaller.ExecuteTask() in C:\_releases\ICE\ICE5.2.100.7\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 62
at Ice.Lib.RunTask.BpmFriendlyTaskLauncher.Run(String sessionIdPrefix, IceContext db, Action taskRunner) in C:\_releases\ICE\ICE5.2.100.0\Source\Server\Services\Lib\RunTask\BpmFriendlyTaskLauncher.cs:line 57
at Ice.Services.Lib.RunTaskSvc.InnerRunTask(Int64 ipTaskNum, Boolean suppressTransaction) in C:\_releases\ICE\ICE5.2.100.0\Source\Server\Services\Lib\RunTask\RunTask.cs:line 349
The failure is inside Epicor’s own base code, not in a customization:
Erp.Internal.MR.MfgLeadTimeProc.calcMfgLeadTime() — line 1383 of MfgLeadTimeProc.cs (ERP 12.2.100.0 / Kinetic 2026.1, ICE 5.2.100.7)
That line calls IceDataContext.SaveChanges<TRow>(row) — a single-row EF Core save. EF Core issues the UPDATE with the row’s original SysRevID in the WHERE clause (optimistic concurrency). It expected to update 1 row and updated 0, which means that by the time the process wrote the row, either:
another session/process had already updated that row (so SysRevID no longer matched), or
the row had been deleted, or
the process itself had already saved that same row earlier in the run and kept using the stale tracked entity, so its cached SysRevID was out of date.
The process has no concurrency handling around that save, so the first mismatch aborts the whole task instead of just that part.
What Epicor development needs to do
Catch DbUpdateConcurrencyException around the save at MfgLeadTimeProc.cs:1383 and handle it per the standard EF Core pattern — reload the entity from the database, re-apply the calculated lead-time values, and retry (bounded retries).
Re-fetch the row immediately before the update rather than writing back an entity that was loaded earlier in the run. Lead-time calculation over a large part set can run for many minutes; anything read up front is stale by the time it’s written.
Handle the deleted-row case explicitly — if the reload returns nothing, log and skip that part/plant instead of throwing.
Do not fail the entire task on one row. The process should log the offending Company/Part/Plant, continue, and report the skipped rows in the task log. Today a single collision kills the run and leaves lead times unpopulated.
Confirm the process is safe to run concurrently. If it isn’t, it needs a guard so two instances (or an overlapping scheduled run) can’t process the same parts.
I share your frustration - we are getting the same error running MRP. What is interesting is it hasn’t happened in our lower-level environments yet, so it was never identified in our pre-update testing. It is only affecting our Prod environment
We have only been ‘successful’ running MRP a handful of times since 7/20, and even those runs had failures with the scheduling processes.
I am having several issues with the scheduling process in MRP. Everything is fine until it goes to the scheduler then it errors out. On several of our parts it causes a hard stop in the system when running a net change for some reason. So MRP never completes. We ended up having to flag a bunch of parts to not run through MRP just to get it to complete. We still have schedule errors though in the log files of the runs. Still haven’t pinpointed what exactly is causing the issue with the parts that were locking the system entirely.
The problem with the concurrency error is that we don’t really know where it went wrong.
It goes something like this:
We query the record
Some other part of the code updates record
Now we try to update the record, we get an error due to stale data
We know when step 3 happens, it’s clear in the stack trace, what we don’t know is where step 2 takes place.
Adding the code to handle the exception is only a band aid that does not guarantee the error won’t popup somewhere else, and we can’t put this catch/retry logic everywhere.
Step 2 could be happening inside custom code, maybe some data directive, which won’t appear in the stack trace, which is why sometimes we can’t duplicate it in our own environments.
For this concurrency error its best to handle it as db specific if possible, that way we can just debug the code from start to the error and find the exact cause, making sure any other potential issue around the same code is fixed as well.
I know AI can be wrong, but conceptually those seemed like really solid suggestions for making this code less fragile, no? Why can’t it try/catch so a single failure doesn’t kill the entire process? And refetch right before updating? It seems like a millisecond window for potential conflicts would be much safer than an hours-long window?
Well, what logging level do you guys need to set so the server logs --or other logs-- WILL capture “step 2”? As SaaS customers we have zero control of this.
I guess I am struggling to see the point of reproducing the same error again without having made any changes or set up any additional logging, wouldn’t that just yield the same information we already have?
The suggested fix is good if we know why the error is happening in the first place, right know we know there is an error and where its thrown but don’t know why.
If we can’t replicate it we would just put the fix blindly and then once deployed to your environment something else could throw another error. We would need to put try/catch on every single SaveChanges which is not ideal.
I don’t think there is any that specific, though I know there is logging for bpms, so turning that on could provide more information. I’m not entirely familiar with all trace flags/logging.
FWIW, there are no bpms that are firing and causing the interference in this environment. Most likely in this case, it was a user performing some update or transaction that caused the conflict, which means the error would be intermittent. The point is that the code should not be so fragile that the entire process blows up when this happens. Likewise, development should be able to engineer their own scenario to interfere with the process in order to reproduce the issue, and then confirm the fix for the issue is successful, vs. putting the burden on the customer to prove that an issue exists.
