Kinetic update broke LaborHed/LaborDtl creation from code, has the required structure changed?

Kinetic ERP
, , ,
21

This is an in-transaction directive. I have asked for clarity from our team as to if it is possible to move these to post transaction but I am not clear on that yet.

22

Holy :safe_harbor: batman!

23

Pretty sure we need to keep this as In-Trans since it works of the ttShipHed that’s passed to it. We need to assign all the LaborDtl records before it commits since the action is fired on the “Ready To Invoice” call which will start a lot of downline processes that need to have everything fleshed out (like labor quantities).

If the labor routine fails we want atomicity so the “Ready To Invoice” will uncheck itself.

24

Moving to a Pre-Processing directive might be your best bet then. Might want to replicate the labor entry portion as a function and activate with its own button click/application studio event call. Would help determine if its a problem with the code or the BPM trigger point.

25

To be fair there are workarounds, but they feel dirty like a hack. Maybe Epicor needs to go back to Db.Validate and create a seperate Context everywhere.

Id be curious if they uplifted their widgets. Remember they always say Widgets = Upgrade Proof

Adding .AsNoTracking() may be good for reading… But what if you want to update some UD Field.

Epicor fixed their MRP by doing:

Db.ReadCurrent(ref JCSyst, LockHint.UpdLock); // Perhaps NoLock can be used for reading

@JeffLeBert wrote a good uplift guide for MARS reasons… But the whole Entity Framework Tracking and how to handle BPMs, I think doesn’t exist (yet).

26

:safe_harbor: SAFE HARBOR!!! :safe_harbor: I am NOT an EF Expert, nor have i consulted with any within the Epicor team yet…
ALSO.. I This is the first time i have see the .as no tracking part of a linq
That said, I took the first comment someone made about this, and i asked our internal Enterprise level ChatGPT which has lots of Epicor Kinetic knowledge built into it, about the command. Here is what it replied.

AsNoTracking() tells Entity Framework to retrieve the JobHead records without adding them to the change tracker.

Db.JobHead.AsNoTracking()

Why use it

Use it for read-only queries where you do not intend to modify and save the returned entities.

Benefits:

  • Lower memory usage
  • Less Entity Framework bookkeeping
  • Often faster query materialization
  • Avoids returning an already-tracked, potentially stale entity from the current DbContext

Example:

var job = Db.JobHead
    .AsNoTracking()
    .FirstOrDefault(x =>
        x.Company == Session.CompanyID &&
        x.JobNum == jobNum);

Important consequence

Changes made to the returned object are not automatically persisted:

job.JobClosed = true;
Db.SaveChanges(); // Does not update JobHead

Because job is detached from the context.

Without AsNoTracking(), Entity Framework tracks the entity and can generate an update when SaveChanges() is called.

In an Epicor BPM

AsNoTracking() is appropriate when:

  • Checking whether a record exists
  • Reading values for validation or calculation
  • Building a lookup or result set
  • You do not need to update that retrieved row

Do not use it when you expect to modify the retrieved JobHead row and have the BPM persist that change through normal Entity Framework change tracking.

27

So what is probably happening in the very simple No Updating BPM above is that because it is being queried without .AsNoTracking() the BASE then somewhere does a Db.SaveChanges and you are basically a side-effect, or some other BPM also queries JobHead without .AsNoTracking() or no Select(x => new { only columns you need });

If I were to have to do an Update in EF Core I would do something like:
Not that you should update it bypassing the BOs, just for example sake

using (var context = Ice.Services.ContextFactory.CreateContext<ErpContext>())
{
    foreach (var plant in context.Plant
                                 .Where(row => row.Company == Session.CompanyID)
                                 .ToList())
    {
        // Do Work
    }
}

Many customers don’t notice is because it all depends if you are selecting all columns, versus explicit columns, not just in your existing BPM but in the BPMs that come before and after.

A simple OnBinNumChanged BPM may begin the tracking if you don’t query JobHead there correctly, and it will flow to Update eventually.

28

You and I both. I need to make a local Entity Framework 6 and EF Core example with AdventureWorks to tinker around the differences.

29

SO my :safe_harbor: Safe harbor :safe_harbor: THEORY on what is going on with YOUR BPM.. (according to more searching with ChatGPT).

  1. you read the JobHead… but in this instance you only get one value… this is probably safe and doesnt need the AsNoTracking() and would be redundant
  2. BUT the other queries on JobAsmbl and JobOper are probably causing the issue… (Note that ```ShipDtl`` also reads in the full dataset, and could be read only as well, but i doubt it is utilized in the labor transaction.. That said, it wouldn’t hurt to put the AsNoTracking() there as well)
    Why?
    Well, you read those into memory, but then you call the labor service. It will ALSO be reading those into memory and then writing them. Remember that when you write to those two tables, it also writes to the jobhead as well. (using a job service)…

MY THOUGHT and THEORY is that if you put the AsNoTracking onto these two tables, since you are not modifying them here, it should resolve.

30

for full transparency… here is the full ChatGPT analysis when i posted the BPM Code provided above… note that it calls out several additional errors… not sure if they are totally true, but I will pass them on in case it helps.

Assessment

In the code shown, adding AsNoTracking() specifically to this JobHead query should be functionally redundant:

var jobHead = Db.JobHead
    .AsNoTracking()
    .Where(jh =>
        jh.Company == Session.CompanyID &&
        jh.JobNum == shipDetail.JobNum)
    .Select(jh => new { jh.ProdQty })
    .FirstOrDefault();

The result contains only ProdQty. No JobHead entity is materialized, so there is normally nothing for EF to track. Microsoft explicitly documents that projections containing scalar values but no entity instances are not tracked.

Therefore, if that change truly eliminated the error, one of these is likely true:

  1. The actual failing code originally returned the full JobHead entity, not just new { jh.ProdQty }.
  2. Other code changed simultaneously.
  3. The error is intermittent and the apparent fix was coincidental.
  4. Epicor’s EF implementation or query interceptor behaves differently in that particular release.
  5. The tracking conflict is elsewhere, and the query change altered execution timing or context state enough to hide it.

I would not treat AsNoTracking() on that scalar projection as the root-cause fix.

The more likely tracking problem

These three queries do materialize complete entities into the BPM’s shared Db context:

select sd
select ja
select jo

Specifically:

var shipDetails = Db.ShipDtl
    .Where(...)
    .ToList();

var jobAssemblies = Db.JobAsmbl
    .Where(...)
    .ToList();

var jobOperations = Db.JobOper
    .Where(...)
    .ToList();

Those ShipDtl, JobAsmbl, and JobOper rows are tracking queries by default.

Then the code creates LaborSvc using the same context:

ServiceRenderer.GetService<Erp.Contracts.LaborSvcContract>(Db)

The Labor BO can internally load, attach, update, or validate some of those same job entities. That creates the real collision risk:

  • BPM code has already materialized and tracked a JobOper.
  • LaborSvc.DefaultOprSeq() internally loads or attaches that same JobOper.
  • EF encounters two instances or incompatible entity states for the same key.
  • The exception surfaces inside DefaultOprSeq() or Update(), even though the earlier query created the context contamination.

The exact error text matters. Typical manifestations include:

  • “An entity object cannot be referenced by multiple instances of IEntityChangeTracker.”
  • “Attaching an entity of type X failed because another entity of the same type already has the same primary key value.”
  • “The relationship could not be changed because one or more foreign-key properties are non-nullable.”
  • Optimistic concurrency or unexpected state-manager errors.

Recommended correction

All database reads in this BPM appear read-only. Do not materialize tracked Epicor entities merely to obtain a few values.

Better query shape

var shipDetails = Db.ShipDtl
    .Where(sd =>
        sd.Company == Session.CompanyID &&
        sd.PackNum == shipHead.PackNum)
    .Select(sd => new
    {
        sd.JobNum,
        sd.OurJobShipQty
    })
    .ToList();

foreach (var shipDetail in shipDetails)
{
    var prodQty = Db.JobHead
        .Where(jh =>
            jh.Company == Session.CompanyID &&
            jh.JobNum == shipDetail.JobNum)
        .Select(jh => (decimal?)jh.ProdQty)
        .FirstOrDefault() ?? decimal.Zero;

    var assemblySeqs = Db.JobAsmbl
        .Where(ja =>
            ja.Company == Session.CompanyID &&
            ja.JobNum == shipDetail.JobNum)
        .Select(ja => ja.AssemblySeq)
        .ToList();

    foreach (var assemblySeq in assemblySeqs)
    {
        var operations = Db.JobOper
            .Where(jo =>
                jo.Company == Session.CompanyID &&
                jo.JobNum == shipDetail.JobNum &&
                jo.AssemblySeq == assemblySeq &&
                jo.LaborEntryMethod != "B" &&
                !jo.OpComplete)
            .Select(jo => new
            {
                jo.OprSeq,
                jo.RunQty
            })
            .ToList();

        // LaborSvc processing...
    }
}

This eliminates entity tracking structurally. It is better than scattering AsNoTracking() because no entity instances are returned at all.

Alternatively, apply AsNoTracking() to the queries that actually return entities:

Db.ShipDtl.AsNoTracking()
Db.JobAsmbl.AsNoTracking()
Db.JobOper.AsNoTracking()

But scalar projection is the cleaner design.

Other problems in the code

1. The service is not disposed

Epicor documentation says service instances retrieved through ServiceRenderer should be disposed. Use:

using (var laborSvc =
    Ice.Assemblies.ServiceRenderer
        .GetService<Erp.Contracts.LaborSvcContract>(Db))
{
    // processing
}

Epicor explicitly recommends disposing BO service instances retrieved through ServiceRenderer.

2. Db.SaveChanges() appears unnecessary

The code does not directly modify tracked Db entities. All writes go through laborSvc.Update().

This final call:

Db.SaveChanges();

is therefore suspicious and probably unnecessary. Worse, because the query entities are tracked, SaveChanges() asks EF to inspect the accumulated graph after many BO calls. That increases the chance of state-manager and concurrency problems.

Unless other omitted code modifies Db entities, remove it.

3. The outer transaction is high risk

The code wraps multiple Labor BO operations, potentially many labor records, inside one explicit transaction:

using (var txScope = IceContext.CreateDefaultTransactionScope())

That means:

  • Every service call participates in one large transaction.
  • Locks are retained across all shipments, assemblies, operations, and Labor updates.
  • One failure rolls back the entire batch.
  • BO methods may have internal transaction assumptions.
  • Nested service calls share the same context and transaction state.
  • Transaction duration grows combinatorially with shipment complexity.

This is fragile architecture. The transaction should generally be scoped to one logical labor transaction, or left to the Labor BO unless atomicity across all records is genuinely required.

4. laborHedRow is captured before Update

This pattern deserves scrutiny:

var laborHedRow = laborDataSet.LaborHed
    .FirstOrDefault(lh => lh.RowMod == "A");

laborSvc.Update(ref laborDataSet);

After Update, the BO may merge, replace, or alter rows in the dataset. Holding a reference obtained before the update is not always safe. Reacquire it afterward:

laborSvc.Update(ref laborDataSet);

var laborHedRow = laborDataSet.LaborHed.FirstOrDefault();

5. The code creates one LaborHed per operation

A new LaborHed is created inside the operation loop:

laborSvc.GetNewLaborHed1(...)

That means one header for every job operation. This may be intentional, but it is usually an unnecessarily expensive and fragmented transaction model. A more natural structure may be:

  • One LaborHed per employee/date/session.
  • Multiple LaborDtl rows under that header.
  • One final validation and update sequence.

Whether that is valid depends on the intended labor accounting semantics.

Bottom line

The claim that AsNoTracking() fixed the shown JobHead query is technically weak because that query already projects only a scalar property.

The strongest diagnosis is:

The BPM loads full ShipDtl, JobAsmbl, and JobOper entities into the same Db context subsequently used by LaborSvc. The Labor BO then interacts with overlapping job entities, producing an EF tracking or state-manager conflict.

I would make these changes first:

  1. Project all read-only queries into scalar or anonymous values.
  2. Dispose laborSvc.
  3. Remove Db.SaveChanges() unless direct Db updates exist.
  4. Reduce or eliminate the outer transaction scope.
  5. Reacquire dataset rows after BO updates.
  6. Obtain the complete EF exception and inner exception before accepting AsNoTracking() as the causal fix.

Epicor documentation

:page_facing_up: Article: Technical Information

Describes BPM technical behavior, directive types, validation, and custom-code execution context.

:page_facing_up: Article: Calling BOs From Custom Code

Documents the supported ServiceRenderer pattern and states that retrieved service instances should be disposed.

:page_facing_up: Article: Troubleshooting

Provides guidance for isolating BPM and customization-related failures.

:page_facing_up: Article: Querying the Database

Documents the Epicor Db database context and LINQ querying model.

:page_facing_up: Article: BPM Widgets Reference

Provides the BPM workflow and widget reference.

31

for full transparency… here is the full ChatGPT analysis when i posted the BPM Code provided above… note that it calls out several additional errors… not sure if they are totally true, but I will pass them on in case it helps.

Assessment

In the code shown, adding AsNoTracking() specifically to this JobHead query should be functionally redundant:

var jobHead = Db.JobHead
    .AsNoTracking()
    .Where(jh =>
        jh.Company == Session.CompanyID &&
        jh.JobNum == shipDetail.JobNum)
    .Select(jh => new { jh.ProdQty })
    .FirstOrDefault();

The result contains only ProdQty. No JobHead entity is materialized, so there is normally nothing for EF to track. Microsoft explicitly documents that projections containing scalar values but no entity instances are not tracked.

Therefore, if that change truly eliminated the error, one of these is likely true:

  1. The actual failing code originally returned the full JobHead entity, not just new { jh.ProdQty }.
  2. Other code changed simultaneously.
  3. The error is intermittent and the apparent fix was coincidental.
  4. Epicor’s EF implementation or query interceptor behaves differently in that particular release.
  5. The tracking conflict is elsewhere, and the query change altered execution timing or context state enough to hide it.

I would not treat AsNoTracking() on that scalar projection as the root-cause fix.

The more likely tracking problem

These three queries do materialize complete entities into the BPM’s shared Db context:

select sd
select ja
select jo

Specifically:

var shipDetails = Db.ShipDtl
    .Where(...)
    .ToList();

var jobAssemblies = Db.JobAsmbl
    .Where(...)
    .ToList();

var jobOperations = Db.JobOper
    .Where(...)
    .ToList();

Those ShipDtl, JobAsmbl, and JobOper rows are tracking queries by default.

Then the code creates LaborSvc using the same context:

ServiceRenderer.GetService<Erp.Contracts.LaborSvcContract>(Db)

The Labor BO can internally load, attach, update, or validate some of those same job entities. That creates the real collision risk:

  • BPM code has already materialized and tracked a JobOper.
  • LaborSvc.DefaultOprSeq() internally loads or attaches that same JobOper.
  • EF encounters two instances or incompatible entity states for the same key.
  • The exception surfaces inside DefaultOprSeq() or Update(), even though the earlier query created the context contamination.

The exact error text matters. Typical manifestations include:

  • “An entity object cannot be referenced by multiple instances of IEntityChangeTracker.”
  • “Attaching an entity of type X failed because another entity of the same type already has the same primary key value.”
  • “The relationship could not be changed because one or more foreign-key properties are non-nullable.”
  • Optimistic concurrency or unexpected state-manager errors.

Recommended correction

All database reads in this BPM appear read-only. Do not materialize tracked Epicor entities merely to obtain a few values.

Better query shape

var shipDetails = Db.ShipDtl
    .Where(sd =>
        sd.Company == Session.CompanyID &&
        sd.PackNum == shipHead.PackNum)
    .Select(sd => new
    {
        sd.JobNum,
        sd.OurJobShipQty
    })
    .ToList();

foreach (var shipDetail in shipDetails)
{
    var prodQty = Db.JobHead
        .Where(jh =>
            jh.Company == Session.CompanyID &&
            jh.JobNum == shipDetail.JobNum)
        .Select(jh => (decimal?)jh.ProdQty)
        .FirstOrDefault() ?? decimal.Zero;

    var assemblySeqs = Db.JobAsmbl
        .Where(ja =>
            ja.Company == Session.CompanyID &&
            ja.JobNum == shipDetail.JobNum)
        .Select(ja => ja.AssemblySeq)
        .ToList();

    foreach (var assemblySeq in assemblySeqs)
    {
        var operations = Db.JobOper
            .Where(jo =>
                jo.Company == Session.CompanyID &&
                jo.JobNum == shipDetail.JobNum &&
                jo.AssemblySeq == assemblySeq &&
                jo.LaborEntryMethod != "B" &&
                !jo.OpComplete)
            .Select(jo => new
            {
                jo.OprSeq,
                jo.RunQty
            })
            .ToList();

        // LaborSvc processing...
    }
}

This eliminates entity tracking structurally. It is better than scattering AsNoTracking() because no entity instances are returned at all.

Alternatively, apply AsNoTracking() to the queries that actually return entities:

Db.ShipDtl.AsNoTracking()
Db.JobAsmbl.AsNoTracking()
Db.JobOper.AsNoTracking()

But scalar projection is the cleaner design.

Other problems in the code

1. The service is not disposed

Epicor documentation says service instances retrieved through ServiceRenderer should be disposed. Use:

using (var laborSvc =
    Ice.Assemblies.ServiceRenderer
        .GetService<Erp.Contracts.LaborSvcContract>(Db))
{
    // processing
}

Epicor explicitly recommends disposing BO service instances retrieved through ServiceRenderer.

2. Db.SaveChanges() appears unnecessary

The code does not directly modify tracked Db entities. All writes go through laborSvc.Update().

This final call:

Db.SaveChanges();

is therefore suspicious and probably unnecessary. Worse, because the query entities are tracked, SaveChanges() asks EF to inspect the accumulated graph after many BO calls. That increases the chance of state-manager and concurrency problems.

Unless other omitted code modifies Db entities, remove it.

3. The outer transaction is high risk

The code wraps multiple Labor BO operations, potentially many labor records, inside one explicit transaction:

using (var txScope = IceContext.CreateDefaultTransactionScope())

That means:

  • Every service call participates in one large transaction.
  • Locks are retained across all shipments, assemblies, operations, and Labor updates.
  • One failure rolls back the entire batch.
  • BO methods may have internal transaction assumptions.
  • Nested service calls share the same context and transaction state.
  • Transaction duration grows combinatorially with shipment complexity.

This is fragile architecture. The transaction should generally be scoped to one logical labor transaction, or left to the Labor BO unless atomicity across all records is genuinely required.

4. laborHedRow is captured before Update

This pattern deserves scrutiny:



var laborHedRow = laborDataSet.LaborHed
    .FirstOrDefault(lh => lh.RowMod == "A");

laborSvc.Update(ref laborDataSet);

After Update, the BO may merge, replace, or alter rows in the dataset. Holding a reference obtained before the update is not always safe. Reacquire it afterward:



laborSvc.Update(ref laborDataSet);

var laborHedRow = laborDataSet.LaborHed.FirstOrDefault();

5. The code creates one LaborHed per operation

A new LaborHed is created inside the operation loop:



laborSvc.GetNewLaborHed1(...)

That means one header for every job operation. This may be intentional, but it is usually an unnecessarily expensive and fragmented transaction model. A more natural structure may be:

  • One LaborHed per employee/date/session.
  • Multiple LaborDtl rows under that header.
  • One final validation and update sequence.

Whether that is valid depends on the intended labor accounting semantics.

Bottom line

The claim that AsNoTracking() fixed the shown JobHead query is technically weak because that query already projects only a scalar property.

The strongest diagnosis is:

The BPM loads full ShipDtl, JobAsmbl, and JobOper entities into the same Db context subsequently used by LaborSvc. The Labor BO then interacts with overlapping job entities, producing an EF tracking or state-manager conflict.

I would make these changes first:

  1. Project all read-only queries into scalar or anonymous values.
  2. Dispose laborSvc.
  3. Remove Db.SaveChanges() unless direct Db updates exist.
  4. Reduce or eliminate the outer transaction scope.
  5. Reacquire dataset rows after BO updates.
  6. Obtain the complete EF exception and inner exception before accepting AsNoTracking() as the causal fix.

Epicor documentation

:page_facing_up: Article: Technical Information

Describes BPM technical behavior, directive types, validation, and custom-code execution context.

:page_facing_up: Article: Calling BOs From Custom Code

Documents the supported ServiceRenderer pattern and states that retrieved service instances should be disposed.

:page_facing_up: Article: Troubleshooting

Provides guidance for isolating BPM and customization-related failures.

:page_facing_up: Article: Querying the Database

Documents the Epicor Db database context and LINQ querying model.

:page_facing_up: Article: BPM Widgets Reference

Provides the BPM workflow and widget reference.

32

One other note.. I think that ChatGPT picked up on one additional topic that I have been promoting when writing BPMs…
Sometimes we (programmers) are lazy. we ask the BPM to “read the record”… and like any good soldier, it follows orders, reading hundreds of columns, returning them to your program in memory.
BUT, in reality, we only needed one (or a small number) of fields.
It would be much better to only return the values needed instead of all of them.
Example, in the case of the code above, you read the entire Ship Detail record, when you only “Needed” the Job number and the quantity shipped. This is what you DID do correctly in reading the JobHead table… but in the other three tables, that was not used.

33

UPDATE: Looks like we found a work around to the BPM error that’s happening internally with Epicor running this process. We deteched the records on Update. I’ll let @ryanw describe in more detail.

We also wrote a new Kinetic Diff tool (like @josecgomez @hkeric.wci console one) that runs 100% HTML that allows us to walk the entire ETAP log and selectively view the payload and response across the conversation so as to try to see if there was some new fields we were missing. But in the end, the old BPM code (post MARS fixes) still worked – just now with the new detach workaround.

Now, we’re off to write a one-off app to fix over 4,000 jobs that have accumulated over the last 3 weeks. :face_with_symbols_on_mouth:

Anyway, thanks to all that participated here! :right_facing_fist::left_facing_fist:

34

Would you please reference what you mean by that in the post above or post the code?

If that’s too much to ask no worries.

35

@timshuwy If you are querying using the widget (like for conditions or Fill Table by query) , does the underlying problem with change tracking still exist? Or do the widgets always return entities with no change tracking?

36

That’s a lot of people’s concerns, whether widgets are now broken and also BASE code that Epicor themselves wrote before understanding all the nuances/differences of EFCore.

37

You are correct, widgets should NOT be broken and should be the safest pattern for BPMs, so these are a high priority. C# widgets are more prone to breaking since we have no control over what you enter there.

38

Change tracking has nothing to do with how you write BPMs. Widgets or C# widget is just “inserted code” into the process and should not change how change logs work.

39

You and he are talking past each other.
He is talking about EF Core entity tracking.

40

Ahh.. whoops.. i got it…
Well, that said, just because you use widgets doesnt mean that you are exempt from writing good logical widget code. I have seen some bad practice with widgets as well, and if you do the right (wrong) thing with a widget, i am sure you can get some EFCore thing to trigger. (like, i have seen people create loops with widgets (wow).