Function libraries sporadically return as disabled since Linux Migration

Kinetic ERP
, ,
1

Since the 2026.100.7 upgrade, some (all?) of our function libraries are intermittently returning as Disabled, whether called publicly via HTTP or internally from other libraries or directives. When the issue occurs publicly, I receive only a 404. However, if I try the request three or four more times, it eventually succeeds and the function is executed as normal.

For our most critical application, I implemented a stop gap that simply retries if a 404 is received. This stopgap doesn’t however solve all of the inter-function library calls, or calls from directives that are failing.

Here’s a stack trace from an inter-library call that’s failing. The library “CustomOrderSet” is not disabled.

Ice.Common.EpicorServerException: BPM runtime caught an unexpected exception of 'InvalidOperationException' type.
See more info in the Inner Exception section of Exception Details.
---> System.InvalidOperationException: The library 'CustomOrderSet' is disabled.
at Epicor.Functions.Runtime.EfxRuntimeExtensions.GetFunctionInfo(IFunctionsCatalog functionsCatalog, FunctionRefId refId, String forCompany) in C:\_releases\ICE\ICE5.2.100.0\Source\Server\Customization\Epicor.Functions.Runtime\EfxRuntimeExtensions.cs:line 27
at Epicor.Customization.Bpm.CustomizationBase2`2.InvokeFunction(String libraryId, String functionId, ITuple input) in C:\_releases\ICE\ICE5.2.100.7\Source\Server\Customization\Epicor.Customization.Bpm\CustomizationBase2.Plugins.cs:line 61
at Epicor.Customization.Bpm.BO.MasterUpdatePostProcessingDirective_Custom_CheckManufacturedInventory_757C06110ED345FD82C0316AC4CA1BC2.A002_InvokeEpicorFunctionAction2()
at Epicor.Customization.Bpm.BO.MasterUpdatePostProcessingDirective_Custom_CheckManufacturedInventory_757C06110ED345FD82C0316AC4CA1BC2.ExecuteCore(Int32 step)
at Epicor.Customization.Bpm.DirectiveBase`2.Execute() in C:\_releases\ICE\ICE5.2.100.7\Source\Server\Customization\Epicor.Customization.Bpm\DirectiveBase.Generic.cs:line 347
at Epicor.Customization.Bpm.DirectiveBase`2.Execute(TParam parameters) in C:\_releases\ICE\ICE5.2.100.7\Source\Server\Customization\Epicor.Customization.Bpm\DirectiveBase.Generic.cs:line 197
--- End of inner exception stack trace ---
at Epicor.Customization.Bpm.DirectiveBase`2.Execute(TParam parameters) in C:\_releases\ICE\ICE5.2.100.7\Source\Server\Customization\Epicor.Customization.Bpm\DirectiveBase.Generic.cs:line 213
at Epicor.Customization.Bpm.MethodCustomizationBase2`2.<>c__DisplayClass12_0.<RunDirectives>b__5(MethodDirectiveBase`2 dir) in C:\_releases\ICE\ICE5.2.100.7\Source\Server\Customization\Epicor.Customization.Bpm\MethodCustomizationBase2.cs:line 198
at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Func`2 predicate, Boolean& found)
at Epicor.Customization.Bpm.MethodCustomizationBase2`2.RunDirectives(TParam parameters)
at Epicor.Customization.Bpm.CustomizationBase2`2.Execute(TParam parameters) in C:\_releases\ICE\ICE5.2.100.7\Source\Server\Customization\Epicor.Customization.Bpm\CustomizationBase2.cs:line 99
at Epicor.Customization.Bpm.BO.SalesOrderSvcCustomization.MasterUpdate(Boolean lCheckForOrderChangedMsg, Boolean lcheckForResponse, String cTableName, Int32 iCustNum, Int32 iOrderNum, Boolean lweLicensed, Boolean& lContinue, String& cResponseMsg, String& cCreditShipAction, String& cDisplayMsg, String& cCompliantMsg, String& cResponseMsgOrdRel, String& cAgingMessage, String& cShipByDateMessage, String& cNeedByDateMessage, String& cMsgTypeSaleable, SalesOrderTableset& ds)
at Erp.Services.BO.SalesOrderSvcFacade.MasterUpdate(Boolean lCheckForOrderChangedMsg, Boolean lcheckForResponse, String cTableName, Int32 iCustNum, Int32 iOrderNum, Boolean lweLicensed, Boolean& lContinue, String& cResponseMsg, String& cCreditShipAction, String& cDisplayMsg, String& cCompliantMsg, String& cResponseMsgOrdRel, String& cAgingMessage, String& cShipByDateMessage, String& cNeedByDateMessage, String& cMsgTypeSaleable, SalesOrderTableset& ds) in C:\_releases\ERP\ERP12.2.100.7\Source\Server\Services\BO\SalesOrder\SalesOrderSvcFacade.cs:line 7190
at InvokeStub_SalesOrderSvcFacade.MasterUpdate(Object, Object, IntPtr*)
at System.Reflection.MethodBaseInvoker.InvokeWithManyArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
--- End of stack trace from previous location ---
at Epicor.RESTApi.RestApi.OperationInvokeHelper.CallInvoker(Type svcType, MethodInfo mi, Object[] inputs, Boolean isRoot) in C:\_releases\ICE\ICE5.2.100.0\Source\Server\Framework\Epicor.RESTApi.Common\RestApi\OperationInvokeHelper.cs:line 296
at Epicor.RESTApi.DataSources.CustomMethodCaller.CallMethodWithDynamicData(String serviceId, String methodName, IInputParamResolver inputResolver, Action`1 headersPublisher, Action`1 verifyMethod) in C:\_releases\ICE\ICE5.2.100.0\Source\Server\Framework\Epicor.RESTApi.Common\DataSources\Main\CustomMethodCaller.cs:line 71
at lambda_method279619(Closure, Object, Object[])
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at CoreWCF.Channels.ServiceModelHttpMiddleware.InvokeAsync(HttpContext context)
at CoreWCF.Channels.MetadataMiddleware.InvokeAsync(HttpContext context)
at Ice.Hosting.AspNetCore.Middleware.DynamicAssemblyPartMiddleware.Invoke(HttpContext context, CurrentCallInformationService currentCallInformation, ControllerLoader controllerLoader) in C:\_releases\ICE\ICE5.2.100.0\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\Middleware\DynamicAssemblyPartMiddleware.cs:line 32
at Ice.Hosting.AspNetCore.Middleware.IPEnforcerMiddleware.Invoke(HttpContext httpContext) in C:\_releases\ICE\ICE5.2.100.0\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\Middleware\IPEnforcerMiddleware.cs:line 169
at Ice.Hosting.AspNetCore.ETags.ETagMiddleware.Invoke(HttpContext httpContext) in C:\_releases\ICE\ICE5.2.100.0\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\ETags\ETagMiddleware.cs:line 89
at Ice.Hosting.AspNetCore.ETags.ETagMiddleware.Invoke(HttpContext httpContext) in C:\_releases\ICE\ICE5.2.100.0\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\ETags\ETagMiddleware.cs:line 107
at Ice.Hosting.AspNetCore.Middleware.DecompressRequestMiddleware.Invoke(HttpContext context) in C:\_releases\ICE\ICE5.2.100.0\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\Middleware\DecompressRequestMiddleware.cs:line 41
at Ice.Hosting.AspNetCore.Middleware.AuthenticationMiddleware.InvokeAsync(HttpContext httpContext, CurrentCallInformationService callInformation) in C:\_releases\ICE\ICE5.2.100.0\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\Middleware\AuthenticationMiddleware.cs:line 76
at Ice.Hosting.AspNetCore.Middleware.CallHeaderMiddleware.InvokeAsync(HttpContext httpContext) in C:\_releases\ICE\ICE5.2.100.0\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\Middleware\CallHeaderMiddleware.cs:line 52
at Ice.Hosting.AspNetCore.Middleware.OperationDisposerMiddleware.InvokeAsync(HttpContext httpContext) in C:\_releases\ICE\ICE5.2.100.0\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\Middleware\OperationDisposerMiddleware.cs:line 22

Anyone experiencing something similar?

2

First step is to reboot your app server.

3

Is that the Restart Site button in the Cloud Management Portal? I haven’t used that yet and it sounds scary during production hours.

4

Think so, don’t be scared.

If you want your stuff to work, this is your first step.

5

Yes, seen exactly this too. Also hoping a restart fixes it.

6

This actually isnt a 2026.100.7 thing I believe. Its a linux server thing. We got on the flex, so we are still on 2025 :folded_hands: and seeing the same issue.

Restarting the app server is also pretty quick. Did that last night and it was only a couple minutes.

One heck of a decision to do TWO major upgrades on the same weekend.

7

Maybe, maybe not. Probably related to the stale functions issue, which has been going on well before any linux containers.

8

Maybe. Cant say ive ever ran into it before so I had assumed new.

I also had to do the restart because I had a deleted BPM that was still running. :roll_eyes:

Sure would be nice if they worked on stability rather than pushing out breaking updates…

9

I’ve never had function issues like this, this is a first, and we use them extensively.

10

I would die.

11

I’ve had other issues :slight_smile: just none that impacted us in such a weird way.

12

Continuing to have this issue in 2025.2 after restarting the site.

13

Our issues appears to have improved, but I’m continuing to monitor for awhile before I declare success.

14

Problem is continuing to happen. Demoting and publishing works for a little while. Then randomly it starts giving a 404 again. Confirm on the rest help page the library has disappeared. No errors are logged to the admin logs which means the call is not even reaching the server at all. Load balancing misdirection? I don’t know, but as usual support has been infuriatingly unhelpful, only suggesting what I told them I already tried (rebooting).

15

Oh that reminds me I had weird issues with the rest help page as well. The library would sometimes show up, then when i clicked it would disappear, then when i clicked a different library it would show back up.

16

Okay, new issue showed up, but now changes to custom code arent being recognized.

I demoted the library, made some changes, removed an email that got sent to me whenever it got called, then promoted it.

I still get emails.

Never had this happen before and Ive done quite a bit of playing around with functions. So what, now everytime I change a function I have to restart the app server?!

17

I’ve had this for a couple of customers previously. My assumption is that there are multiple application servers running (or in the case I was looking at I believed multiple task agents) because I was getting the same inconsistent behaviour - making changes to a BPM/Function were not sticking.

Let me try and find the case and see what the resolution was. OK found it, updated below:

image
image1633×377 57.1 KB

and the response from UK support was:

image
image826×231 5.35 KB

18

I know a guy who had that exact problem a year ago. It was brutal.

I’d have quit my job. I live in functions and iterate a few times a minute sometimes.

19

A load balance issue is what I was thinking too.

20

I also have the 404 issue and I think I had the no-change issue as well.

I was given PRB0288709

I don’t see this issue in pilot, which is also linux and latest version of Epicor, so I agree is smells of load balancer or other config unique to live.