Function error logged

[Aaron_Moreng]

We just went live with a rather complex integration between our online ordering portal and Epicor ERP.
This integration makes heavy use of Epicor functions.

All is working well, ordering are coming through to hit the function and create orders (well, in this case it was creating a quote, but alas) until I saw one order on our portal that was not showing as being linked to Epicor.

I correlated the event logs in the application server with the same time the order was submitted to us and I saw to interesting logs.
The first log was generated from a parsing error on Newtonsoft, which makes sense because I consume some serialized JSON in this function.

First error:

Newtonsoft.Json.JsonReaderException: After parsing a value an unexpected character was encountered: {. Path '[0]', line 5, position 0.
   at Newtonsoft.Json.JsonTextReader.ParsePostValue(Boolean ignoreComments) in /_/Src/Newtonsoft.Json/JsonTextReader.cs:line 1480
   at Newtonsoft.Json.JsonTextReader.Read() in /_/Src/Newtonsoft.Json/JsonTextReader.cs:line 426
   at Newtonsoft.Json.JsonWriter.WriteToken(JsonReader reader, Boolean writeChildren, Boolean writeDateConstructorAsDate, Boolean writeComments) in /_/Src/Newtonsoft.Json/JsonWriter.cs:line 663
   at Newtonsoft.Json.JsonWriter.WriteToken(JsonReader reader, Boolean writeChildren) in /_/Src/Newtonsoft.Json/JsonWriter.cs:line 514
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateJToken(JsonReader reader, JsonContract contract) in /_/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.cs:line 232
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id) in /_/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.cs:line 918
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) in /_/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.cs:line 196
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) in /_/Src/Newtonsoft.Json/JsonSerializer.cs:line 907
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) in /_/Src/Newtonsoft.Json/JsonConvert.cs:line 828
   at EFx.EpicorBridge.Implementation.Order_CreateQuoteImpl.A005_CustomCodeAction()
   at EFx.EpicorBridge.Implementation.Order_CreateQuoteImpl.RunStep(Int32 workflowStep)
   at Epicor.Functions.FunctionBase`3.Run() in C:\_Releases\ICE\RL10.2.600.0FW\Source\Server\Internal\Lib\Epicor.Functions.Core\FunctionBase.cs:line 90
   at Epicor.Functions.FunctionBase`3.Run(TInput input) in C:\_Releases\ICE\RL10.2.600.0FW\Source\Server\Internal\Lib\Epicor.Functions.Core\FunctionBase.cs:line 75
   at Epicor.Functions.FunctionRestAdapter`2.Run(IFunctionRestHost host, JObject input) in C:\_Releases\ICE\RL10.2.600.0FW\Source\Server\Internal\Lib\Epicor.Functions.Core\FunctionRestAdapter.cs:line 46
   at Epicor.RESTApi.Functions.Controllers.EpicorFunctionController.Post(Boolean production, String company, String library, String function, JObject data) in C:\_Releases\ICE\UD10.2.600.10FW\Source\Server\Internal\RESTApi.Plugins\Epicor.RESTApi.EFxPlugin\Controllers\EpicorFunctionController.cs:line 69
   at lambda_method(Closure , Object , Object[] )
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass6_2.<GetExecutor>b__2(Object instance, Object[] methodParameters)
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.HttpServer.<SendAsync>d__24.MoveNext()

The second error was even less descriptive and simple was an “Ice.Common.BusinessObjectException: Error int he application”. Surely related, not sure how.

Ice.Common.BusinessObjectException: Error in the application.
   at Epicor.Functions.FunctionBase`3.ProcessRecordedExceptions() in C:\_Releases\ICE\RL10.2.600.0FW\Source\Server\Internal\Lib\Epicor.Functions.Core\FunctionBase.cs:line 134
   at Epicor.Functions.FunctionBase`3.Run(TInput input) in C:\_Releases\ICE\RL10.2.600.0FW\Source\Server\Internal\Lib\Epicor.Functions.Core\FunctionBase.cs:line 77
   at Epicor.Functions.FunctionRestAdapter`2.Run(IFunctionRestHost host, JObject input) in C:\_Releases\ICE\RL10.2.600.0FW\Source\Server\Internal\Lib\Epicor.Functions.Core\FunctionRestAdapter.cs:line 46
   at Epicor.RESTApi.Functions.Controllers.EpicorFunctionController.Post(Boolean production, String company, String library, String function, JObject data) in C:\_Releases\ICE\UD10.2.600.10FW\Source\Server\Internal\RESTApi.Plugins\Epicor.RESTApi.EFxPlugin\Controllers\EpicorFunctionController.cs:line 69
   at lambda_method(Closure , Object , Object[] )
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass6_2.<GetExecutor>b__2(Object instance, Object[] methodParameters)
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.HttpServer.<SendAsync>d__24.MoveNext()

The function itself isn’t fancy, but it appears the first error choked on the custom code block as shown:

The code is as follows:

//iterate over incoming params to add part lines

dynamic prodList = Newtonsoft.Json.JsonConvert.DeserializeObject(productList);

//msg = prodList.ToString();
int rowCnt = 0;
foreach(var product in prodList)
{
        
        //Get New Quote Dtl
        this.CallService<Erp.Contracts.QuoteSvcContract>(qs =>{
        qs.GetNewQuoteDtl(ref tsQuote, quoteNum); 
        //ChangePartNumMaster
        
        string partNum = product.PartNum.ToString();
        bool lIsPhantom = false;
        bool lIsSalesKit = false;
        string uomCode = "";
        string rowType = "";
        Guid SysRowID = Guid.Empty;
        bool salesKitView = false;
        bool removeKitComponents = false;
        bool suppressUserPrompts = false;
        bool runChkPrePartInfo = true;
        string vMessage = "";
        string vPMessage = "";
        string vBMessage = "";
        bool vSubAvail = false;
        string vMsgType = "";
        bool getPartXRefInfo = true;
        bool checkChangeKitParent = true;
        string cDeleteComponentsMessage = "";
        bool multipleMatch = false;
        bool promptToExplodeBOM = false;
        string explodeBOMerrMessage = "";
        
        tsQuote.QuoteDtl[rowCnt].PartNum = product.PartNum.ToString();
        tsQuote.QuoteDtl[rowCnt].RowMod = "A";
        qs.ChangePartNumMaster(ref partNum, ref lIsPhantom, ref lIsSalesKit, ref uomCode, rowType, SysRowID, salesKitView, removeKitComponents, suppressUserPrompts,runChkPrePartInfo, out vMessage,out vPMessage, out vBMessage, out vSubAvail, out vMsgType, getPartXRefInfo, checkChangeKitParent,out cDeleteComponentsMessage, out multipleMatch, out promptToExplodeBOM, out explodeBOMerrMessage, ref tsQuote);
        
        //ChangePartNum
        bool lSubstitutePartsExist = false;
        tsQuote.QuoteDtl[rowCnt].RowMod = "A";
        qs.ChangePartNum(ref tsQuote, lSubstitutePartsExist, uomCode);
        
        //Update
        tsQuote.QuoteDtl[rowCnt].RowMod = "A";
        qs.Update(ref tsQuote);
        rowCnt++;
        }
      );
      
     
      //msg += product.PartNum.ToString()+ ": ";
}

This simple deserializes a string which looks something like this depending on the order submitted:

  "productList": "[\r\n{\"PartNum\": \"32247001\"},\r\n{\"PartNum\":\"45647010\"}\r\n]",

and breaks it apart into a list and iterates over it creating Quote lines.

I have not encountered this error before in any testing nor have I seen it in any other cases (so far).

The first thing I will check tomorrow is to see if we can reproduce the error. The second will be to see if I can replicate what data was being passed to me and if that produces the error.
Is there anything else that is obvious to others that might have caused such a failure?

The other interesting thing is that even if it did choke on the serialization, it did not create the quote head, which comes first prior to any custom code block. It’s like the whole thing blew up…

Appreciate the second look, I’ve been up too long as it is

[Mark_Wonsil]

@Aaron_Moreng

dynamic prodList = Newtonsoft.Json.JsonConvert.DeserializeObject(productList);

Maybe wrap this line up in a try-catch block and log it when there is an error.

Somebody on this list once suggested to make the import dead simple: accept the input string and write it to a UD table. The table has a status field that is set to something like “imported”. Next, process the record (either in the same function or get fancy and treat it like a queue and have another async task process it separately). After a successful import, update the status field to “processed”. They suggested that now you have some observability where you can easily find records that did not import, orders imported per day, etc. Heck, you could really get fancy and check with your web portal and verify that the two are identical and change the status to “verified”.

[Olga (Olga Klimova)]

@Aaron_Moreng

Ice.Common.BusinessObjectException: Error in the application.

I think there should be another error in the server log, corresponding to this Ice.Common.BusinessObjectException: Error in the application.

[utaylor (Utah Taylor)]

Mark! I was just about to suggest this!

@Aaron_Moreng , there’s this server application design which I am not sure applies here, but I suppose it could…

It’s called a microservice architecture and the idea is that each service is self contained and should implement a single business capability. You break one solution (i.e. your function here) into several, smaller independent pieces.

When I look at the function and I pair it with the knowledge of how entering a quote goes…

You first have to create a header. Then you have to add some line(s).

You may wish to split this function into two or more functions:

1. create the header
2. add a line to that header. (repeat this step over and over).
3. Add/update a quote contact
4. Add/update a sales rep

In this way you can call each of the functions separately in one code block, but put each one in a try catch loop as Mark suggested so that the program will not move onto the next function call without the previous one completing.

I am not a programmer so maybe this is terrible advice.

Microservice Architecture is usually a part of application design, not necessarily programming. Object oriented programming and microservice architecture for designing a server application are similar in that they use independent objects to perform a unified task- microservices are independent like objects/classes are in a C# program, but when used together they perform one unified task. If something is wrong with one microservice you know where your problem is… much like if there is an error in a c# class or object or method call, you know where to start… Maybe I am reaching here, but hey!

Microservices architecture | Microsoft Docs

[utaylor (Utah Taylor)]

for the record I like Mark’s idea a lot- no need to redesign anything about the function except adding a try catch.

Function error logged

Maybe wrap this line up in a try-catch block and log it when there is an error. Somebody on this list once suggested to make the import dead simple: accept the input string and write it to a UD table. The table has a status field that is set to something like “imported”. Next, process the record (either in the same function or get fancy and treat it like a queue and have another async task process it separately). After a successful import, update the status field to “processed”. They suggested…

[Aaron_Moreng]

Hey,
I chose a more RPC type architecture for this particular integration. The function exposes a request signature to consume all the necessary information needed and then processes it server side via the function.
In this case, I think the issue is probably on the web app site and going from JavaScript to PHP to .NET can be a little tricky. Browser code collects the order info, sends it to backend process which then calls my API which then calls Epicor and all returns a quote number/sales order number back through.
Like I said the integration itself and everything is working perfectly, it’s this one weird one-off I am concerned about.
I thought about it and I think it was likely a bad call from the web app to my API which caused it. The following error is typical for bad parsing:

@Aaron_Moreng

Newtonsoft.Json.JsonReaderException: After parsing a value an unexpected character was encountered: {. Path '[0]', line 5, position 0.

I believe it was not my “create lines” code parsing it but rather a translation from the actual function signature that caused it. That’s my theory at least.

[utaylor (Utah Taylor)]

Nice, this app sounds awesome.

You going to be at insights? Would love to see it.

[Aaron_Moreng]

Haha, that’s exactly the design I implemented for Epicor to web app integration. Event listeners are collecting specific events across the system and writing them to a UD table, then invoking a function (passing in the UD table sysrowID of the event).

The function consumes the sysrowID to look up the event and deserializes the contents

and calls the web app web service as designed. Web app does its thing to update the record on the portal and responds back. Epicor listens for response and records it in table.
This actually saved my butt yesterday because we ran into a snag with migrating legacy data over to the new portal. Meanwhile, all of yesterday we ran business in Epicor, so we had potential for de-syncing events on legacy orders.

Today, got the legacy data imported and I built a user process scheduler BPM to re-play yesterdays events against the function to call the web service. Worked like a charm and everything is now in sync.
+1 for that architecture vs. real time HTTP calls for syncing two systems.

[Aaron_Moreng]

Bro, I don’t even think we will be on Epicor in the near future Company is likely changing ERP systems this year.

[Mark_Wonsil]

@Aaron_Moreng

Company is likely changing ERP systems this year.

That’s too bad. However, since you have this nice API layer in there, all you’ll have to do is write the link to the new ERP system and you’re back in business. Great architecture work Aaron!

[utaylor (Utah Taylor)]

WOWWWWWW This is incredible.

[utaylor (Utah Taylor)]

Insane architecture work.

[Aaron_Moreng]

Thanks Mark. I probably can’t help myself but to write up a eulogy on my experience and growth both personally and professionally with Epicor and the wonderful people on this forum but I’ll try to save that for another time
I recall a quote you said at some point years ago to the effect of “you haven’t really grown in IT until the system you love working with is taking out from underneath you”.

[utaylor (Utah Taylor)]

Brings a tear to my eye.

[Aaron_Moreng]

wrapped it up in a try/catch and dumped the incoming variable value to a file. I was able to determine the serialized data I was being sent was being sent incorrectly (missing a comma between json records) and it blew up. Nice suggestion!
I am convinced observability is perhaps the most important part of any integration. Got to be able to pinpoint where it’s going wrong!
Thank you sir