Kinetic Americas - Incident (Production, Scheduled stuff failing @ end of month on a weekend)

GabeFranco · March 1, 2026, 7:03am

From Epicor Status/Emails re: Kinetic/Americas/Production:

Feb-28, 1:40pm PST
We are currently investigating an incident that is impacting a subset of customers in your region. 

We apologize for any inconvenience this may cause and are working to resolve the issue as quickly as possible. 

We will provide updates as we have them and appreciate your patience while we work to resolve the issue.
 
Feb-28, 5:21pm PST
We apologize for any inconvenience and will keep you informed as we have more information. 

Thank you for your understanding as we work to fix the problem.
 
Feb-28, 7:05pm PST
We apologize for any inconvenience and will keep you informed as we have more information. 
Thank you for your understanding as we work to fix the problem.
 
Feb-28, 8:19pm PST
We apologize for any inconvenience and will keep you informed as we have more information. 

Thank you for your understanding as we work to fix the problem.
 
Feb-28, 10:51pm PST
[in progress]
We have identified the cause of the incident and are implementing a solution.

We are closely monitoring the situation to ensure the issue is fully resolved.

ENVIRONMENT
Production

I believe this is the second time in recent memory that there has been a failure in Production at End of Month affecting scheduled things…

I guess if I didn’t learn the first time not to trust scheduled things, I learn now.

Entry #: 1, IsError?: TRUE, MsgType: ServerError
2026-03-01 05:46:17 AM
MsgText: Program Ice.Services.Lib.RunTask when executing task 6652070 raised an unexpected exception with the following message: RunTask:
System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while executing the command definition. See the inner exception for details.
 ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'ICECommon_5_0_100.Ice.ReportStore'.
Could not use view or function 'Ice.ReportStore' because of binding errors.
   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, SqlCommand command, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean isAsync, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String method)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1.Dispatch[TTarget,TInterceptionContext,TResult](TTarget target, Func`3 operation, TInterceptionContext interceptionContext, Action`3 executing, Action`3 executed)
   at System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(DbCommand command, DbCommandInterceptionContext interceptionContext)
   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
ClientConnectionId:bd707f45-3017-451a-a838-68319f118810
Error Number:208,State:1,Class:16
   --- End of inner exception stack trace ---
   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
   at System.Data.Entity.Core.Objects.Internal.ObjectQueryExecutionPlan.Execute[TResultType](ObjectContext context, ObjectParameterCollection parameterValues)
   at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass41_0.<GetResults>b__0()
   at System.Data.Entity.Core.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)
   at System.Data.Entity.Internal.LazyEnumerator`1.MoveNext()
   at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
   at Ice.Core.SsrsReporting.ReportProcessorBase.DeployReportIfNeeded(String printProgram, String fullReportPath, SsrsConfigurationInformation configurationInfo, Func`1 reportingServiceCreator) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 428
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport_HttpClient(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 335
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 242
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReportForRouted(RenderedSsrsReport renderedReport) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 608
   at Ice.Core.SsrsReporting.ReportProcessorBase.ProcessReportPart(String reportLocation) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 140
   at Ice.Core.SsrsReporting.StandardReportProcessor.ProcessReportParts() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\StandardReportProcessor.cs:line 31
   at Ice.Core.SsrsReporting.ReportProcessorBase.ProcessReport(ReportDatabaseBuilder reportDatabaseBuilder, ReportInstanceInformation reportInstanceInformation) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 87
   at Ice.Core.RptBase.ReportSsrsDatabaseBuilder.RenderReports(ByteArrayCacheWithTemporaryBackingFile reportCache, ReportInstanceInformation reportInstanceInformation) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\ReportSsrsDatabaseBuilder.cs:line 365
   at Ice.Core.RptBase.ReportSsrsDatabaseBuilder.<>c__DisplayClass6_0.<CreateWorkflow>b__0(ReportInstanceInformation instance) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\ReportSsrsDatabaseBuilder.cs:line 182
   at Epicor.Utilities.Memoizer.<>c__DisplayClass0_0`2.<Memoize>b__0(TArgument argument) in C:\_releases\ICE\ICE5.0.100.15\Source\Shared\Framework\Epicor.ServiceModel\Utilities\Memoizer.cs:line 36
   at Ice.Core.RoutingAndBreaking.EmailReportAction.EmailReportsFromReportRequests(RetryChildNodeSettings retrySettings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RoutingAndBreaking\EmailReportAction.cs:line 325
   at Ice.Core.RoutingAndBreaking.EmailReportAction.HandleProcess(Object sender, ProcessEventArgs e) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RoutingAndBreaking\EmailReportAction.cs:line 189
   at Ice.Core.RoutingAndBreaking.BaseNode.OnProcess(ProcessEventArgs e) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RoutingAndBreaking\BaseNode.cs:line 193
   at Ice.Core.RoutingAndBreaking.BaseNode.HandleProcess(Object sender, ProcessEventArgs e) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RoutingAndBreaking\BaseNode.cs:line 139
   at Ice.Core.RoutingAndBreaking.BaseNode.OnProcess(ProcessEventArgs e) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RoutingAndBreaking\BaseNode.cs:line 193
   at Ice.Core.RoutingAndBreaking.BaseNode.HandleProcess(Object sender, ProcessEventArgs e) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RoutingAndBreaking\BaseNode.cs:line 139
   at Ice.Core.RoutingAndBreaking.BaseNode.OnProcess(ProcessEventArgs e) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RoutingAndBreaking\BaseNode.cs:line 193
   at Ice.Core.RoutingAndBreaking.BaseNode.HandleProcess(Object sender, ProcessEventArgs e) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RoutingAndBreaking\BaseNode.cs:line 139
   at Ice.Core.RoutingAndBreaking.Workflow.OnProcess(ProcessEventArgs e) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RoutingAndBreaking\Workflow.cs:line 330
   at Ice.Core.RoutingAndBreaking.Workflow.Run(ReportInstanceInformation reportInstance) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RoutingAndBreaking\Workflow.cs:line 269
   at Ice.Core.RptBase.ReportSsrsDatabaseBuilder.RouteAndBreakReport(Func`2 executeCommand, Func`2 executeReader, SqlObjectsCreated sqlObjectsCreated) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\ReportSsrsDatabaseBuilder.cs:line 392
   at Ice.Core.RptBase.ReportSsrsDatabaseBuilder.ProcessReportWithDataInPlace(Func`2 executeCommand, Func`2 executeReader, SqlObjectsCreated sqlObjectsCreated) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\ReportSsrsDatabaseBuilder.cs:line 112
   at Ice.Core.RptBase.ReportDatabaseBuilder.ProcessReportWithDataInPlace(SqlConnection connection) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\ReportDatabaseBuilder.cs:line 130
   at Ice.Core.RptBase.ReportDatabaseBuilder.XMLClose() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\ReportDatabaseBuilder.cs:line 91
   at Ice.Core.RptTaskBase`1.XMLClose() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\RptTaskBase.cs:line 225
   at Ice.Core.RptTaskBase`1.RunDynamicCriteriaReport(IEnumerable`1 criteriaParameters, IEnumerable`1 criteriaMappings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\RptTaskBase.cs:line 161
   at Ice.Lib.DynamicCriteria.DynamicCriteriaReport.RunProcess(Int64 instanceTaskNum, String outputFileName) in C:\_releases\ICE\ICE5.0.100.0\Source\Server\Internal\Lib\DynamicCriteria\DynamicCriteriaReport.cs:line 44
   at Ice.Core.TaskBase`1.StartProcess(Int64 instanceTaskNum, String outputFileName) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\TaskBase\TaskBase.cs:line 80
   at Ice.Hosting.TaskCaller.InnerExecuteTask(IceDataContext newContext) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 113
   at Ice.Hosting.TaskCaller.ExecuteTask() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 61
   at Ice.Hosting.TaskCaller.ExecuteTask(IceDataContext dataContext, Boolean suppressTransaction) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 41
   at Ice.Services.Lib.RunTaskSvc.<>c__DisplayClass21_2.<InnerRunTask>b__1() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Services\Lib\RunTask\RunTask.cs:line 414
   at Ice.Lib.RunTask.BpmFriendlyTaskLauncher.Run(String sessionIdPrefix, IceContext db, Action taskRunner) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Services\Lib\RunTask\BpmFriendlyTaskLauncher.cs:line 57
   at Ice.Services.Lib.RunTaskSvc.InnerRunTask(Int64 ipTaskNum, Boolean suppressTransaction) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Services\Lib\RunTask\RunTask.cs:line 411

…I guess the upside is that we have a Sunday to hopefully run our failed tasks if they fix it by then.

GabeFranco · March 1, 2026, 8:36am

We are pleased to inform you that the issue has been resolved.

Thank you for your patience during this time. Please do not hesitate to raise a ticket in EpicCare Service Portal if you have any concerns or questions.

- March 01, 1:39am CST

JMPCONJ · March 1, 2026, 2:07pm

Saw their email notification yesterday and panicked…but fortunately we weren’t in the “affected subset” this time around. We got bit last time by an SSRS fail.

Getting really worried about this whole setup…the frequent outages and “maintenance windows” and container upgrades and all of it. Too much instability. We’re lucky in that the weekends are largely free in our operations…but I can’t imagine the impact this is having on large-scale 24x7x365 shops…

GabeFranco · March 1, 2026, 5:49pm

Running the SSRS jobs manually that failed sucked, but not too bad.

Now, I’m waiting on a high priority ticket to be responded to on a Sunday so I can get my Task Agent running again since ive got a backlog of a dozen tasks, half stuck in ACTIVE state and half in PENDING… wheeeeee

I’m running a SysTask Data Fix from an earlier time this same thing happened that just happens to not be expired.. but i’m making noise to Customer Support because, what the heck, man. I don’t need to pick up a drinking problem on Sunday

JMPCONJ · March 2, 2026, 1:30pm

Yikes…sorry to hear it. Getting very disillusioned with all this…it just shouldn’t be this difficult.

ajyoung11 · March 2, 2026, 2:45pm

We noticed around 12:30 pm CST on Saturday. We put in a ticket at 1:30pm because it was intermittent and we would have success occasionally. It took almost 3 HOURS for support to respond to the ticket. That is when they finally updated the status page too. I also tried calling 888-374-2679 and that didn’t work. It seemed like that number wasn’t in service. There should be a way of reporting an emergency cloud system down case that is like the Bat phone.