Just me having all the fun? Waiting on a DBA to help… we are getting SQL out of memory errors on SaaS 2025.1.15 Flex 2. Seemingly random triggers. Did a search in my server logs and I can see that it happened a few times over the past 2 days, but today 100 occurances of this exception.
<Op Utc="2025-12-11T14:29:22.1640641Z" act="Ice:BO:SysMonitor/SysMonitorSvcContract/GetMonitorDataKeepIdleTime" correlationId="038ba4b9-16cc-41e7-98a2-de3780b84d46" dur="9.1655" cli="10.54.3.197:55020" usr="xxxx" machine="LV12345" pid="78816" tid="190">
<Exception><![CDATA[Microsoft.Data.SqlClient.SqlException (0x80131904): There is insufficient memory in the resource pool 'hkTempDB' to run this operation on memory-optimized tables. See 'http://go.microsoft.com/fwlink/?LinkID=614951' for more information.
at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, SqlCommand command, Boolean callerHasConnectionLock, Boolean asyncClose)
at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at Microsoft.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean isAsync, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String method)
at Microsoft.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at Microsoft.Data.SqlClient.SqlCommand.ExecuteReader()
at Ice.Services.BO.SysMonitorSvc.GetMonitorDataKeepIdleTime(SqlConnection connection, ISession session, IReadOnlyCollection`1 userCompanies, Boolean querySysRptLst, String workstationId, Boolean querySysTask, Boolean querySysTaskLog, Boolean historyOnly, DateTime startedOn, Boolean retrieveAllTasks) in C:\_releases\ICE\ICE5.0.100.0\Source\Server\Services\BO\SysMonitor\SysMonitor.cs:line 110
at Ice.Services.BO.SysMonitorSvcFacade.GetMonitorDataKeepIdleTime(Boolean querySysRptLst, String workstationId, Boolean querySysTask, Boolean querySysTaskLog, Boolean historyOnly, DateTime startedOn, Boolean retrieveAllTasks) in C:\_releases\ICE\ICE5.0.100.0\Source\Server\Services\BO\SysMonitor\SysMonitorSvcFacade.cs:line 52
at InvokeStub_SysMonitorSvcFacade.GetMonitorDataKeepIdleTime(Object, Span`1)
at System.Reflection.MethodBaseInvoker.InvokeWithManyArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
--- End of stack trace from previous location ---
at Ice.Hosting.RestApi.OperationInvokeHelper.CallInvoker(Type svcType, MethodInfo mi, Object[] inputs, Boolean isRoot, IceDataContext dataContext) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Framework\Epicor.Ice\Hosting\RestApi\OperationInvokeHelper.cs:line 401
at Ice.Hosting.RestApi.OperationInvokeHelper.MakeServiceCall(IceDataContext dataContext, Type svcType, MethodInfo mi, Object[] parameters) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Framework\Epicor.Ice\Hosting\RestApi\OperationInvokeHelper.cs:line 350
at Ice.Hosting.RestApi.OperationInvokeHelper.CallCustomMethod(String serviceName, MethodInfo mi, Object[] paramList) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Framework\Epicor.Ice\Hosting\RestApi\OperationInvokeHelper.cs:line 305
at Epicor.RESTApi.DataSources.CustomMethodCaller.CallMethodWithDynamicData(String serviceId, String methodName, IInputParamResolver inputResolver, Action`1 headersPublisher, Action`1 verifyMethod) in C:\_releases\ICE\ICE5.0.100.0\Source\Server\Framework\Epicor.RESTApi.Common\DataSources\Main\CustomMethodCaller.cs:line 60
at Ice.Hosting.AspNetCore.Controllers.ServiceMethodCallController.InvokePostMethod(Object input) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\Controllers\ServiceMethodCallController.cs:line 57
at lambda_method2394907(Closure, Object, Object[])
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at CoreWCF.Channels.ServiceModelHttpMiddleware.InvokeAsync(HttpContext context)
at CoreWCF.Channels.MetadataMiddleware.InvokeAsync(HttpContext context)
at Ice.Hosting.AspNetCore.Middleware.DynamicAssemblyPartMiddleware.Invoke(HttpContext context, CurrentCallInformationService currentCallInformation, ControllerLoader controllerLoader) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\Middleware\DynamicAssemblyPartMiddleware.cs:line 32
at Ice.Hosting.AspNetCore.Middleware.IPEnforcerMiddleware.Invoke(HttpContext httpContext) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\Middleware\IPEnforcerMiddleware.cs:line 42
at Ice.Hosting.AspNetCore.ETags.ETagMiddleware.Invoke(HttpContext httpContext) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\ETags\ETagMiddleware.cs:line 89
at Ice.Hosting.AspNetCore.ETags.ETagMiddleware.Invoke(HttpContext httpContext) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\ETags\ETagMiddleware.cs:line 107
at Ice.Hosting.AspNetCore.Middleware.DecompressRequestMiddleware.Invoke(HttpContext context) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\Middleware\DecompressRequestMiddleware.cs:line 41
at Ice.Hosting.AspNetCore.Middleware.AuthenticationMiddleware.InvokeAsync(HttpContext httpContext, CurrentCallInformationService callInformation) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\Middleware\AuthenticationMiddleware.cs:line 76
at Ice.Hosting.AspNetCore.Middleware.CallHeaderMiddleware.InvokeAsync(HttpContext httpContext) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\Middleware\CallHeaderMiddleware.cs:line 51
at Ice.Hosting.AspNetCore.Middleware.OperationDisposerMiddleware.InvokeAsync(HttpContext httpContext) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Hosting\AspNetCore\Ice.Hosting.AspNetCore\Middleware\OperationDisposerMiddleware.cs:line 23
ClientConnectionId:3628ec0f-8c15-4951-932b-ab155fe3a459
Error Number:41805,State:109,Class:16]]></Exception>
</Op>