SSRS errors continuing since August

aosemwengie1 · December 18, 2025, 3:39pm

Whatever cloud maintenance they did last night didn’t help . . . and maybe made it worse.

DaveOlender · December 18, 2025, 3:59pm

Ms. Alisa: I am seeing the same thing but it appears to be random for me. If I retry the Upload or Preview it will work the second time (randomly).
DaveO

aosemwengie1 · December 18, 2025, 4:00pm

Yep, and same with previewing the reports - it will fail once, twice, sometimes 3x, then finally go through, at random. Extremely frustrating and a huge waste of time given this has now been unresolved for 5 months.

Olga · December 18, 2025, 4:09pm

Is it for some updated report rdl only? Or existing report that was working before?

kve · December 18, 2025, 4:09pm

It fails regularly when uploading an RDL (modified or new), and it fails often when Printing / Print-Previewing any report, new or old, stock or customized.

We had one day near the beginning of November where we had 36 failed report printing tasks out of 91 total. Over 35% failure rate.

EDIT: Most days it’s closer to 5% or 10%. Have not had a single day where there were no failures.

aosemwengie1 · December 18, 2025, 4:09pm

Any report. It can happen when uploading a rdl or just print previewing a base or custom report in Epicor.

Olga · December 18, 2025, 4:12pm

are there more details in that exception box? Because now it only show common error from SSRS

aosemwengie1 · December 18, 2025, 4:47pm

Ice.Core.SsrsReporting.SsrsCaller.SsrsException: An error occurred within the report server database.  This may be due to a connection failure, timeout or low disk condition within the database.
   at Ice.Core.SsrsReporting.SsrsCaller.RestReportingService.ThrowIfNotSuccess(HttpResponseMessage responseMessage) in F:\_Releases\12.0.100\Current\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\RestReportingService.cs:line 337
   at Ice.Core.SsrsReporting.SsrsCaller.RestReportingService.CreateCatalogItem(String ItemType, String Name, String Parent, Boolean Overwrite, Byte[] Definition, Property[] Properties, Warning[]& Warnings) in F:\_Releases\12.0.100\Current\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\RestReportingService.cs:line 74
   at Ice.Core.SsrsReporting.ReportProcessorBase.DeployReportIfNeeded(String printProgram, String fullReportPath, SsrsConfigurationInformation configurationInfo, Func`1 reportingServiceCreator) in F:\_Releases\12.0.100\Current\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 489
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport_HttpClient(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in F:\_Releases\12.0.100\Current\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 335
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in F:\_Releases\12.0.100\Current\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 242
   at Ice.Core.SsrsReporting.ReportProcessorBase.ProcessReportPart(String reportLocation) in F:\_Releases\12.0.100\Current\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 158
   at Ice.Core.SsrsReporting.StandardReportProcessor.ProcessReportParts() in F:\_Releases\12.0.100\Current\Source\Server\Internal\Lib\TaskLib\SsrsReporting\StandardReportProcessor.cs:line 31
   at Ice.Core.RoutingAndBreaking.ReportPersister.Persist(ReportInstanceInformation reportInstance, Func`2 reportsRenderer, Action`1 fillSysRptLstRow, Action`2 processReport, Func`3 filterTableAttachmentsFunc)
   at Ice.Core.RptBase.ReportSsrsDatabaseBuilder.RenderUnroutedSsrsReport() in F:\_Releases\12.0.100\Current\Source\Server\Internal\Lib\TaskLib\RptBase\ReportSsrsDatabaseBuilder.cs:line 355
   at Ice.Core.RptTaskBase`1.XMLClose() in F:\_Releases\12.0.100\Current\Source\Server\Internal\Lib\TaskLib\RptBase\RptTaskBase.cs:line 219
   at Erp.Internal.JC.JobTrav.RunProcess(Int64 instanceTaskNum, String outputFileName) in C:\_releases\ERP\ERP12.0.100.0\Source\Server\Internal\JC\JobTrav\JobTrav.cs:line 579
   at Ice.Hosting.TaskCaller.InnerExecuteTask(IceDataContext newContext) in C:\_releases\ICE\ICE5.0.100.12\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 70
   at Ice.Hosting.TaskCaller.ExecuteTask() in C:\_releases\ICE\ICE5.0.100.12\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 61
   at Ice.Lib.RunTask.BpmFriendlyTaskLauncher.Run(String sessionIdPrefix, IceContext db, Action taskRunner) in C:\_releases\ICE\ICE5.0.100.12\Source\Server\Services\Lib\RunTask\BpmFriendlyTaskLauncher.cs:line 57
   at Ice.Services.Lib.RunTaskSvc.InnerRunTask(Int64 ipTaskNum, Boolean suppressTransaction) in C:\_releases\ICE\ICE5.0.100.12\Source\Server\Services\Lib\RunTask\RunTask.cs:line 411

GabeFranco · December 18, 2025, 4:47pm

I have a detailed accounting of every various error message we’ve received from this SSRS failure in my case, CS0005076035, there is literally so much there it lags the web browser now

Error output varies, but it looks like a concurrency error on client/server code between AppServer and SSRS server, when it gets to “copy the rdls from .zip in CustomReportStore to SSRS server” - and multiple reports are running same instant - tries to overwrite .rdl file on ssrs server while open for reading, etc. fails. needs to have a queue or similar implemented when same report style (files) are printed on epicor side so ssrs server can be thread safe. because report style 1 can share files with report style 2, can share files with report style 3. middleware code will not queue these up and execute simultaneously. so not adequate to queue up report styles - code needs to be aware of .rdl files and queue up differing styles with overlapping rdl files.

I’ve been working with Epicor for months on this issue, at least they added multi-level inner exception output to the error messaging now, which is helpful.

One sample error below (note there are about 9 possible errors but all seem related to thread safety)

Entry #: 1, IsError?: TRUE, MsgType: ServerError
2025-12-18 03:47:39 PM
MsgText: Program Ice.Services.Lib.RunTask when executing task 6415316 raised an unexpected exception with the following message: RunTask:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.SocketsHttpHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendCoreAsync>g__Core|5_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendCoreAsync>g__Core|5_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   at Ice.Core.SsrsReporting.SsrsCaller.SoapCaller.Call[TContent,TResponse,TSoapHeader](String actionName, TContent content, TSoapHeader soapHeader) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\SoapCaller.cs:line 43
   at Ice.Core.SsrsReporting.SsrsRendererBase.TraceReportRendered(String printProgram, String ssrsRenderFormat, Func`1 timedAction) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsRendererBase.cs:line 81
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport_HttpClient(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 352
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 242
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReportForPrintOrEmailReport(RenderedSsrsReport renderedReport, Boolean serverSidePrint, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 685
   at Ice.Core.SsrsReporting.ReportProcessorBase.ProcessReportPart(String reportLocation) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 149
   at Ice.Core.SsrsReporting.ReportProcessorBase.ProcessReport(ReportDatabaseBuilder reportDatabaseBuilder, ReportInstanceInformation reportInstanceInformation) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 87
   at Ice.Core.RoutingAndBreaking.ReportPersister.Persist(ReportInstanceInformation reportInstance, Func`2 reportsRenderer, Action`1 fillSysRptLstRow, Action`2 processReport, Func`3 filterTableAttachmentsFunc) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RoutingAndBreaking\ReportPersister.cs:line 58
   at Ice.Core.RptBase.ReportSsrsDatabaseBuilder.RenderUnroutedSsrsReport() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\ReportSsrsDatabaseBuilder.cs:line 355
   at Ice.Core.RptTaskBase`1.XMLClose() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\RptTaskBase.cs:line 219
   at Erp.Internal.IM.PrintMovement.RunProcess(Int64 instanceTaskNum, String outputFileName) in C:\_releases\ERP\ERP12.0.100.0\Source\Server\Internal\IM\PrintMovement\PrintMovement.cs:line 255
   at Ice.Hosting.TaskCaller.InnerExecuteTask(IceDataContext newContext) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 70
   at Ice.Hosting.TaskCaller.ExecuteTask() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 61
   at Ice.Lib.RunTask.BpmFriendlyTaskLauncher.Run(String sessionIdPrefix, IceContext db, Action taskRunner) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Services\Lib\RunTask\BpmFriendlyTaskLauncher.cs:line 57
   at Ice.Services.Lib.RunTaskSvc.InnerRunTask(Int64 ipTaskNum, Boolean suppressTransaction) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Services\Lib\RunTask\RunTask.cs:line 411

It is almost always overlapping SSRS generation triggering these bugs: (cross company seems to trigger more) - note started on and ended on times for these failures)

Olga · December 18, 2025, 4:51pm

So the most detailed error is in system monitor, and in your case I see this error almost always:

The item ‘/SaaS_XXX/reports/ProductionDetailReport/ProdDtl’ cannot be found. —> Microsoft.ReportingServices.Diagnostics.Utilities.ItemNotFoundException: The item ‘/SaaS_XXX/reports/ProductionDetailReport/ProdDtl’ cannot be found.

Olga · December 18, 2025, 4:52pm

What is in System Monitor task error?

aosemwengie1 · December 18, 2025, 4:53pm

The error I pasted above was copied directly from the system monitor. Here is where I copied that detail from:

GabeFranco · December 18, 2025, 4:53pm

That one is probably my 2nd most common error, “cannot be found”

This one is my most common one:

Entry #: 1, IsError?: TRUE, MsgType: ServerError
2025-12-17 06:48:54 PM
MsgText: Program Ice.Services.Lib.RunTask when executing task 6413373 raised an unexpected exception with the following message: RunTask:
Ice.Core.SsrsReporting.SsrsCaller.SsrsException: An error occurred within the report server database.  This may be due to a connection failure, timeout or low disk condition within the database.
   at Ice.Core.SsrsReporting.SsrsCaller.RestReportingService.ThrowIfNotSuccess(HttpResponseMessage responseMessage) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\RestReportingService.cs:line 337
   at Ice.Core.SsrsReporting.SsrsCaller.RestReportingService.CreateCatalogItem(String ItemType, String Name, String Parent, Boolean Overwrite, Byte[] Definition, Property[] Properties, Warning[]& Warnings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\RestReportingService.cs:line 74
   at Ice.Core.SsrsReporting.ReportProcessorBase.DeployReportIfNeeded(String printProgram, String fullReportPath, SsrsConfigurationInformation configurationInfo, Func`1 reportingServiceCreator) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 489
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport_HttpClient(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 335
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 242
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReportForPrintOrEmailReport(RenderedSsrsReport renderedReport, Boolean serverSidePrint, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 639
   at Ice.Core.SsrsReporting.ReportProcessorBase.ProcessReportPart(String reportLocation) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 153
   at Ice.Core.SsrsReporting.StandardReportProcessor.ProcessReportParts() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\StandardReportProcessor.cs:line 31
   at Ice.Core.RoutingAndBreaking.ReportPersister.Persist(ReportInstanceInformation reportInstance, Func`2 reportsRenderer, Action`1 fillSysRptLstRow, Action`2 processReport, Func`3 filterTableAttachmentsFunc) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RoutingAndBreaking\ReportPersister.cs:line 58
   at Ice.Core.RptBase.ReportSsrsDatabaseBuilder.RenderUnroutedSsrsReport() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\ReportSsrsDatabaseBuilder.cs:line 355
   at Ice.Core.RptTaskBase`1.XMLClose() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\RptTaskBase.cs:line 219
   at Erp.Internal.JC.JCR35.RunProcess(Int64 instanceTaskNum, String outputFileName) in C:\_releases\ERP\ERP12.0.100.0\Source\Server\Internal\JC\JCR35\JCR35.cs:line 921
   at Ice.Hosting.TaskCaller.InnerExecuteTask(IceDataContext newContext) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 70
   at Ice.Hosting.TaskCaller.ExecuteTask() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 61
   at Ice.Lib.RunTask.BpmFriendlyTaskLauncher.Run(String sessionIdPrefix, IceContext db, Action taskRunner) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Services\Lib\RunTask\BpmFriendlyTaskLauncher.cs:line 57
   at Ice.Services.Lib.RunTaskSvc.InnerRunTask(Int64 ipTaskNum, Boolean suppressTransaction) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Services\Lib\RunTask\RunTask.cs:line 411

Olga · December 18, 2025, 4:55pm

I see, so it only shows top level message: An error occurred within the report server database. This may be due to a connection failure, timeout or low disk condition within the database.

and the details are in SSRS log.

GabeFranco · December 18, 2025, 4:56pm

…and we don’t get to see those in SaaS, so stuck waiting for Epicor CS to back and forth with Microsoft Support on this issue since 2025-08-25.. (i think it was when 2025.1 came out?)

aosemwengie1 · December 18, 2025, 4:56pm

Exactly.

Olga · December 18, 2025, 5:01pm

I assume it is Job Traveler report. So did you edit something in RDL before print or it was already used previously as it is?

aosemwengie1 · December 18, 2025, 5:03pm

Same error happens all the time on both base and custom reports. I can not say emphatically enough that this has absolutely nothing to do with customization.

Olga · December 18, 2025, 5:14pm

I just see CreateCatalogItem in the stack that means it is trying to create some report in SSRS. Question is if it is necessary in your case. As I understand it should be done only after you edited on of the report RDLs.

GabeFranco · December 18, 2025, 5:15pm

This code path seems to run on every execution of a report that generates this error. Not only after edit. It happens for Epicor oob report style too.