:dumpster_fire: No April Fools Joke - Systems Issues Again

[Randy (Randy Stulce)]

We’re back up too

[hackaphreaka (Garret Travers)]

Ah yes, the classic ‘overloaded SSRS service one of the reporting machine’ problem.

[JMPCONJ (John Magee)]

@hackaphreaka

‘overloaded SSRS service one of the reporting machine’

All your base are - ah nevermind

[Don (Vito Signorelli)]

Mostly up, odd question. Anyone able to run a production detail report on a job? Only seeing issues with that report.

[Randy (Randy Stulce)]

I just tried one and worked for us.

[klincecum (Kevin Lincecum)]

@klincecum

hopefully I’m not in that subset!

I spoke too soon.

Seems to be ok now.

[Don (Vito Signorelli)]

Thank you for checking! I have not gotten one through for 3 different people.

[Randy (Randy Stulce)]

Is there any kind of error message?

[Don (Vito Signorelli)]

Just the regular…

Program Ice.Services.Lib.RunTask when executing task 1657330 raised an unexpected exception with the following message: RunTask:
Ice.Core.SsrsReporting.SsrsCaller.SsrsException: An error occurred within the report server database. This may be due to a connection failure, timeout or low disk condition within the database.
at Ice.Core.SsrsReporting.SsrsCaller.RestReportingService.ThrowIfNotSuccess(HttpResponseMessage responseMessage) in C:_releases\ICE\ICE5.1.100.12\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\RestReportingService.cs:line 337
at Ice.Core.SsrsReporting.SsrsCaller.RestReportingService.CreateCatalogItem(String ItemType, String Name, String Parent, Boolean Overwrite, Byte Definition, Property Properties, Warning& Warnings) in C:_releases\ICE\ICE5.1.100.12\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\RestReportingService.cs:line 74
at Ice.Core.SsrsReporting.SsrsCaller.SsrsFolderHelper.CreateReport(String reportPath, Byte reportBytes) in C:_releases\ICE\ICE5.1.100.12\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\SsrsFolderHelper.cs:line 99
at Ice.Core.SsrsReporting.ReportProcessorBase.DeployReportIfNeeded(String printProgram, String fullReportPath, SsrsConfigurationInformation configurationInfo, Func1 reportingServiceCreator) in C:\_releases\ICE\ICE5.1.100.12\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 529 at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport_HttpClient(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.1.100.12\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 336 at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.1.100.12\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 243 at Ice.Core.SsrsReporting.ReportProcessorBase.ProcessReportPart(String reportLocation) in C:\_releases\ICE\ICE5.1.100.12\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 159 at Ice.Core.RoutingAndBreaking.ReportPersister.Persist(ReportInstanceInformation reportInstance, Func2 reportsRenderer, Action1 fillSysRptLstRow, Action2 processReport, Func3 filterTableAttachmentsFunc) in C:\_releases\ICE\ICE5.1.100.12\Source\Server\Internal\Lib\TaskLib\RoutingAndBreaking\ReportPersister.cs:line 57 at Ice.Core.RptBase.ReportSsrsDatabaseBuilder.RenderUnroutedSsrsReport() in C:\_releases\ICE\ICE5.1.100.12\Source\Server\Internal\Lib\TaskLib\RptBase\ReportSsrsDatabaseBuilder.cs:line 355 at Ice.Core.RptTaskBase1.XMLClose() in C:_releases\ICE\ICE5.1.100.12\Source\Server\Internal\Lib\TaskLib\RptBase\RptTaskBase.cs:line 219
at Erp.Internal.JC.JCR35.RunProcess(Int64 instanceTaskNum, String outputFileName) in C:_releases\ERP\ERP12.1.100.0\Source\Server\Internal\JC\JCR35\JCR35.cs:line 921
at Ice.Hosting.TaskCaller.InnerExecuteTask(IceDataContext newContext) in C:_releases\ICE\ICE5.1.100.12\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 71
at Ice.Hosting.TaskCaller.ExecuteTask() in C:_releases\ICE\ICE5.1.100.12\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 62
at Ice.Lib.RunTask.BpmFriendlyTaskLauncher.Run(String sessionIdPrefix, IceContext db, Action taskRunner) in C:_releases\ICE\ICE5.1.100.0\Source\Server\Services\Lib\RunTask\BpmFriendlyTaskLauncher.cs:line 57
at Ice.Services.Lib.RunTaskSvc.InnerRunTask(Int64 ipTaskNum, Boolean suppressTransaction) in C:_releases\ICE\ICE5.1.100.0\Source\Server\Services\Lib\RunTask\RunTask.cs:line 349

[aosemwengie1 (Alisa Osemwengie)]

That is the error that is supposed to be fixed in 2025.2.16.

[Chris_Conn]

No problems on prem (again)…

[Don (Vito Signorelli)]

Yeah I’m not going to hold my breath. Ticket open since early december for that one.

[JMPCONJ (John Magee)]

@Don

Ticket open since early december

Year?

[Don (Vito Signorelli)]

Luckily just 2025

[Don (Vito Signorelli)]

Can confirm this is not fixed in .16

[aosemwengie1 (Alisa Osemwengie)]

More details? Do you have .16 in non prod or or prod? What errors are you seeing?

[Don (Vito Signorelli)]

Sure, we were told through the ticket that i entered yesterday during this whole issue that we needed to update so we bit the bullet last night. Pilot was already on .16 so we updated to .16 on prod as well. The error is identical, it really hates production detail but it is intermittent elsewhere also. It happens in the browser as well as the smart client. No difference between the classic and kinetic screens. Here is the exact error…

Program Ice.Services.Lib.RunTask when executing task 1658877 raised an unexpected exception with the following message: RunTask:
Ice.Core.SsrsReporting.SsrsCaller.SsrsException: 2026-04-02 14:56:20.5620 UTC An error occurred within the report server database.  This may be due to a connection failure, timeout or low disk condition within the database.
   at Ice.Core.SsrsReporting.SsrsCaller.RestReportingService.ThrowIfNotSuccess(HttpResponseMessage responseMessage) in C:\_releases\ICE\ICE5.1.100.16\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\RestReportingService.cs:line 337
   at Ice.Core.SsrsReporting.SsrsCaller.RestReportingService.CreateCatalogItem(String ItemType, String Name, String Parent, Boolean Overwrite, Byte[] Definition, Property[] Properties, Warning[]& Warnings) in C:\_releases\ICE\ICE5.1.100.16\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\RestReportingService.cs:line 74
   at Ice.Core.SsrsReporting.ReportProcessorBase.DeployReportIfNeeded(String printProgram, String fullReportPath, SsrsConfigurationInformation configurationInfo, Func`1 reportingServiceCreator) in C:\_releases\ICE\ICE5.1.100.16\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 550
   at Ice.Core.SsrsReporting.ReportProcessorBase.EnsureReportValidated(String key, String fullReportPath) in C:\_releases\ICE\ICE5.1.100.16\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportHashCache.cs:line 93
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport_HttpClient(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.1.100.16\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 338
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.1.100.16\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 243
   at Ice.Core.SsrsReporting.ReportProcessorBase.ProcessReportPart(String reportLocation) in C:\_releases\ICE\ICE5.1.100.16\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 159
   at Ice.Core.SsrsReporting.StandardReportProcessor.ProcessReportParts() in C:\_releases\ICE\ICE5.1.100.16\Source\Server\Internal\Lib\TaskLib\SsrsReporting\StandardReportProcessor.cs:line 31
   at Ice.Core.RoutingAndBreaking.ReportPersister.Persist(ReportInstanceInformation reportInstance, Func`2 reportsRenderer, Action`1 fillSysRptLstRow, Action`2 processReport, Func`3 filterTableAttachmentsFunc)
   at Ice.Core.RptBase.ReportSsrsDatabaseBuilder.RenderUnroutedSsrsReport() in C:\_releases\ICE\ICE5.1.100.16\Source\Server\Internal\Lib\TaskLib\RptBase\ReportSsrsDatabaseBuilder.cs:line 355
   at Ice.Core.RptTaskBase`1.XMLClose() in C:\_releases\ICE\ICE5.1.100.16\Source\Server\Internal\Lib\TaskLib\RptBase\RptTaskBase.cs:line 219
   at Erp.Internal.JC.JCR35.RunProcess(Int64 instanceTaskNum, String outputFileName) in C:\_releases\ERP\ERP12.1.100.0\Source\Server\Internal\JC\JCR35\JCR35.cs:line 921
   at Ice.Hosting.TaskCaller.InnerExecuteTask(IceDataContext newContext) in C:\_releases\ICE\ICE5.1.100.16\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 71
   at Ice.Hosting.TaskCaller.ExecuteTask() in C:\_releases\ICE\ICE5.1.100.16\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 62
   at Ice.Lib.RunTask.BpmFriendlyTaskLauncher.Run(String sessionIdPrefix, IceContext db, Action taskRunner) in C:\_releases\ICE\ICE5.1.100.0\Source\Server\Services\Lib\RunTask\BpmFriendlyTaskLauncher.cs:line 57
   at Ice.Services.Lib.RunTaskSvc.InnerRunTask(Int64 ipTaskNum, Boolean suppressTransaction) in C:\_releases\ICE\ICE5.1.100.0\Source\Server\Services\Lib\RunTask\RunTask.cs:line 349

Looks like the same issue we have been having all along. Unfortunately, I have gotten 1 out of around 30-40 attempts for production detail report. Luckily I just needed the 1 for now.

[aosemwengie1 (Alisa Osemwengie)]

Yikes, thanks for sharing. What does support say about the issue continuing to occur even after the update?

[Don (Vito Signorelli)]

Just waiting on a response. Last response was they will look into it.

[chaddb (Chadd Blanchard)]

The low disk issue rears its ugly head, again.