SSRS errors continuing since August

This makes sense because this method creates report in SSRS and something goes wrong.
Question is, should it create report in the first place, if it did not change and was already used.

1 Like

To me it seems as though the issue is related to a call that is not showing in the call stack because it already returned successful - that call checking to see if it should create file on SSRS or not. It seems to be linked to concurrency in report generation - if another report is running utilizing that same RDL at that moment that function returns that SSRS upload is needed.

1 Like

Well the question is why it returned True to update rdl. Was it correct or not? Was something changed in the report previously or not.

1 Like

No. In all cases of this error there has been no change to the RDL at all. Initially the errors were discovered after the 2025.1 update on my side, with no RDL changes to what was in place prior to the update.

The error happens with Epicor out of box report styles, so there is no possibility for user changing it.

3 Likes

well it still should be installed on the very first run, but other than that nothing else should be changed after.

2 Likes

Support continues to insist that its a “Microsoft” issue.

2 Likes

It’s a very difficult error to track down, for example at Report/Style level, we can see this report/style which is Epicor Out of Box (Rpt:ProdDtl/Rdd:JobProdDtl/Style:2) was successfully printed, same company, same user at 2:47, on jobnum 1234, then at 3:54 he got error with CreateCatalogItem with JobNum 4321. 1 hour later he reprinted the same jobnum, style, rdd/report, no error.

2 Likes

If you look at it as a function of time (in order of systask, ignoring company, user, task, etc.) - it becomes more clear that it is some sort of concurrency issue, the error always has something overlapping in task agent execution - (note overlapping process failed as well) - this is the same error task as above.

2 Likes

Ok I see, this is failure during report creation.
Did you see the one about item not found recently?

2 Likes

Yes, here is the last one from 12/12:

Entry #: 1, IsError?: TRUE, MsgType: ServerError
2025-12-12 08:19:39 PM
MsgText: Program Ice.Services.Lib.RunTask when executing task 6403624 raised an unexpected exception with the following message: RunTask:
Ice.Core.SsrsReporting.SsrsCaller.SsrsException: The SSRS server returned the status code 500 (InternalServerError) with the following error text:
The item '/SaaS695SSRS_27388/reports/ProductionDetailReport/ProdDtl' cannot be found. ---> Microsoft.ReportingServices.Diagnostics.Utilities.ItemNotFoundException: The item '/SaaS695SSRS_27388/reports/ProductionDetailReport/ProdDtl' cannot be found.
   at Ice.Core.SsrsReporting.SsrsCaller.SoapCaller.Call[TContent,TResponse,TSoapHeader](String actionName, TContent content, TSoapHeader soapHeader) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\SoapCaller.cs:line 47
   at Ice.Core.SsrsReporting.SsrsRendererBase.TraceReportRendered(String printProgram, String ssrsRenderFormat, Func`1 timedAction) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsRendererBase.cs:line 81
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport_HttpClient(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 352
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 242
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReportForPrintOrEmailReport(RenderedSsrsReport renderedReport, Boolean serverSidePrint, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 639
   at Ice.Core.SsrsReporting.ReportProcessorBase.ProcessReportPart(String reportLocation) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 153
   at Ice.Core.SsrsReporting.StandardReportProcessor.ProcessReportParts() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\StandardReportProcessor.cs:line 31
   at Ice.Core.RoutingAndBreaking.ReportPersister.Persist(ReportInstanceInformation reportInstance, Func`2 reportsRenderer, Action`1 fillSysRptLstRow, Action`2 processReport, Func`3 filterTableAttachmentsFunc)
   at Ice.Core.RptBase.ReportSsrsDatabaseBuilder.RenderUnroutedSsrsReport() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\ReportSsrsDatabaseBuilder.cs:line 355
   at Ice.Core.RptTaskBase`1.XMLClose() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\RptTaskBase.cs:line 219
   at Erp.Internal.JC.JCR35.RunProcess(Int64 instanceTaskNum, String outputFileName) in C:\_releases\ERP\ERP12.0.100.0\Source\Server\Internal\JC\JCR35\JCR35.cs:line 921
   at Ice.Hosting.TaskCaller.InnerExecuteTask(IceDataContext newContext) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 70
   at Ice.Hosting.TaskCaller.ExecuteTask() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 61
   at Ice.Lib.RunTask.BpmFriendlyTaskLauncher.Run(String sessionIdPrefix, IceContext db, Action taskRunner) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Services\Lib\RunTask\BpmFriendlyTaskLauncher.cs:line 57
   at Ice.Services.Lib.RunTaskSvc.InnerRunTask(Int64 ipTaskNum, Boolean suppressTransaction) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Services\Lib\RunTask\RunTask.cs:line 411

Params:

AgentCompareString: 
AllowAsm: 0
AllowJobProfitability: 1
ArchiveCode: 0
AsmList: 0
AttachmentType: PDF
AutoAction: SSRSClientPrint
ByProdGroup: 0
CallContextData: <ContextDataSet xmlns="http://www.epicor.com/Ice/300/Bpm/Context">
  <BpmData>
    <Password />
    <ButtonValue>0</ButtonValue>
    <Character01 />
    <Character02 />
    <Character03 />
    <Character04 />
    <Character05 />
    <Character06 />
    <Character07 />
    <Character08 />
    <Character09 />
    <Character10 />
    <Character11 />
    <Character12 />
    <Character13 />
    <Character14 />
    <Character15 />
    <Character16 />
    <Character17 />
    <Character18 />
    <Character19 />
    <Character20 />
    <Number01>0.0</Number01>
    <Number02>0.0</Number02>
    <Number03>0.0</Number03>
    <Number04>0.0</Number04>
    <Number05>0.0</Number05>
    <Number06>0.0</Number06>
    <Number07>0.0</Number07>
    <Number08>0.0</Number08>
    <Number09>0.0</Number09>
    <Number10>0.0</Number10>
    <Number11>0.0</Number11>
    <Number12>0.0</Number12>
    <Number13>0.0</Number13>
    <Number14>0.0</Number14>
    <Number15>0.0</Number15>
    <Number16>0.0</Number16>
    <Number17>0.0</Number17>
    <Number18>0.0</Number18>
    <Number19>0.0</Number19>
    <Number20>0.0</Number20>
    <Checkbox01>false</Checkbox01>
    <Checkbox02>false</Checkbox02>
    <Checkbox03>false</Checkbox03>
    <Checkbox04>false</Checkbox04>
    <Checkbox05>false</Checkbox05>
    <Checkbox06>false</Checkbox06>
    <Checkbox07>false</Checkbox07>
    <Checkbox08>false</Checkbox08>
    <Checkbox09>false</Checkbox09>
    <Checkbox10>false</Checkbox10>
    <ShortChar01 />
    <ShortChar02 />
    <ShortChar03 />
    <ShortChar04 />
    <ShortChar05 />
    <ShortChar06 />
    <ShortChar07 />
    <ShortChar08 />
    <ShortChar09 />
    <ShortChar10 />
    <SysRowID>00000000-0000-0000-0000-000000000000</SysRowID>
    <RowMod />
  </BpmData>
  <Client>
    <ClientType>WinClient</ClientType>
    <ProcessId>JCRP4004</ProcessId>
    <AssemblyName>Erp.UIRpt.JobProdDtl</AssemblyName>
    <CustomizationId />
    <CurrentUserId><<<<REDACTED>>>></CurrentUserId>
    <CurrentCompany><<<<REDACTED>>>></CurrentCompany>
    <CurrentPlant><<<<REDACTED>>>></CurrentPlant>
    <CGCCode />
    <SysRowID>00000000-0000-0000-0000-000000000000</SysRowID>
    <RowMod />
  </Client>
</ContextDataSet>
Consolidate: 0
Cur-Comp: <<<<REDACTED>>>>
Cur-CountryCode: US
CUR-LangID: enu
Cur-Plant: <<<<REDACTED>>>>
Cur-TaskClientID: <<<<REDACTED>>>>
Cur-WorkStationID: 
DateFormat: m/d/yyyy
DCD-UserID: <<<<REDACTED>>>>
DecimalsCost: 0
DecimalsGeneral: 0
DecimalsPrice: 0
DesignMode: 0
EMailBCC: 
EMailBody: 
EMailCC: 
EMailTo: 
ExclMiscCharg: 0
FaxNumber: 
FaxSubject: 
FaxTo: 
FirstArtTrans: 0
GlbDecimalsCost: 0
GlbDecimalsGeneral: 0
GlbDecimalsPrice: 0
JobList: 099627
LaborTrans: 1
MaterialTrans: 1
NumericFormat: ,.
PrintAttributes: 0
PrintBreakDown: 0
PrintDowntime: 0
PrinterName: HP LaserJet Pro M402-M403 n-dne PCL 6
PrintProfit: 1
PrintReportParameters: 0
PrintSN: 0
ProcessCompany: 
ProcessID: 
ProcessSystemCode: 
ProcessTaskNum: 0
RecurringTask: 0
ReportCultureCode: en-US
ReportCurrencyCode: USD
ReportID: ProdDtl
ReportParameterDataSetId: JobProdDtl
ReportParameterDataSetSystemCode: Erp
ReportStyleNum: 2
RptFileType: XML
RptPageSettings: Color=False,Landscape=False,AutoRotate=False,PaperSize=[Kind="Letter" PaperName="Letter" Height=1100 Width=850],PaperSource=[SourceName="Printer auto select" Kind="AutomaticFeed"],PrinterResolution=[Kind="Custom" X=600 Y=600]
RptPrinterSettings: PrinterName="HP LaserJet Pro M402-M403 n-dne PCL 6",Copies=1,Collate=False,Duplex=Simplex,FromPage=1,ToPage=0
RptVersion: 
SSRSEnableRouting: 0
SSRSRenderFormat: PDF
SummaryOnly: 0
TaskNote: 
UIXml: 
WorkstationID: <<<<REDACTED>>>>
2 Likes

What is the error in the previous row for the same report?

1 Like

Same Report/RDD/Style (Out of box) printed 3 minutes prior to error run. same user, same company, no error.

Previous error on same report was 2 days prior:

Entry #: 1, IsError?: TRUE, MsgType: ServerError
2025-12-10 09:39:07 PM
MsgText: Program Ice.Services.Lib.RunTask when executing task 6398537 raised an unexpected exception with the following message: RunTask:
Ice.Core.SsrsReporting.SsrsCaller.SsrsException: An error occurred within the report server database.  This may be due to a connection failure, timeout or low disk condition within the database.
   at Ice.Core.SsrsReporting.SsrsCaller.RestReportingService.ThrowIfNotSuccess(HttpResponseMessage responseMessage) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\RestReportingService.cs:line 337
   at Ice.Core.SsrsReporting.SsrsCaller.RestReportingService.CreateCatalogItem(String ItemType, String Name, String Parent, Boolean Overwrite, Byte[] Definition, Property[] Properties, Warning[]& Warnings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\RestReportingService.cs:line 74
   at Ice.Core.SsrsReporting.SsrsCaller.SsrsFolderHelper.CreateReport(String reportPath, Byte[] reportBytes) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\SsrsFolderHelper.cs:line 99
   at Ice.Core.SsrsReporting.ReportProcessorBase.DeployReportIfNeeded(String printProgram, String fullReportPath, SsrsConfigurationInformation configurationInfo, Func`1 reportingServiceCreator) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 489
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport_HttpClient(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 335
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReport(String ssrsRenderFormat, String printProgram, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 242
   at Ice.Core.SsrsReporting.ReportProcessorBase.RenderReportForPrintOrEmailReport(RenderedSsrsReport renderedReport, Boolean serverSidePrint, Boolean ignorePageSettings) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 639
   at Ice.Core.SsrsReporting.ReportProcessorBase.ProcessReportPart(String reportLocation) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\ReportProcessorBase.cs:line 153
   at Ice.Core.SsrsReporting.StandardReportProcessor.ProcessReportParts() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\SsrsReporting\StandardReportProcessor.cs:line 31
   at Ice.Core.RoutingAndBreaking.ReportPersister.Persist(ReportInstanceInformation reportInstance, Func`2 reportsRenderer, Action`1 fillSysRptLstRow, Action`2 processReport, Func`3 filterTableAttachmentsFunc)
   at Ice.Core.RptBase.ReportSsrsDatabaseBuilder.RenderUnroutedSsrsReport() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\ReportSsrsDatabaseBuilder.cs:line 355
   at Ice.Core.RptTaskBase`1.XMLClose() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Internal\Lib\TaskLib\RptBase\RptTaskBase.cs:line 219
   at Erp.Internal.JC.JCR35.RunProcess(Int64 instanceTaskNum, String outputFileName) in C:\_releases\ERP\ERP12.0.100.0\Source\Server\Internal\JC\JCR35\JCR35.cs:line 921
   at Ice.Hosting.TaskCaller.InnerExecuteTask(IceDataContext newContext) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 70
   at Ice.Hosting.TaskCaller.ExecuteTask() in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 61
   at Ice.Lib.RunTask.BpmFriendlyTaskLauncher.Run(String sessionIdPrefix, IceContext db, Action taskRunner) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Services\Lib\RunTask\BpmFriendlyTaskLauncher.cs:line 57
   at Ice.Services.Lib.RunTaskSvc.InnerRunTask(Int64 ipTaskNum, Boolean suppressTransaction) in C:\_releases\ICE\ICE5.0.100.15\Source\Server\Services\Lib\RunTask\RunTask.cs:line 411

2 Likes

This is the error when attempting to upload:


An error occurred within the report server database.  This may be due to a connection failure, timeout or low disk condition within the database.

Exception caught in: Epicor.ServiceModel

Error Detail 
============
Correlation ID:  5b3a495f-6009-49ec-9c43-1748e63b7ff5
Description:  An error occurred within the report server database.  This may be due to a connection failure, timeout or low disk condition within the database.
Program:  Ice.Core.TaskBase.dll
Method:  .ctor
Line Number:  40
Column Number:  11

Client Stack Trace 
==================
   at Epicor.ServiceModel.Channels.ImplBase.CallWithCommunicationFailureRetry(String methodName, ProxyValuesIn valuesIn, ProxyValuesOut valuesOut, RestRpcValueSerializer serializer)
   at Epicor.ServiceModel.Channels.ImplBase.CallWithMultistepBpmHandling(String methodName, ProxyValuesIn valuesIn, ProxyValuesOut valuesOut, Boolean useSparseCopy)
   at Epicor.ServiceModel.Channels.ImplBase.Call(String methodName, ProxyValuesIn valuesIn, ProxyValuesOut valuesOut, Boolean useSparseCopy)
   at Ice.Proxy.BO.SsrsReportDesignImpl.UploadAndRenderDesign(Guid sysRowID, Byte[] compressedReportDefinitions, String outputFormat)
   at Ice.Adapters.SsrsReportDesignAdapter.UploadAndRenderDesign(Guid sysRowID, Byte[] compressedReportDefinitions, String outputFormat)
   at Ice.UI.App.SsrsReportDesign.Transaction.UploadAndRender()
   at Ice.UI.App.SsrsReportDesign.ReportDesignForm.BaseToolbarsManager_ToolClick(Object sender, ToolClickEventArgs e)
   at Infragistics.Win.UltraWinToolbars.ToolClickEventHandler.Invoke(Object sender, ToolClickEventArgs e)
   at Infragistics.Win.UltraWinToolbars.UltraToolbarsManager.OnToolClick(ToolClickEventArgs e)
   at Infragistics.Win.UltraWinToolbars.UltraToolbarsManager.FireEvent(ToolbarEventIds id, EventArgs e)
   at Infragistics.Win.UltraWinToolbars.ToolBase.OnToolClick()
   at Infragistics.Win.UltraWinToolbars.ToolMenuItem.OnClick()
   at Infragistics.Win.UltraWinToolbars.PopupMenuItemUIElement.DoClickProcessing(MouseEventArgs e)
   at Infragistics.Win.UltraWinToolbars.PopupMenuItemUIElement.OnMouseUp(MouseEventArgs e)
   at Infragistics.Win.TextUIElementBase.OnMouseUp(MouseEventArgs e)
   at Infragistics.Win.ControlUIElementBase.ProcessMouseUpHelper(Object sender, MouseEventArgs e)
   at Infragistics.Win.ControlUIElementBase.ProcessMouseUp(Object sender, MouseEventArgs e)
   at Infragistics.Win.Utilities.ProcessEvent(Control control, ProcessEvent eventToProcess, EventArgs e)
   at Infragistics.Win.UltraWinToolbars.PopupControlBase.OnMouseUp(MouseEventArgs e)
   at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
2 Likes

Here is a new one. Love the exceptional detail.

Server Side Exception



Exception caught in: Epicor.ServiceModel

Error Detail 
============
Correlation ID:  fb24760f-c63a-4ba2-8694-81a0f626be62
Program:  Ice.Core.TaskBase.dll
Method:  .ctor
Line Number:  40
Column Number:  11

Client Stack Trace 
==================
   at Epicor.ServiceModel.Channels.ImplBase.CallWithCommunicationFailureRetry(String methodName, ProxyValuesIn valuesIn, ProxyValuesOut valuesOut, RestRpcValueSerializer serializer)
   at Epicor.ServiceModel.Channels.ImplBase.CallWithMultistepBpmHandling(String methodName, ProxyValuesIn valuesIn, ProxyValuesOut valuesOut, Boolean useSparseCopy)
   at Epicor.ServiceModel.Channels.ImplBase.Call(String methodName, ProxyValuesIn valuesIn, ProxyValuesOut valuesOut, Boolean useSparseCopy)
   at Ice.Proxy.BO.SsrsReportDesignImpl.UploadAndRenderDesign(Guid sysRowID, Byte[] compressedReportDefinitions, String outputFormat)
   at Ice.Adapters.SsrsReportDesignAdapter.UploadAndRenderDesign(Guid sysRowID, Byte[] compressedReportDefinitions, String outputFormat)
   at Ice.UI.App.SsrsReportDesign.Transaction.UploadAndRender()
   at Ice.UI.App.SsrsReportDesign.ReportDesignForm.BaseToolbarsManager_ToolClick(Object sender, ToolClickEventArgs e)
   at Infragistics.Win.UltraWinToolbars.ToolClickEventHandler.Invoke(Object sender, ToolClickEventArgs e)
   at Infragistics.Win.UltraWinToolbars.UltraToolbarsManager.OnToolClick(ToolClickEventArgs e)
   at Infragistics.Win.UltraWinToolbars.UltraToolbarsManager.FireEvent(ToolbarEventIds id, EventArgs e)
   at Infragistics.Win.UltraWinToolbars.ToolBase.OnToolClick()
   at Infragistics.Win.UltraWinToolbars.ToolMenuItem.OnClick()
   at Infragistics.Win.UltraWinToolbars.PopupMenuItemUIElement.DoClickProcessing(MouseEventArgs e)
   at Infragistics.Win.UltraWinToolbars.PopupMenuItemUIElement.OnMouseUp(MouseEventArgs e)
   at Infragistics.Win.TextUIElementBase.OnMouseUp(MouseEventArgs e)
   at Infragistics.Win.ControlUIElementBase.ProcessMouseUpHelper(Object sender, MouseEventArgs e)
   at Infragistics.Win.ControlUIElementBase.ProcessMouseUp(Object sender, MouseEventArgs e)
   at Infragistics.Win.Utilities.ProcessEvent(Control control, ProcessEvent eventToProcess, EventArgs e)
   at Infragistics.Win.UltraWinToolbars.PopupControlBase.OnMouseUp(MouseEventArgs e)
   at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
2 Likes

What is the server log exception?
Client stack is not very helpful actually.
We only know server method SsrsReportDesignImpl.UploadAndRenderDesign, but its details still need to be found on server.

1 Like

Thank you @GabeFranco and @aosemwengie1 for continuing to troubleshoot this and @Olga for listening.

2 Likes

If only epicor support was listening . . .