Yes, we’re on our second escalation on this case.
Sounds like we’re in the same boat as you, Alisa. 
Today, I’m going to create schedules for each individual person/group and schedule each of their different weekly reports in their own schedule. Then I’ll set each person’s schedule 1 minute apart. If what Support says is true, we shouldn’t see nearly as many errors. Wish me luck. 
We haven’t had any issues since I figured out the weird workaround of uploading via classic report style, so I have no idea if the hotfix actually did anything for us.
@GabeFranco brilliantly figured how to reproduce this on our system so I am hoping we will finally hear something other than crickets from support. I am curious if you can repro too? The problem occurs if two people try to run the same report at the same time. It doesn’t have to be the same record, they just have to click to submit the report at the same time. In our case, it can even be in two separate companies. For example, if two people click to print a purchase order at the same moment, one goes through the other fails. We are most frequently seeing the issue on purchase order, production detail report, and customer statement, but occasionally happening on others as well.
I am also deeply curious if anybody who is on prem on 2025.1.x can reproduce this issue or if its truly a cloud-only issue down to the specifics of the cloud infrastructure. Anybody?
Thank you so much for staying with this. I remember I had to do the same thing here for months, along with many other members in this forum: Order Entry screen showing {{strings.XXXX}} in stead of field names - #167 by utaylor.
Thanks again for continuing to try and find the root cause.
Oddly enough @aosemwengie1, this problem was also due to two people opening the same kinetic form at the same time.
Yes, we have always been able to reproduce our issue in the Live environment by running a report more than once at the same time. The easiest way for us to test this was to put the same report (but different params, different data) on the same schedule and run a group of them together. Around 50-75 % will fail. I’ve seen it happen on a variety of standard reports (Shop Load, Sales Gross Margin, and others) with interactive users as well.
We have not been able to replicate this issue in any of our non-live environments though, which seems strange to me… unless it’s related to the lack of transactions happening there.
I don’t understand how they aren’t coming up with a fix for this. Two people can’t run a report at once isn’t a reasonable limitation.
We can’t repro in non prod either. Maybe the infrastructure design is different there than it is in live, I don’t know.
According to support its just a logging configuration change so its not going to fix anything.
With the logging config change, it looks like the error messages are a bit more verbose, showing inner exceptions. It also seems like the frequency increased after the change. We’ve provided support the detail from half a dozen new errors today.
The SSRS server returned the status code 500 (InternalServerError) with the following error text:
An error occurred during rendering of the report. ---> Microsoft.ReportingServices.ReportProcessing.UnhandledReportRenderingException: An error occurred during rendering of the report. ---> Microsoft.ReportingServices.OnDemandReportRendering.ReportRenderingException: An error occurred during rendering of the report. ---> System.NullReferenceException: Object reference not set to an instance of an object.
at Ice.Core.SsrsReporting.SsrsCaller.SoapCaller.Call[TContent,TResponse,TSoapHeader](String actionName, TContent content, TSoapHeader soapHeader) in F:\_Releases\12.0.100\Current\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\SoapCaller.cs:line 47
at Ice.Core.SsrsReporting.SsrsCaller.ReportExecutionService.Render(String format, String deviceInfo, String& extension, String& mimeType, String& encoding, Warning[]& warnings, String[]& streamIds) in F:\_Releases\12.0.100\Current\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsCaller\ReportExecutionService.cs:line 75
at Ice.Core.SsrsReporting.SsrsRendererBase.TraceReportRendered(String printProgram, String ssrsRenderFormat, Func`1 timedAction) in F:\_Releases\12.0.100\Current\Source\Server\Internal\Lib\TaskLib\SsrsReporting\SsrsRendererBase.cs:line 81
Program Ice.Services.Lib.RunTask when executing task 6242273 raised an unexpected exception with the following message: RunTask:
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Text.StringBuilder.ExpandByABlock(Int32 minBlockCharCount)
at System.Text.StringBuilder.AppendWithExpansion(Char& value, Int32 valueCount)
at System.Text.StringBuilder.Append(String value)
at Ice.Core.RptBase.SqlTableBuilder.BuildTable(String tableName, IEnumerable`1 columns, SqlObjectsCreated sqlObjectsCreated, Boolean accessedDirectly, Boolean dropExistingTables)
System.Net.Sockets.SocketException (11001): No such host is known.
at System.Net.NameResolutionPal.ProcessResult(SocketError errorCode, GetAddrInfoExContext* context)
at System.Net.NameResolutionPal.GetAddressInfoExCallback(Int32 error, Int32 bytes, NativeOverlapped* overlapped)
--- End of stack trace from previous location ---
at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.SocketsHttpHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendCoreAsync>g__Core|5_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendCoreAsync>g__Core|5_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Ice.Core.SsrsReporting.SsrsCaller.RestReportingService.GetItemDefinition(String itemPath)
WHat is the saying, its darkest before the dawn? Or its darkest 8 weeks after the problem was discovered and 4 weeks after the hotfix failed to fix the problem? I always get those mixed up.
So is it just they don’t feel this is an issue that’s urgent? I have to imagine in a very high transactional environment this would be wicked. Please, I do not want to start a dumpsterfire or start crap, but I am asking, who knows about this and why isn’t it being prioritized?
Ditto, I am on-prem so this “doesn’t affect me,” but – not to be too sentimental – we are all in this together. I do not see how this can go on for so long.
Reminds me of this - I can’t believe that was 4 years ago. That one affected all users, cloud or on-prem.
I can’t even edit an RDL in Pilot now with the download/modify/upload method.
No errors but the RDL doesn’t update to my edited version.
Hmmm. I am really hoping all the reports converted fine and that I don’t need to mess with them then. I think @Hally said that he was using solution workbench, does that work?
Yep, they were supposed to have this fixed in .14 but its not fixed. I put in a ticket explaining that it isn’t fixed and they asked me to provide a screen recording of how to modify a custom report.
I’d just send the help article.
Thanks for clarifying before I went insane.
Going to do an on prem install on the weekend…