Finally got someone experiencing the message on a very very simple BAQ report:
Program Ice.Services.Lib.RunTask when executing task 24567 raised an unexpected exception with the following message: RunTask:
Ice.Core.SsrsReporting.SsrsCaller.SsrsException: An error occurred within the report server database. This may be due to a connection failure, timeout or low disk condition within the database.
Finally received an update on the stock status report which still won’t run.
The issue is coming from our SSRS Server; the SSRS instance is experiencing fairly significant concurrency issues, with deadlocks and blocking occurring quite frequently within the SSRS Service and within the instance that hosts the temporary databases used during the rendering process.
Unfortunately, given the nature of deadlocks, the issue presents in several different ways across clients, which makes it more difficult to identify, since each client is experiencing a different side effect of the deadlocks and blocking. There isn't just one type occurring; several different types are happening.
We are currently taking a multiple-prong approach to address this issue. We have raised a priority ticket with Microsoft and have already had several calls with their engineers as they attempt to troubleshoot the issue on their end. The other prong is me, I am working on analyzing the deadlock and blocking events using deadlock graphs and the blocked processes report.
Tell me again how everything will be magical once all customers are migrated to the cloud?
What could go wrong?
So miraculously the stock status report now runs after whatever happened last night . . . without knowing the details its hard to have confidence the solution will stick but I am going to enjoy the functionality while it lasts! Curious to see what the situation will be with the other errors with load on the system tomorrow.
Nope, back to failing today.
Program Ice.Services.Lib.RunTask when executing task 1514106 raised an unexpected exception with the following message: RunTask:
System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while executing the command definition. See the inner exception for details.
—> Microsoft.Data.SqlClient.SqlException (0x80131904): A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)
…yup.
Program Ice.Services.Lib.RunTask when executing task 2812025 raised an unexpected exception with the following message: RunTask:
Ice.Core.SsrsReporting.SsrsCaller.SsrsException: The SSRS server returned the status code 500 (InternalServerError) with the following error text:
The item ‘/REDACTED-rpt/reports/CustomReports/Tenants/REDACTED/CTS-BOL’ cannot be found. —> Microsoft.ReportingServices.Diagnostics.Utilities.ItemNotFoundException: The item ‘/REDACTED-rpt/reports/CustomReports/Tenants/REDACTED/CTS-BOL’ cannot be found.
Completely random, same BOL printed OK since.
So I just checked the KB article KB0144074 on this “major incident” which appears to have been updated recently. Now it says this:
Since we are flex 2, live is still on 2025.1.12 and this statement seems to imply the issue is fixed at 2025.1.15+. Anybody on .15 or live on 2025.2 who can confirm the issue is actually resolved this time?
Watching closely.
I think that there are multiple root causes at play here to trigger the set of ssrs errors, and that was one specific case that got addressed.
On 2025.1.15 and have been experiencing the random ssrs errors randomly as “usual”
The KB article does not list any other unresolved PRBs or ERPS for this specific issue and it literally says “resolved” so . . . I guess its just wrong. Sigh.
Definitely still getting the low disk space/etc error.
So when is the maintenance? Feb 22nd is a Sunday not a Thursday.
They didn’t specify the year so by default this will occur 22-Feb-2029.
Well, i’m glad they are giving us more of a notice period, now.
“Emergency”?
Whatever they do I hope it makes reporting better.
Imagine a future in which specific PRBs are included in the notice so that we know what changes are being addressed and are aware what testing to conduct afterwards…
Why, so we can be even more disappointed?
