Hey I’ve got an odd one today. We have a set of reports that are scheduled to run every Monday morning at 5:00am. For about a year, they have kicked off just fine. This morning, they started resending over and over again. We had to restart the task agent because they just kept coming.
This afternoon, we were working on re-enabling the reports and the one set to run on Monday at 5:00am just started firing off reports again like crazy non-stop. Any idea where to look or what might be triggering this crazy thing?
@dr_dan Do you have any app servers that have been restored recently or dead ones that got restarted?
There was a post about the dns endpoint not getting cleared that caused an issue like this.
As a matter of fact, we just moved our Pilot and Test environments to a new server the past week. I remember it was fairly tedious to get all the references updated so I could use the new environments. Do you think that could have something to do with it?
I’m not sure what I’m looking for exactly… I have 5 rows in BPActionQueue, each with a 3 in the AttemptCounter column. I get 5 apparently related rows in the BPActionQueueData table, and then in MailQueue I have about 120 rows each with a 3 in the AttemptCounter column. Each appears to be a notification related to RMA being received (which is not good that they didn’t send).
Where do you go about putting that? Does it run automatically when you restore Pilot or do you have to go to SQL Management Studio or something to that effect and run that?
You’re all going to laugh at this… but I figured out what the issue was. It turns out, I had setup a BPM to run when the Schedule updated the Next Run On field. It was a trick I read somewhere in here to automate a BPM. Effectively, it’s a utility to complete tasks and close old quotes. It was working like a charm for the longest time…
Once I remembered about this BPM, I suspected it might be the culprit. So I moved all of the reports to a new schedule but left the old one still enabled. Sure enough, the reports ran just fine. Interestingly, though, I noticed that the original schedule appeared not to have run… as in the Next Run On date was in the past stuck still at 5:00am earlier in the morning. So I reviewed the Event Viewer and sure enough there was an error that started at 5:00am and had been occurring every 2 minutes or so since.
I guess something in one of the quotes decided that it needed a valid Assigned To value and it was causing the BPM to error. I’m not sure why, but the error wasn’t showing up in Event viewer when the reports were tied in with the schedule (or I somehow overlooked it). But this was causing the Next Run On not to update because every time it tried, it would error in the BPM. Something must tell it to keep trying and consequently, every 2 minutes or so it was trying to re-execute the list of tasks on that schedule - all my scheduled reports. I guess it just kept sending them to the task agent and task agent just kept on doing it’s job.
So that’s the long and short of it. I’ll try to find that original thread that I used to learn how to automate the utility and put a little nugget of caution in there. I guess I need to find a way to handle the errors so that it still allows the transaction to update in the SysAgentSched table.
