We are having issues with scheduled tasks getting stuck as an active task.
I want to know if it’s possible to run a function on a schedule of every five or so minutes that checks if the SysTask.LastActivityOn column is greater than one hour. If the activity was last acted on an hour ago I want it canceled.
This is what an observability tool does. It monitors from the outside. You can see the potential problem with scheduling a function that checks for hanging scheduled functions as they could hang too.
There are REST endpoints for the Task Agent. They can be called from PowerShell running in the Windows Task Scheduler. The trick is to know how long a function should run before killing it. Some systems look at a historical average and use that for comparison as an example. Maybe a notification would be wise and then someone can make the call whether or not to cancel the processes.
long taskToKill = 0;
foreach (var baqResults_iterator2 in TasksTable_CC.SysTaskList.Where(r => r != null))
{
taskToKill = baqResults_iterator2.SysTaskNum;
this.CallService<Ice.Contracts.SysTaskSvcContract>(ewb => ewb.DeleteByID(taskToKill));
}
I figure I can put a task in like MRP and see if it gets stuck long enough to kick this guy off, but would like to have something more reliable to test it with.
All testing is done in our pilot system so no worries on breaking anything