Multi-Company Direct Process Crashing AppServer

We copied PRD to TST which we do quite often, but this time when we run the Multi-Company Direct Process (NO Service Bus) just basic process it hangs and it crashes the AppServer for some reason – we didn’t change anything, no passwords I also verified all the service accounts – no permissions have changed… Absolutely nothing and I am receiving some weird Event Log Errors, which by googling haven’t found much help – I have no custom code – its just almost like one of those CLR Security Config Bugs.

See Screenshots. No changes to AD, we rebooted Server – Everytime I restart the appServer it gets stuck again eventually during running. I recycled IIS, I regened db, I did everything one would do – checked credentials etc…

@aidacra

Full Event Log

Log Name:      Application
Source:        ServiceModel Audit 4.0.0.0
Date:          3/29/2017 6:46:30 PM
Event ID:      4
Task Category: MessageAuthentication
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      TST-CM-RPT1.spartanmotors.com
Description:
Message authentication failed.
 Service: net.tcp://tst-cm-rpt1.spartanmotors.com/EpicorERP/Ice/Lib/RunTask.svc
 Action: Ice:Lib:RunTask/RunTaskSvcContract/RunSubTask
 ClientIdentity: SPARTANMOTORS\txs31; S-1-5-21-206286035-3648347409-1977274830-29373
 ActivityId: <null>
 FaultException: Invalid username or password.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="ServiceModel Audit 4.0.0.0" />
    <EventID Qualifiers="49158">4</EventID>
    <Level>2</Level>
    <Task>2</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-03-29T22:46:30.000000000Z" />
    <EventRecordID>75087</EventRecordID>
    <Channel>Application</Channel>
    <Computer>TST-CM-RPT1.spartanmotors.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>net.tcp://tst-cm-rpt1.spartanmotors.com/EpicorERP/Ice/Lib/RunTask.svc</Data>
    <Data>Ice:Lib:RunTask/RunTaskSvcContract/RunSubTask</Data>
    <Data>SPARTANMOTORS\txs31; S-1-5-21-206286035-3648347409-1977274830-29373</Data>
    <Data>&lt;null&gt;</Data>
    <Data>FaultException: Invalid username or password.</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        ASP.NET 4.0.30319.0
Date:          3/29/2017 6:46:30 PM
Event ID:      1325
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      TST-CM-RPT1.spartanmotors.com
Description:
An unhandled exception occurred and the process was terminated.

Application ID: DefaultDomain

Process ID: 5188

Exception: System.Runtime.Serialization.SerializationException

Message: Type 'System.ServiceModel.Channels.ReceivedFault' in Assembly 'System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' is not marked as serializable.

StackTrace:    at System.Runtime.Serialization.FormatterServices.InternalGetSerializableMembers(RuntimeType type)
   at System.Runtime.Serialization.FormatterServices.GetSerializableMembers(Type type, StreamingContext context)
   at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitMemberInfo()
   at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitSerialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjectInfoInit, IFormatterConverter converter, ObjectWriter objectWriter, SerializationBinder binder)
   at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Write(WriteObjectInfo objectInfo, NameInfo memberNameInfo, NameInfo typeNameInfo)
   at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Serialize(Object graph, Header[] inHeaders, __BinaryWriter serWriter, Boolean fCheck)
   at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph, Header[] headers, Boolean fCheck)
   at System.Runtime.Remoting.Channels.CrossAppDomainSerializer.SerializeObject(Object obj, MemoryStream stm)
   at System.AppDomain.Serialize(Object o)
   at System.AppDomain.MarshalObject(Object o)

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="ASP.NET 4.0.30319.0" />
    <EventID Qualifiers="49152">1325</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-03-29T22:46:30.000000000Z" />
    <EventRecordID>75088</EventRecordID>
    <Channel>Application</Channel>
    <Computer>TST-CM-RPT1.spartanmotors.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>An unhandled exception occurred and the process was terminated.

Application ID: DefaultDomain

Process ID: 5188

Exception: System.Runtime.Serialization.SerializationException

Message: Type 'System.ServiceModel.Channels.ReceivedFault' in Assembly 'System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' is not marked as serializable.

StackTrace:    at System.Runtime.Serialization.FormatterServices.InternalGetSerializableMembers(RuntimeType type)
   at System.Runtime.Serialization.FormatterServices.GetSerializableMembers(Type type, StreamingContext context)
   at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitMemberInfo()
   at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitSerialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjectInfoInit, IFormatterConverter converter, ObjectWriter objectWriter, SerializationBinder binder)
   at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Write(WriteObjectInfo objectInfo, NameInfo memberNameInfo, NameInfo typeNameInfo)
   at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Serialize(Object graph, Header[] inHeaders, __BinaryWriter serWriter, Boolean fCheck)
   at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph, Header[] headers, Boolean fCheck)
   at System.Runtime.Remoting.Channels.CrossAppDomainSerializer.SerializeObject(Object obj, MemoryStream stm)
   at System.AppDomain.Serialize(Object o)
   at System.AppDomain.MarshalObject(Object o)
</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        .NET Runtime
Date:          3/29/2017 6:46:30 PM
Event ID:      1026
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      TST-CM-RPT1.spartanmotors.com
Description:
Application: w3wp.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ServiceModel.FaultException

Exception Info: System.ServiceModel.Security.MessageSecurityException

Server stack trace: 
   at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityDuplexChannel.ProcessMessage(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityDuplexChannel.TryReceive(TimeSpan timeout, Message& message)
   at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Messaging.IMessage)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(System.Runtime.Remoting.Proxies.MessageData ByRef, Int32)
   at Ice.Contracts.RunTaskSvcContract.RunSubTask(System.String, System.Collections.Generic.List`1<Epicor.Hosting.TaskCaller.SubTaskParameter>)
   at Ice.Proxy.Lib.RunTaskImpl.RunSubTask(System.String, System.Collections.Generic.List`1<Epicor.Hosting.TaskCaller.SubTaskParameter>)

Exception Info: System.UnauthorizedAccessException
   at Epicor.ServiceModel.Channels.ImplBase`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].ShouldRethrowNonRetryableException(System.Exception, System.Data.DataSet[])
   at Ice.Proxy.Lib.RunTaskImpl.RunSubTask(System.String, System.Collections.Generic.List`1<Epicor.Hosting.TaskCaller.SubTaskParameter>)
   at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid3[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Runtime.CompilerServices.CallSite, System.__Canon, System.__Canon, System.__Canon)
   at Ice.Hosting.SubTaskLauncher.CallTaskInternal(System.String, System.String, System.String, System.Collections.Generic.List`1<Epicor.Hosting.TaskCaller.SubTaskParameter>, System.TimeSpan)
   at Erp.Internal.SI.SIM130.launchSubProcess(System.String, System.String, System.Collections.Generic.List`1<Epicor.Hosting.TaskCaller.SubTaskParameter>, System.String, Ice.Hosting.SubTaskLauncher)
   at Erp.Internal.SI.SIM130+<>c__DisplayClass45_1.<RunProcess>b__2()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart()


Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name=".NET Runtime" />
    <EventID Qualifiers="0">1026</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-03-29T22:46:30.000000000Z" />
    <EventRecordID>75089</EventRecordID>
    <Channel>Application</Channel>
    <Computer>TST-CM-RPT1.spartanmotors.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Application: w3wp.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ServiceModel.FaultException

Exception Info: System.ServiceModel.Security.MessageSecurityException

Server stack trace: 
   at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityDuplexChannel.ProcessMessage(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityDuplexChannel.TryReceive(TimeSpan timeout, Message&amp; message)
   at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Messaging.IMessage)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(System.Runtime.Remoting.Proxies.MessageData ByRef, Int32)
   at Ice.Contracts.RunTaskSvcContract.RunSubTask(System.String, System.Collections.Generic.List`1&lt;Epicor.Hosting.TaskCaller.SubTaskParameter&gt;)
   at Ice.Proxy.Lib.RunTaskImpl.RunSubTask(System.String, System.Collections.Generic.List`1&lt;Epicor.Hosting.TaskCaller.SubTaskParameter&gt;)

Exception Info: System.UnauthorizedAccessException
   at Epicor.ServiceModel.Channels.ImplBase`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].ShouldRethrowNonRetryableException(System.Exception, System.Data.DataSet[])
   at Ice.Proxy.Lib.RunTaskImpl.RunSubTask(System.String, System.Collections.Generic.List`1&lt;Epicor.Hosting.TaskCaller.SubTaskParameter&gt;)
   at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid3[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Runtime.CompilerServices.CallSite, System.__Canon, System.__Canon, System.__Canon)
   at Ice.Hosting.SubTaskLauncher.CallTaskInternal(System.String, System.String, System.String, System.Collections.Generic.List`1&lt;Epicor.Hosting.TaskCaller.SubTaskParameter&gt;, System.TimeSpan)
   at Erp.Internal.SI.SIM130.launchSubProcess(System.String, System.String, System.Collections.Generic.List`1&lt;Epicor.Hosting.TaskCaller.SubTaskParameter&gt;, System.String, Ice.Hosting.SubTaskLauncher)
   at Erp.Internal.SI.SIM130+&lt;&gt;c__DisplayClass45_1.&lt;RunProcess&gt;b__2()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart()

</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Application Error
Date:          3/29/2017 6:46:30 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      TST-CM-RPT1.spartanmotors.com
Description:
Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96
Faulting module name: KERNELBASE.dll, version: 6.3.9600.18340, time stamp: 0x57366075
Exception code: 0xe0434352
Fault offset: 0x0000000000008a5c
Faulting process id: 0x1444
Faulting application start time: 0x01d2a8de30541d7e
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 8c91bb21-14d1-11e7-80f6-00505686f220
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-03-29T22:46:30.000000000Z" />
    <EventRecordID>75090</EventRecordID>
    <Channel>Application</Channel>
    <Computer>TST-CM-RPT1.spartanmotors.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>w3wp.exe</Data>
    <Data>8.5.9600.16384</Data>
    <Data>5215df96</Data>
    <Data>KERNELBASE.dll</Data>
    <Data>6.3.9600.18340</Data>
    <Data>57366075</Data>
    <Data>e0434352</Data>
    <Data>0000000000008a5c</Data>
    <Data>1444</Data>
    <Data>01d2a8de30541d7e</Data>
    <Data>c:\windows\system32\inetsrv\w3wp.exe</Data>
    <Data>C:\Windows\system32\KERNELBASE.dll</Data>
    <Data>8c91bb21-14d1-11e7-80f6-00505686f220</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>

Did you try removing the mc-direct process from the schedule and re-creating it?

Yes =) I’ll do another clean db copy next… Its starts 4-5 and then the 6th company it clogs up… Complaining about credentials – yet all credentials are ok, and Direct Server Process doesnt really use any credentials so not sure what its complaining about =)

Memory is okay on server.


Task Agent Log

investigate if you need to increase the receive and send timeouts in your web.config.
Error Detail:
The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '1.00:00:00'.
Stack Trace:

Server stack trace: 
   at System.ServiceModel.Channels.StreamConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
   at System.ServiceModel.Channels.SessionConnectionReader.Receive(TimeSpan timeout)
   at System.ServiceModel.Channels.SynchronizedMessageSource.Receive(TimeSpan timeout)
   at System.ServiceModel.Channels.TransportDuplexSessionChannel.Receive(TimeSpan timeout)
   at System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceive(TimeSpan timeout, Message& message)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityDuplexChannel.TryReceive(TimeSpan timeout, Message& message)
   at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Ice.Contracts.RunTaskSvcContract.RunTask(Int64 ipTaskNum)
   at Ice.Proxy.Lib.RunTaskImpl.RunTask(Int64 ipTaskNum) in C:\_Releases\ICE\3.1.500.7\Source\Shared\Contracts\Lib\RunTask\RunTaskImpl.cs:line 118
   at Ice.TaskAgentCore.TaskAgent.<>c__DisplayClass31_4.<CallRunTaskService>b__8(RunTaskImpl impl)
   at Ice.TaskAgentCore.ImplCaller.RunTaskImplCaller`1.<>c__DisplayClass12_0.<Call>b__0(TImpl impl)
   at Ice.TaskAgentCore.ImplCaller.RunTaskImplCaller`1.Call[TResult](Func`2 doWork, ExceptionBehavior communicationExceptionBehavior, ExceptionBehavior timeoutExceptionBehavior)
   at Ice.TaskAgentCore.ImplCaller.RunTaskImplCaller`1.Call(Action`1 doWork, ExceptionBehavior communicationExceptionBehavior, ExceptionBehavior timeoutExceptionBehavior)
   at Ice.TaskAgentCore.TaskAgent.<>c__DisplayClass31_0.<CallRunTaskService>b__6(Object obj)
Inner Exception:The read operation failed, see inner exception.
   at System.Net.Security.NegotiateStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.NegotiateStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.ServiceModel.Channels.StreamConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)







EpicorERP: An error occurred trying to run tasks for agent: Task Description: Multi-Company Direct Server Process User: manager

Error Detail:
The message could not be dispatched because the service at the endpoint address 'net.tcp://tst-cm-rpt1.spartanmotors.com/EpicorERP/Ice/BO/SysTask.svc' is unavailable for the protocol of the address.
Stack Trace:
Server stack trace: 
   at System.ServiceModel.Channels.ConnectionUpgradeHelper.DecodeFramingFault(ClientFramingDecoder decoder, IConnection connection, Uri via, String contentType, TimeoutHelper& timeoutHelper)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.SendPreamble(IConnection connection, ArraySegment`1 preamble, TimeoutHelper& timeoutHelper)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.DuplexConnectionPoolHelper.AcceptPooledConnection(IConnection connection, TimeoutHelper& timeoutHelper)
   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.ClientSecurityChannel`1.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open()

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at System.ServiceModel.ICommunicationObject.Open()
   at Epicor.ServiceModel.Channels.ChannelEntry`1.CreateNewChannel() in C:\_Releases\ICE\3.1.500.7\Source\Shared\Framework\Epicor.ServiceModel\Channels\ChannelEntry.cs:line 301
   at Epicor.ServiceModel.Channels.ImplBase`1.GetChannel() in C:\_Releases\ICE\3.1.500.7\Source\Shared\Framework\Epicor.ServiceModel\Channels\ImplBase.cs:line 519
   at Epicor.ServiceModel.Channels.ImplBase`1.HandleContractBeforeCall() in C:\_Releases\ICE\3.1.500.7\Source\Shared\Framework\Epicor.ServiceModel\Channels\ImplBase.cs:line 631
   at Ice.Proxy.BO.SysTaskImpl.GetNewSysTaskParam(SysTaskDataSet ds, Int64 sysTaskNum) in C:\_Releases\ICE\3.1.500.7\Source\Shared\Contracts\BO\SysTask\SysTaskImpl.cs:line 306
   at Ice.TaskAgentCore.TaskAgent.<>c__DisplayClass28_3.<CreateTaskInstance>b__7(SysTaskImpl impl)
   at Ice.TaskAgentCore.ImplCaller.ImplCaller`1.<>c__DisplayClass4_0.<Call>b__0(TImpl impl)
   at Ice.TaskAgentCore.ImplCaller.ImplCaller`1.Call[TResult](Func`2 doWork, ExceptionBehavior communicationExceptionBehavior, ExceptionBehavior timeoutExceptionBehavior)
   at Ice.TaskAgentCore.ImplCaller.ImplCaller`1.Call(Action`1 doWork, ExceptionBehavior communicationExceptionBehavior, ExceptionBehavior timeoutExceptionBehavior)
   at Ice.TaskAgentCore.TaskAgent.CreateTaskInstance(ImplCaller`1 sysTaskCaller, SysAgentDataSet sysAgentData, SysAgentSchedRow agentSched)

The odd part is that I have the exact same DB in other Environments, Multi-Company stuff running fine =) – just cant find the needle in the haystack.

Multi Company Log - seems to always just be starting:

I might have fixed it; but its very odd to me at least, not sure of the cause.

I made a fresh backup of PRD and copied it to TST and it resolved the issue… Weird, do you know if there is a Risk if you take a db backup while a multi-company direct process is in progress (probably a interim table carrying embedded credentials with it)? I might have hit a snapshot of that kind – probably had items in-transit which moved to the TST company.

From your test database, please provide a screenshot of your System Agent record

and from your System Agent, click on Actions > Edit Task Agent rules and provide a screenshot of that screen as well.

According to Erik Johnson from Epicor:
> “If you are restoring a DB across servers, you can also be confusing the Task Agent processes (which assume they have exclusive access to the data targeted to a particular app server).”

We could have simply grabbed a Database snapshot a second to soon while it did some processing and it embedded the PRD credentials – simple fix would be in the Task Agent in TST change the Print password to whatever PRD has and it would have fixed it – or – even better, grab another database copy.

Atleast, I have some closure now =) to the needle in the haystack, something I havent encountered yet.