SQL Server - Transaction is in doubt error

ERP 10
1

Hello E10help,

We have been having issues with our Epicor server needing to rebooted in the middle of the night (All users get an “Unsecured fault” error). I’ve actually found that it’s an issue with the SQL Service/s and not the server itself - usually if I restart the services in a correct manner then it can recover without a reboot.

I’ve been looking into the logging and found that there’s usually a “Transaction is in doubt (TransactionInDoubtException)” error before seeing the issue with SQL service/s.

Here’s the error directly from one of my logs:

<Exception><![CDATA[
The transaction is in doubt. (TransactionInDoubtException)
   at System.Transactions.TransactionStateInDoubt.EndCommit(InternalTransaction tx)
   at System.Transactions.CommittableTransaction.Commit()
   at System.Transactions.TransactionScope.InternalDispose()
   at System.Transactions.TransactionScope.Dispose()
   at Ice.TablesetBound`3.InnerUpdate(IceDataContext dataContext, TFullTableset tableset) in c:\_Releases\ICE\3.1.400.13\source\Framework\Epicor.Ice\Services\TablesetBound.cs:line 816
   at Ice.Services.BO.ReportMonitorSvc.Update(ReportMonitorTableset& ds) in c:\_Releases\ICE\3.1.400.0\source\Server\Services\BO\ReportMonitor\ReportMonitor.Designer.cs:line 623
   at Ice.Services.BO.ReportMonitorSvcFacade.Update(ReportMonitorTableset& ds) in c:\_Releases\ICE\3.1.400.0\source\Server\Services\BO\ReportMonitor\ReportMonitorSvcFacade.cs:line 190
   at SyncInvokeUpdate(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at Epicor.Hosting.OperationBoundInvoker.InnerInvoke(Object instance, Func`2 func) in c:\_Releases\ICE\3.1.400.13\source\Framework\Epicor.System\Hosting\OperationBoundInvoker.cs:line 59
   at Epicor.Hosting.OperationBoundInvoker.Invoke(Object instance, Func`2 func) in c:\_Releases\ICE\3.1.400.13\source\Framework\Epicor.System\Hosting\OperationBoundInvoker.cs:line 47
   at Epicor.Hosting.Wcf.EpiOperationInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs) in c:\_Releases\ICE\3.1.400.13\source\Framework\Epicor.System\Hosting\Wcf\EpiOperationInvoker.cs:line 23
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at Syste
m.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
-------- Inner exception --------->>>
Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. (SqlException)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)
   at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
   at System.Data.SqlClient.SqlDelegatedTransaction.SinglePhaseCommit(SinglePhaseEnlistment enlistment)
-------- Inner exception --------->>>
The wait operation timed out (Win32Exception)

]]></Exception>

Almost immediately after that error, I see a bunch of these errors then -

 <Exception><![CDATA[
A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable) (SqlException)
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Ice.Services.ContextFactory.SetLockTimeout(DbConnection dbConnection) in c:\_Releases\ICE\3.1.400.13\source\Framework\Epicor.Ice\Services\ContextFactory.cs:line 160
   at Ice.Services.ContextFactory.CreateContext() in c:\_Releases\ICE\3.1.400.13\source\Framework\Epicor.Ice\Services\ContextFactory.cs:line 67
   at Ice.Security.UserLogOnStatusManager.LogonSucceeded(String userID) in c:\_Releases\ICE\3.1.400.13\source\Framework\Epicor.Ice\Security\UserLogOnStatusManager.cs:line 195
   at Ice.Manager.Security.Authenticate(SysUserFile userRow, String password, String& reason) in c:\_Releases\ICE\3.1.400.13\source\Framework\Epicor.Ice\Manager\Security.cs:line 136
   at Ice.Security.UsernameValidator.Validate(String userName, String password) in c:\_Releases\ICE\3.1.400.13\source\Framework\Epicor.Ice\Security\UsernameValidator.cs:line 91
]]></Exception>

If anyone has any ideas as to what this error means and how I can start to troubleshoot why it’s occurring that would be much appreciated.

2

FWIW, I see this related to your Report Monitor

3

Usually related not to Epicor but common SQL Error which is caused because of your network, infrastructure issues.

For example if you use CommVault for VMWare to take a snapshot and it takes a snapshot while you have lets say MRP Running (tends to freeze SQL or APP Server for a louse 1-3 seconds) it may cause “The transaction is in doubt.” error. #Experience I’ve seen if often with Backups, Anti-Virus, Network Issues, SonicWall and other things causing it.

Just make sure your connection is smooth and you have no maintenance tasks running during the same time you have heavy SQL/Epicor activity such as MRP

Also if you look at Event Viewer on SQL or App you might see more “Connection state is broken events” during the same period.

1 Like