We had downtime for one minute twice yesterday after months of zero unscheduled downtime. This lasted long enough to cause all of our users who run off a terminal server (Remote Desktop) environment to lose their connection. We run across multiple network locations and other applications continued to run so we are confident there was no specific WAN related issue to connect to our Data Center to get to the servers. What we have narrowed down is the following error on the APP server a few seconds before the downtime:
Query failed to run
System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while executing the command definition. See the inner exception for details. —> System.Data.SqlClient.SqlException: A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error) at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync() at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket() at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer() at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.Entity.Infrastructure.Interception.InternalDispatcher1.Dispatch[TTarget,TInterceptionContext,TResult](TTarget target, Func3 operation, TInterceptionContext interceptionContext, Action3 executing, Action3 executed)
at System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(DbCommand command, DbCommandInterceptionContext interceptionContext)
at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
— End of inner exception stack trace —
at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
at System.Data.Entity.Core.Objects.Internal.ObjectQueryExecutionPlan.Execute[TResultType](ObjectContext context, ObjectParameterCollection parameterValues)
at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess) at System.Data.Entity.Core.Objects.ObjectQuery1.<>c__DisplayClass7.b__5()
at System.Data.Entity.Core.Objects.ObjectQuery1.GetResults(Nullable1 forMergeOption)
at System.Data.Entity.Core.Objects.ObjectQuery1.<System.Collections.Generic.IEnumerable<T>.GetEnumerator>b__0() at System.Data.Entity.Internal.LazyEnumerator1.MoveNext()
at System.Linq.Enumerable.Single[TSource](IEnumerable1 source) at System.Data.Entity.Core.Objects.CompiledQuery.ExecuteQuery[TResult](ObjectContext context, Object[] parameterValues) at System.Data.Entity.Core.Objects.CompiledQuery.Invoke[TArg0,TArg1,TArg2,TArg3,TResult](TArg0 arg0, TArg1 arg1, TArg2 arg2, TArg3 arg3) at Epicor.Data.DBExpressionCompiler.<>c__DisplayClass35_15.b__3(TContext c, Func5 query) in C:\_Releases\ICE\ICE3.2.300.16\Source\Framework\Epicor.System\Data\DBExpressionCompiler.Generated.cs:line 1146 at Epicor.Data.DBExpressionCompiler.GetResult[TContext,TQuery,TResult](Func3 executeQuery, Cache cacheSetting, TContext dataContext, TQuery query) in C:_Releases\ICE\ICE3.2.300.16\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs:line 445
at Epicor.Data.DBExpressionCompiler.InvokeSingle[TContext,TQuery,TResult](Expression expression, Cache currentCacheSetting, Boolean cacheQuery, TContext dataContext, Func2 getDataCacheKey, Func2 compileQuery, Func`3 executeQuery) in C:_Releases\ICE\ICE3.2.300.16\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs:line 301
(ctx, sysTaskNum_ex, paramName_ex, paramCharacter_ex) => ctx.SysTaskParam.Where(row => (((row.SysTaskNum == sysTaskNum_ex) AndAlso (row.ParamName == paramName_ex)) AndAlso (row.ParamCharacter == paramCharacter_ex))).Any()