Error when running reports in Test Environmentt

Daniel_White · March 2, 2026, 12:06pm

I am getting the following error while running reports in one of our test environments which i have just migrated Live to -

Program Ice.Services.Lib.RunTask when executing task 13391 raised an unexpected exception with the following message: RunTask:
System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while executing the command definition. See the inner exception for details.
 ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid column name 'CustXRefPartNum_c'.
Invalid column name 'CustXRefCustID_c'.
Invalid column name 'Customer_c'.
   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, SqlCommand command, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean isAsync, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String method)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1.Dispatch[TTarget,TInterceptionContext,TResult](TTarget target, Func`3 operation, TInterceptionContext interceptionContext, Action`3 executing, Action`3 executed)
   at System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(DbCommand command, DbCommandInterceptionContext interceptionContext)
   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
ClientConnectionId:b41872f1-d0be-4590-aa19-65f8a6576d03
Error Number:207,State:1,Class:16
   --- End of inner exception stack trace ---
   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
   at System.Data.Entity.Core.Objects.Internal.ObjectQueryExecutionPlan.Execute[TResultType](ObjectContext context, ObjectParameterCollection parameterValues)
   at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass41_0.<GetResults>b__0()
   at System.Data.Entity.Core.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)
   at System.Data.Entity.Internal.LazyEnumerator`1.MoveNext()
   at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
   at System.Data.Entity.Core.Objects.CompiledQuery.ExecuteQuery[TResult](ObjectContext context, Object[] parameterValues)
   at Epicor.Data.DBExpressionCompiler.GetResult[TDataContext,TQuery,TResult](Func`3 executeQuery, Cache cacheSetting, TDataContext dataContext, TQuery query) in C:\_releases\ICE\ICE5.1.100.6\Source\Server\Framework\Epicor.System\Data\EntityFramework\DBExpressionCompiler.cs:line 441
   at Epicor.Data.DBExpressionCompiler.InvokeSingle[TDataContext,TQuery,TResult](Operation operation, Expression expression, Cache currentCacheSetting, Boolean cacheQuery, TDataContext dataContext, String callingMethod, Func`2 getDataCacheKey, Func`2 compileQuery, Func`3 executeQuery)
   at Epicor.Data.DBExpressionCompiler.<>c__DisplayClass55_0`4.<Compile>b__0(TDataContext dataContext, TArg1 arg1, TArg2 arg2)
   at Erp.Internal.PM.POForm.FindFirstPart(String company, String partNum) in C:\_releases\ERP\ERP12.1.100.0\Source\Server\Internal\PM\POForm\POForm.Queries.cs:line 242
   at Erp.Internal.PM.POForm.doPODetailCalculations() in C:\_releases\ERP\ERP12.1.100.0\Source\Server\Internal\PM\POForm\POForm.cs:line 631
   at Erp.Internal.PM.POForm.RunProcess(Int64 instanceTaskNum, String outputFileName) in C:\_releases\ERP\ERP12.1.100.0\Source\Server\Internal\PM\POForm\POForm.cs:line 410
   at Ice.Core.TaskBase`1.StartProcess(Int64 instanceTaskNum, String outputFileName) in C:\_releases\ICE\ICE5.1.100.6\Source\Server\Internal\Lib\TaskLib\TaskBase\TaskBase.cs:line 53
   at Ice.Hosting.TaskCaller.InnerExecuteTask(IceDataContext newContext) in C:\_releases\ICE\ICE5.1.100.6\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 114
   at Ice.Hosting.TaskCaller.ExecuteTask() in C:\_releases\ICE\ICE5.1.100.6\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 62
   at Ice.Hosting.TaskCaller.ExecuteTask(IceDataContext dataContext, Boolean suppressTransaction) in C:\_releases\ICE\ICE5.1.100.6\Source\Server\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 42
   at Ice.Services.Lib.RunTaskSvc.<>c__DisplayClass19_2.<InnerRunTask>b__1() in C:\_releases\ICE\ICE5.1.100.0\Source\Server\Services\Lib\RunTask\RunTask.cs:line 352
   at Ice.Lib.RunTask.BpmFriendlyTaskLauncher.Run(String sessionIdPrefix, IceContext db, Action taskRunner) in C:\_releases\ICE\ICE5.1.100.0\Source\Server\Services\Lib\RunTask\BpmFriendlyTaskLauncher.cs:line 57
   at Ice.Services.Lib.RunTaskSvc.InnerRunTask(Int64 ipTaskNum, Boolean suppressTransaction) in C:\_releases\ICE\ICE5.1.100.0\Source\Server\Services\Lib\RunTask\RunTask.cs:line 349

It seems to be moaning about 3 custom fields but those fields do not exist in any of our environments or reports

Justas · March 2, 2026, 7:01pm

have you checked your selected fields in ssrs dataset query expression?

DaveOlender · March 2, 2026, 7:43pm

Double check your UD Tables to make sure they are in Sync. If not run Data Model Regen? Just a guess.

Daniel_White · March 3, 2026, 8:15am

None of the fields exist in our DB and never have done. It is happening on out of the box reports as well.

Daniel_White · March 3, 2026, 8:16am

Yeah, my report guy checked the PO reports as they were the first i tested and the fields above do not appear in the report query

Hally · March 3, 2026, 8:59am

So just to clarify you restored your live to test?
Did you do a data model regeneration after the restore?

If not do that then recycle your app pool.

Daniel_White · March 3, 2026, 9:07am

Hi Simon,
Yeah, Live to a Test Environment
Data Model Regen done at the time and also 3 or 4 times since
Also recycled App Pool

TABLE_NAME	COLUMN_NAME
Customer	AccountManager_c
IMCustomer	AccountManager_c
Customer_UD	AccountManager_c
Part_UD	BakeTemp_c
Part	BakeTemp_c
IMPart	BakeTemp_c
IMPart	BakeTime_c
Part	BakeTime_c
Part_UD	BakeTime_c
IMCustomer	BTEmail_c
Customer	BTEmail_c
Customer_UD	BTEmail_c
IMQuoteQty	BurdenMarkUpSC_c
QuoteQty_UD	BurdenMarkUpSC_c
QuoteQty	BurdenMarkUpSC_c
Vendor_UD	CertExpDate1_c
IMVendor	CertExpDate1_c
Vendor	CertExpDate1_c
Vendor	CertExpDate2_c
IMVendor	CertExpDate2_c
Vendor_UD	CertExpDate2_c
PartRev_UD	Chemistry_c
PartRev	Chemistry_c
IMPartRev	Chemistry_c
PartClass	ClassSite_c
PartClass_UD	ClassSite_c
IMPartClass	ClassSite_c
IMPartLot	CMI_c
PartLot_UD	CMI_c
PartLot	CMI_c
PartRev_UD	Concessions_c
PartRev	Concessions_c
JobHead_UD	Concessions_c
JobHead	Concessions_c
IMPartRev	Concessions_c
IMJobHead	Concessions_c
DMRCorAct_UD	ContainmentCmt_c
IMDMRCorAct	ContainmentCmt_c
DMRCorAct	ContainmentCmt_c
Part_UD	CustID_c
Part	CustID_c
IMPart	CustID_c
IMPart	CustomerPartNum_c
Part_UD	CustomerPartNum_c
Part	CustomerPartNum_c
IMRMADtl	CustomerReturnsNumber_c
RMADtl_UD	CustomerReturnsNumber_c
RMADtl	CustomerReturnsNumber_c
OrderRel_UD	ExcludeFromOTD_c
OrderRel	ExcludeFromOTD_c
OrderDtl	ExcludeFromOTD_c
OrderDtl_UD	ExcludeFromOTD_c
IMOrderRel	ExcludeFromOTD_c
IMOrderDtl	ExcludeFromOTD_c
IMPORel	GoodsInwardsPriority_c
PORel	GoodsInwardsPriority_c
PORel_UD	GoodsInwardsPriority_c
Vendor	InfoSec_c
IMVendor	InfoSec_c
Vendor_UD	InfoSec_c
PartPlant	InspectionReq_c
PartPlant_UD	InspectionReq_c
IMPartPlant	InspectionReq_c
Column	is_computed
IMQuoteQty	LaborMarkUpSC_c
QuoteQty	LaborMarkUpSC_c
QuoteQty_UD	LaborMarkUpSC_c
QuoteQty_UD	MaterialMarkUpSC_c
QuoteQty	MaterialMarkUpSC_c
IMQuoteQty	MaterialMarkUpSC_c
IMJobHead	MfgArea_c
JobHead	MfgArea_c
JobHead_UD	MfgArea_c
RcvDtl_UD	MfgNumName_c
RcvDtl	MfgNumName_c
IMRcvDtl	MfgNumName_c
IMRcvDtl	MfgPartNum_c
RcvDtl	MfgPartNum_c
RcvDtl_UD	MfgPartNum_c
QuoteQty_UD	MiscCostMarkUpSC_c
QuoteQty	MiscCostMarkUpSC_c
IMQuoteQty	MiscCostMarkUpSC_c
IMQuoteQty	MiscCostSC_c
QuoteQty	MiscCostSC_c
QuoteQty_UD	MiscCostSC_c
Part	MSL_c
PartXRefMfg_UD	MSL_c
PartXRefMfg	MSL_c
Part_UD	MSL_c
IMPart	MSL_c
IMPartXRefMfg	MSL_c
QuoteQty	MtlBurMarkUpSC_c
QuoteQty_UD	MtlBurMarkUpSC_c
IMQuoteQty	MtlBurMarkUpSC_c
IMQuoteQty	OtherSite_c
QuoteQty_UD	OtherSite_c
QuoteQty	OtherSite_c
Vendor_UD	PCBSupplier_c
IMVendor	PCBSupplier_c
Vendor	PCBSupplier_c
POHeader	POAck_c
IMPOHeader	POAck_c
POHeader_UD	POAck_c
POHeader_UD	POCancel_c
IMPOHeader	POCancel_c
POHeader	POCancel_c
OrderHed_UD	PORevision_c
OrderHed	PORevision_c
IMOrderHed	PORevision_c
DMRCorAct	PreventCmt_c
IMDMRCorAct	PreventCmt_c
DMRCorAct_UD	PreventCmt_c
IMQuoteQty	PriceBurMarkUpSC_c
QuoteQty_UD	PriceBurMarkUpSC_c
QuoteQty	PriceBurMarkUpSC_c
QuoteQty	PriceBurProfitSC_c
QuoteQty_UD	PriceBurProfitSC_c
IMQuoteQty	PriceBurProfitSC_c
IMQuoteQty	PriceLbrMarkUpSC_c
QuoteQty_UD	PriceLbrMarkUpSC_c
QuoteQty	PriceLbrMarkUpSC_c
QuoteQty	PriceLbrProfitSC_c
QuoteQty_UD	PriceLbrProfitSC_c
IMQuoteQty	PriceLbrProfitSC_c
IMQuoteQty	PriceMscChrgMarkUpSC_c
QuoteQty_UD	PriceMscChrgMarkUpSC_c
QuoteQty	PriceMscChrgMarkUpSC_c
QuoteQty	PriceMtlBurMarkUpSC_c
QuoteQty_UD	PriceMtlBurMarkUpSC_c
IMQuoteQty	PriceMtlBurMarkUpSC_c
IMQuoteQty	PriceMtlMarkUpSC_c
QuoteQty_UD	PriceMtlMarkUpSC_c
QuoteQty	PriceMtlMarkUpSC_c
QuoteQty_UD	PriceSubMarkUpSC_c
QuoteQty	PriceSubMarkUpSC_c
IMQuoteQty	PriceSubMarkUpSC_c
IMQuoteQty	PriceTotalMarkupSC_c
QuoteQty	PriceTotalMarkupSC_c
QuoteQty_UD	PriceTotalMarkupSC_c
IMPORel	PriorityComments_c
PORel	PriorityComments_c
PORel_UD	PriorityComments_c
OrderDtl	ReasonCode_c
OrderDtl_UD	ReasonCode_c
IMOrderDtl	ReasonCode_c
IMVendor	ReBall_c
Vendor_UD	ReBall_c
Vendor	ReBall_c
OrderDtl	Review_c
OrderDtl_UD	Review_c
IMOrderDtl	Review_c
IMPartXRefMfg	RoHS_c
PartXRefMfg	RoHS_c
PartXRefMfg_UD	RoHS_c
IMVendor	Scope_c
Vendor_UD	Scope_c
Vendor	Scope_c
IMPart	SecClass_c
Part	SecClass_c
Part_UD	SecClass_c
PartClass	SiteName_c
PartClass_UD	SiteName_c
IMPartClass	SiteName_c
Part	SourceLev_c
Part_UD	SourceLev_c
IMPart	SourceLev_c
PartLot	STI_ProjectID_c
PartLot_UD	STI_ProjectID_c
IMPartLot	STI_ProjectID_c
IMPORel	STI_ProjectID_c
PORel_UD	STI_ProjectID_c
PORel	STI_ProjectID_c
PORel	STI_WBSPhase_c
PORel_UD	STI_WBSPhase_c
IMPORel	STI_WBSPhase_c
IMPartLot	STI_WBSPhase_c
PartLot_UD	STI_WBSPhase_c
PartLot	STI_WBSPhase_c
QuoteQty_UD	SubcontractMarkUpSC_c
QuoteQty	SubcontractMarkUpSC_c
IMQuoteQty	SubcontractMarkUpSC_c
Vendor	SupCert1_c
Vendor_UD	SupCert1_c
IMVendor	SupCert1_c
IMVendor	SupCert2_c
Vendor_UD	SupCert2_c
Vendor	SupCert2_c
IMBAQExtDatasource	TokenServer_ClientID
BAQExtDatasource	TokenServer_ClientID
BAQExtDatasource	TokenServer_ClientSecret
IMBAQExtDatasource	TokenServer_ClientSecret
IMQuoteQty	TotalBurCostSC_c
QuoteQty	TotalBurCostSC_c
QuoteQty_UD	TotalBurCostSC_c
QuoteQty_UD	TotalCostSC_c
QuoteQty	TotalCostSC_c
IMQuoteQty	TotalCostSC_c
IMQuoteQty	TotalLbrCostSC_c
QuoteQty_UD	TotalLbrCostSC_c
QuoteQty	TotalLbrCostSC_c
QuoteQty_UD	TotalMarkUpSC_c
QuoteQty	TotalMarkUpSC_c
IMQuoteQty	TotalMarkUpSC_c
IMQuoteQty	TotalMtlBurCostSC_c
QuoteQty	TotalMtlBurCostSC_c
QuoteQty_UD	TotalMtlBurCostSC_c
QuoteQty_UD	TotalMtlCostSC_c
QuoteQty	TotalMtlCostSC_c
IMQuoteQty	TotalMtlCostSC_c
IMQuoteQty	TotalSubCostSC_c
QuoteQty	TotalSubCostSC_c
QuoteQty_UD	TotalSubCostSC_c
IMShipVia	TransitTime_c
ShipVia	TransitTime_c
ShipVia_UD	TransitTime_c
Customer_UD	WDAllowedEarly_c
IMCustomer	WDAllowedEarly_c
Customer	WDAllowedEarly_c
Customer	WDAllowedLate_c
IMCustomer	WDAllowedLate_c
Customer_UD	WDAllowedLate_c

As you can see from the above extract from Live, (The Test environment is the same), those fields do not exist.

Hally · March 3, 2026, 9:23am

Non of these fields exist in either database?

I am now confused… You don’t have any environment with those fields? I note in your header you mention 9.05.700c

And only have the one task agent in your ice.ExtServiceRegistration table?

You are 100% that SSRS setup in the app server is pointing to the correct version of the rdls?

The only other thing to check is do those fields exist in the related RDDS. Not sure how they could, but worth a check.

A bit tricky to diagnose looking from the outside in, but that’s where I’d look.

Daniel_White · March 3, 2026, 9:43am

Nope. They are in none of the Kinetic environments. E9 is what one of our sites are still currently using which is why i have setup the test environment so that i can test our migration scripts with the data from that Site. I havent even got to that point yet, all i did was copy the Live Environment to the Cutover environment.

Daniel_White · March 3, 2026, 9:56am

OK so i had a quick look in the DB as when i went into company maintenance it was complaining about the Enterprise Search URL not having https in it and only being http. Nothing was filled in on the screen but in the sysconfig table it was only showing http in the URL. I changed this to https and did a regen of the data model on both App servers and then restarted the app pool and no the reports are working?
Not sure how the url being incorrect would have caused any issue or maybe it was because i did the regen and app pool restart on both App servers?

Hally · March 3, 2026, 9:58am

I feel you need to redeploy your cutover env and deploy the SSRS reports to a different root folder.

I’m assuming your cutover env is a later version to live, and you registered the an new copy of the db, and upgraded it? After your deployment it should have prompted you to run the conversion workbench when you logged in for the first time.. Did that happen?

I a out of ideas after that. Sorry

Daniel_White · March 3, 2026, 10:00am

Thanks Simon i appreciate the advice, sometimes it helps bouncing ideas of other and your reply game me the idea to look in the DB at the sysconfig. It looks like it is sorted now BUT i dont know what exactly sorted it

Hally · March 3, 2026, 10:27am

Eeeew that’s even worse… Not knowing the root cause

Daniel_White · March 3, 2026, 10:30am

I have told EPICOR what i did, hopefully they will be able to identify. Still doesnt make sense that it was erroring on fields that have never existed?

tsimon · March 3, 2026, 1:23pm

Hmmm I did have an issue with QuickShip that was similar. It was telling me that I had the error below, but I did not have that UD field on the table they were saying. This did get resolved in QuickShip 2025.2.7 but I wonder if it something similar.

An item with the same key has already been added. Key: CheckBox01
PRB for reference: PRB0304007