Misspending my youth on an External BAQ using Dynamic Query in a Data Directive, only works where Appserver and Database share a machine

We’re trying to build a new way to forecast, as we sell manufactured parts assembled from purchased parts and we want to stop forecasting the manufactured items as they clog up the system with 1000s of unfirm jobs.

I built a solution that works perfectly in DEV:

1 - A stored procedure runs every day and analyzes the BOMs of everything we sold, returning a table in a separate DB containing adjustments needed to our forecast;

2 - An external BAQ grabs the data from that table;

3 - A system agent schedule triggers a data directive which contains the following:

Ice.Contracts.DynamicQuerySvcContract tQuery = Ice.Assemblies.ServiceRenderer.GetService<Ice.Contracts.DynamicQuerySvcContract>();

Ice.Tablesets.DynamicQueryTableset dsQuery = tQuery.GetByID("PROD_Update");

Ice.Tablesets.QueryExecutionTableset dsBAQ = tQuery.GetQueryExecutionParameters(dsQuery);

DataSet results = tQuery.Execute(dsQuery, dsBAQ);

and then calls the forecast BO, makes some comparisons and updates the forecast. The whole thing works really well in DEV and solved the whole mess of issues we’ve been having with forecasting since we went live.

In PROD, it fails. The only difference is that DEV’s database lives on the same machine as the AppServer, while PROD’s is on a separate VM.

I spent pretty much my whole youth trying to figure it out, or would have if I had any youth left. I’ve got it to write all kinds of error messages and have narrowed it down a bit.

Basically, the first three expressions above work fine. However, when I try to run Execute or ExecuteByID the transaction aborts.

Strangely, if I try to parse out the results DataSet, for example using results.Tables.Count.ToString(); I can see that the expected 3 tables are there; but if I run results.Tables[0].Rows.Count(); the result is zero.

Also strangely, by the way, the Datasource and external BAQ work perfectly, including if I use the BL Tester to run the DynamicQuery methods GetByID and Execute.

By calling the inner exception in a catch statement I was able to get this message from the server:

Network access for Distributed Transaction Manager (MSDTC) has been disabled. Please enable DTC for network access in the security configuration for MSDTC using the Component Services Administrative tool.

I’ve wrangled the IPSEC people into meeting me Sunday to make this change because we want to do changes without anyone in the system. Unfortunately, I don’t think it will work.

In order to test it I have gone the other way around, accessing a database on DEV where I can change those settings. Before making changes I got the same error, and then enabled network access for MSDTC and added a firewall rule. The only comprehensive info I could find was here:

Unfortunately, all it did was stop producing the inner exception and detailed message. I still get the transaction aborted.

The full code for my BPM is here. It obviously has a few other functions but I’ve stripped them all out as above for testing.


//SF_ForecastUpdate 0.09
  using (CallContext.Current.TemporarySessionCreator.SetCompanyID("RDCAN").Create() )
    {
    using (var txScope = IceContext.CreateDefaultTransactionScope() )
      {
         try
           {
                        
            //Create a BAQ object
            
            Ice.Contracts.DynamicQuerySvcContract tQuery = Ice.Assemblies.ServiceRenderer.GetService<Ice.Contracts.DynamicQuerySvcContract>(Db);
            
            if (tQuery != null)
              {
              //Load the BAQ
              
              Ice.Tablesets.DynamicQueryTableset dsQuery = tQuery.GetByID("Updated_Forecast");
              
              //Get BAQ execution parameters (there are non in this case)
              
              Ice.Tablesets.QueryExecutionTableset dsBAQ = tQuery.GetQueryExecutionParameters(dsQuery);
              
              //Execute the BAQ
              
              DataSet results = tQuery.Execute(dsQuery, dsBAQ);
              
              //loop through the latest forecast and update the original  
              
              var lastUpdateRow = results.Tables[0].Rows[0];
              
              if (Convert.ToDateTime(lastUpdateRow[12]) >= DateTime.Now.AddHours(-20)) //this will only run if the update calculator has run within the last day
                {
                
                for (int r = 0; r < results.Tables[0].Rows.Count ; r++)
                  {
                  
                  if ((r%50) == 0)
                    {
                    
                    var mrp = (from mp in Db.SysTask where mp.Company == this.Session.CompanyID select mp).FirstOrDefault();
                    
                    if (mrp.TaskDescription == "ProcessMRP" && mrp.TaskStatus == "Active")
                      {
                      string vFrom = "epicor.globalalert@rangerdesign.com";
   
                      string emailTo = "steve.fossey@rangerdesign.com" ;
                      
                      string vSubject = "Forecast Updater Stopped for MRP";
                      
                      var mailer = this.GetMailer(async: true);
                      
                      var message = new Ice.Mail.SmtpMail();
                      
                      message.SetFrom(vFrom);
                      
                      message.SetTo(emailTo);
    
                      message.SetSubject(vSubject);
                      
                      mailer.Send(message);
                      
                      throw new System.ArgumentException("MRP started before forecast updater finished", mrp.TaskDescription);
                      
                      } //end if (mrp.TaskDescription == "ProcessMRP" && mrp.TaskStatus == "Active")
                    
                    } //end if ((r%50) == 0)
                        
                        
  
                  //get the record to change based on part, plant and forecast date 
                  
                  DataRow row = results.Tables[0].Rows[r];
                  
                  string partNum = row[1].ToString();
                  
                  string plant = row[2].ToString();
                  
                  DateTime foreDate = Convert.ToDateTime(row[3]);
                  
                  int custNum = 0;
                  
                  string parentPartNum = "";
  
                  int pageSize = 0;
                  
                  int absolutePage = 0;
                  
                  bool morePages = false;
                  
string test = r.ToString() + " BAQ: " +  partNum + " Qty: " + row[10].ToString() + Environment.NewLine;
                  
                  //get the existing forecast dataset for this part. plant and forecast date
                  
                  var bo = Ice.Assemblies.ServiceRenderer.GetService<Erp.Contracts.ForecastSvcContract>(Db);
                  
                  var ds = bo.GetByID(partNum, plant, pageSize, foreDate, parentPartNum);
                  
test += "Forecast: " + ds.Forecast[0].PartNum.ToString() +  " Qty: " + ds.Forecast[0].ForeQty.ToString() + Environment.NewLine;
                  
                  if (Convert.ToBoolean(row[11]) || Convert.ToDouble(row[10]) == 0.0 ) //If BAQ NewInactive == true then just update inactive
                    { 
                    //create Before image for rollback on error
                    
                    var origRow = ds.Forecast.NewRow();
                    
                    BufferCopy.Copy(ds.Forecast[0], origRow);
                    
                    ds.Forecast.Add(origRow);
                    
                    //inactivate the forecast
                    
                    ds.Forecast[0].Inactive = true;               
                    
                    ds.Forecast[0].RowMod = "U";                    
                    
                    bo.Update(ref ds);
                    
                    } //end if inactive flag is true
                  
                  else
                    {
                    //create Before image for rollback on error
                    
                    var origRow = ds.Forecast.NewRow();
                    
                    BufferCopy.Copy(ds.Forecast[0], origRow);
                    
                    ds.Forecast.Add(origRow);  
                    
                    //update forecast dataset
                    
                    
                    ds.Forecast[0].ForeQty = Convert.ToDecimal(row[10]);  
                    
               
                    ds.Forecast[0].ForeQtyUOM = row[6].ToString();
                    
                   
                    ds.Forecast[0].ConsumedQty = Convert.ToDecimal(row[7]);
                    
                  
                    ds.Forecast[0].Inactive = Convert.ToBoolean(row[11]);               
                    
                    ds.Forecast[0].RowMod = "U";
                    
                    bo.Update(ref ds);                 
                    
                    } //end else (IE BAQ NewInactive == false)
                  
test += "Update: " + ds.Forecast[0].PartNum.ToString() +  " Qty: " + ds.Forecast[0].ForeQty.ToString();

Ice.Diagnostics.Log.WriteEntry(test);
                  
                  } //end for r in results
              
                } //end if (Convert.ToDateTime(lastUpdateRow[12]) <= DateTime.Now.AddHours(-1))
                else
                  {
                  //if Updated_Forecast not done, email alert
                  
                  string vFrom = "epicor.globalalert@rangerdesign.com";
   
                  string emailTo = "steve.fossey@rangerdesign.com" ;
                  
                  string vSubject = "Forecast Updater Could Not Run";
                  
                  var mailer = this.GetMailer(async: true);
                  
                  var message = new Ice.Mail.SmtpMail();
                  
                  message.SetFrom(vFrom);
                  
                  message.SetTo(emailTo);

                  message.SetSubject(vSubject);
                  
                  mailer.Send(message); 
                  
                 
                  
                  }

              } //end if tQuery != null             

           } catch (Exception e)  //end try
           {
           
           Ice.Diagnostics.Log.WriteEntry("forecast BPM error " + e);
           
           }  // end catch

      txScope.Complete();
      
      } //End TXScope
    
    }//End company ID session

If anyone has any ideas I’ll be very grateful, and will share the business case, process improvement and project once it’s working

Could the external DB table be implemented via. UD table? And do away with external BAQ’s altogether.

1 Like

Thanks for the idea. I don’t know… can you put a stored procedure on an Epicor database?

And can you write to a UD table other than from within Epicor?

Just spit balling …
Can the stored procedure be implemented in a BPM?

But your DEV system proves your solution works. Can you make an external BAQ on the PROD App that sees the DB?

1 Like

that’s the crazy thing. With a BPM on PROD I can access a standard BAQ no problem. And an external BAQ on PROD happily gets data from DEV. And I can read the execution parameters from the external BAQ just fine - but can’t execute.

Spitball away, Jeeves, my rubber duck is out of ideas.

@SteveFossey I think a scheduled UBAQ would work for this. My initial query results are typically what I work with, but you could have a query on a 1 row UD table and then do anything from there. The bpm(s) are post processing on GetList.

Example of three calculations on ttResults
image

Add a schedule to system agent
Set a BAQ Export process of your UBAQ on your schedule.
image

EDIT: There is also now a module you can purchase to schedule.

1 Like

ok, that’s interesting. I set up a couple of servers to test the MSDTC solution and it didn’t work, so I actually built a workaround in a UBAQ. However, a user needs to manually get the list and then run a custom action, and I refactored my BPM code to run as Base in myUBAQ.RunCustom.

I started using data directives on the SysAgentSched table earlier this year and never had them fail before, but I never tried to work with external data before.

So if I ran it on GetList instead of RunCustomAction I would still be able to get the query ttResults in a BPM?

@SteveFossey Same bpm should work.

1 Like

@gpayne, thanks for you help. I took my RunCustomAction code and put it in a new post-processing directive on GetList.

It does indeed run if I open the UBAQ and get list, so that’s good.

However, if I try to call it with the BAQ Export module either as “Submit now” or on a schedule, I get

Query failed to run An error occurred while reading from the store provider's data reader. See the inner exception for details. (ctx, CompanyID, _RelatedToFile, JobNum) => ctx.XFileAttch.With(UpdLock).Where(row => (((row.Company == CompanyID) AndAlso (row.RelatedToFile == _RelatedToFile)) AndAlso (row.Key1 == JobNum)))

I’m not certain about the inner exception; but in System Monitor there’s a note that an exception occurred:

Exception occured: System.ArgumentNullException: Value cannot be null. Parameter name: tableset at Epicor.Customization.Bpm.TablesetWithFilter`1.Attach(TTableset tableset, Boolean asCurrent) in C:\_Releases\ICE\ICE3.2.100.33\Source\Server\Internal\Lib\Epicor.Customization.BPM\TablesetWithFilter.cs:line 40 at Epicor.Customization.Bpm.Ubaq8159D6FFF29C4B6980AB1F13289101AD.GetListDirectiveBase.InitializeCore(GetListParameters parameters, Boolean preparation) at Epicor.Customization.Bpm.DirectiveBase`3.Execute(TParam parameters) in C:\_Releases\ICE\ICE3.2.100.33\Source\Server\Internal\Lib\Epicor.Customization.BPM\DirectiveBase.Generic.cs:line 125 at Epicor.Customization.Bpm.MethodCustomizationBase2`3.<>c__DisplayClass20_0.<RunDirectives>b__5(MethodDirectiveBase`3 dir) in C:\_Releases\ICE\ICE3.2.100.33\Source\Server\Internal\Lib\Epicor.Customization.BPM\MethodCustomizationBase2.cs:line 208 at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source, Func`2 predicate) at Epicor.Customization.Bpm.MethodCustomizationBase2`3.RunDirectives(TParam parameters) in C:\_Releases\ICE\ICE3.2.100.33\Source\Server\Internal\Lib\Epicor.Customization.BPM\MethodCustomizationBase2.cs:line 212 at Epicor.Customization.Bpm.CustomizationBase2`3.Execute(TParam parameters) in C:\_Releases\ICE\ICE3.2.100.33\Source\Server\Internal\Lib\Epicor.Customization.BPM\CustomizationBase2.cs:line 71 at Epicor.Customization.Bpm.Ubaq8159D6FFF29C4B6980AB1F13289101AD.RDCAN_SF_ForecastUpdaterSvcCustomization.GetList(DynamicQueryTableset queryDS, QueryExecutionTableset executionParams, Int32 pageSize, Int32 absolutePage, Boolean& hasMorePage) at Ice.Services.BO.DynamicQuerySvc.TryExecuteUpdatableQuery(DynamicQueryTableset queryDefinition, QueryExecutionTableset executionParams, DataSet& result) in C:\_Releases\ICE\ICE3.2.100.33\Source\Server\Services\BO\DynamicQuery\DynamicQuery.Impl.cs:line 1180 at Ice.Services.BO.DynamicQuerySvc.ExecuteByID(String queryID, QueryExecutionTableset executionParams) in C:\_Releases\ICE\ICE3.2.100.33\Source\Server\Services\BO\DynamicQuery\DynamicQuery.cs:line 84 at Ice.Services.BO.DynamicQuerySvcFacade.ExecuteByID(String queryID, QueryExecutionTableset executionParams) in C:\_Releases\ICE\ICE3.2.100.33\Source\Server\Services\BO\DynamicQuery\DynamicQuerySvcFacade.cs:line 122 at Ice.Internal.XA.BAQReport.RunProcess(Int64 instanceTaskNum, String outputFileName) in C:\_Releases\ICE\3.2.100.0\Source\Server\Internal\XA\XABaqExport\BaqExport.cs:line 46

I can’t think where I might be passing a null value!

I have tried ruling out permission errors by logging in as manager and running the BAQ export process thusly - no joy!

@SteveFossey if you are on SSRS activity monitor when the manual vs the background process runs can you see a difference in the generated query? What user is running the task agent?

Are you writing debug messages to the serverlog or event viewer?

I think I floated up to the level of my incompetence on that one. I figured out how to view the activity monitor but I don’t really know what I’m looking at. I can identify a few of the expensive queries, but I don’t see anything that looks like the fairly simple external/updatable BAQ I’m working with.

I did find a help file on different types of BAQ consumers that can call UBAQs - no mention of EBAQs - but all consumers give the same results, IE calling them manually triggers the post-process directive on GetList, but calling them via a schedule des not. This includes data directives on the SysAgentSched table, BAQ reports on a schedule and process sets on a schedule. The remaining consumer listed is a quick search, and although I can create one I can’t think of a way to call it.

@Epic_Santiago, I’ve seen you post about server architecture before - is it possible you’d have some insight here? the TL;DR is that loading data from an external BAQ works when the database and the appserver are on the same machine (VM) but not when they’re on different machines. The latter is how we have PROD set up. I tried to enable distributed transactions, and have tried both dynamic queries in a BPM and updatable BAQs - they work in DEV where everything is on the same machine, but not in PROD.

I should mention that all these last attempts have yielded no output in my event viewer, so I don’t think it ever got to my Ice.Diagnostics.Log.WriteEntry("Test message"); statement.

The task agent is running as “Manager” which is both the admin account and a domain admin account.

I have tried running BAQ export process as myself (global sec) and as Manager.

In every case with the uBAQ, the only error message I get is actually in the system monitor in Epicor, and is always the Exception occured: System.ArgumentNullException: Value cannot be null. Parameter name: tableset mentioned earlier.

@SteveFossey I thought you were going to move the data into a UD table. I know that works. I also have a UBAQ that just builds a dataset and does not write the temp results.

Is it safe / feasible to write directly to a UD table? That’s something we could do. We were discussing today whether instead of using our separate DB we could just safely put a stored procedure on the PROD DB and have it load a UD table. Then a BPM would have everything within Epicor.

@SteveFossey I would NOT use a SP, but use a bpm to write to the UD table and then use that data to build the forecast.

I was afraid you’d say that, although I fully agree… I think it’s beyond my C# chops as yet. That’s why we went with an external DB in the first place. We’re basically collecting up everything manufactured that we sold today and breaking their BOMs down to the purchased components, and updating the forecast with the results. Takes about 30 minutes to run in a SP.

Thanks very much for the help though, it gave me a few more threads to tug on. I think there must be an execution parameter that is set when I call a UBAQ manually, but not when it’s called by a process. I just haven’t found it yet.

@SteveFossey If the BAQ does the same base sql query as the SP to build the list of orders that you are working then the bom/ revs are a foreach of PartMtl that udpates UDxx to get a total of that material. then you read back UDxx.

I do something similar to calculate the Part ETA above in a series of foreach’s. I take a list of jobs and build a material list then build a timephase for every material in that list.

If you want to PM me your query I will see if what I use a lot would apply.

1 Like

Thanks for reaching out @SteveFossey. BAQ is not my area of expertise and it looks like you have a gnarly use case here. Generally I would avoid DTC and gravitate towards a different solution. That said, did you configure DTC on both the AppServer and database machines or one of them? Since DTC is doing RPC between both machines, it has to be configured on both ends.
You’ve probably already read this but just in case, here’s a Microsoft page on the topic:

Ah. No, I did not catch that it was bidirectional.

I’ll update this post later in more detail but in the end I solved the problem by calling a BAQ report with routing on a proces set on a system agent schedule, and in the routing flowchart using a single “generate report” widget. The BAQ report calls the UBAQ, which triggers the bpm on GetList.

Every other non-manual method failed. This has run three nights without error so I think it’s going to be ok…

1 Like