Issues - Saas Upgrade This Past Weekend

Good Morning,
Starting a thread to see if other saas users are experiencing major issues with the 2021.1.7 delayed option upgrade that occurred this past weekend. We are experiencing some major issues with MRP, Tasks, and on Tuesday and Wednesday, the C# code on some, but not all customizations disappeared.

We were told they were re-indexing the database (did not finish until Monday afternoon). We were not given this information until after our Full MRP failed 5-6 times since they reported the upgrade was completed. This was the reason MRP and tasks were failing. However, it took us until last night to complete a successful full MRP run.

We’ve had a rogue MRP process start both Monday and Tuesday nights at 9/10PM CST that we did not start and wiped out or cancelled runs that we started 9-10 hours earlier.

Epicor is not giving us much information on the above issue.

We have not heard back regarding the missing C# code.

We did not have any of these issues in Pilot last month when they upgraded.

Any information would be appreciated.

Thanks,
Eric

This is the second time we’ve had this happen to one of our customizations. On ordertracker… I had to re-import from pilot.

Whats you case number for missing c# code? I could open a case and reference that in mine

CS0002680257 is the case we have open. We do not have a response yet.

To make matters worse, we tested all customizations on Saturday when this was given back to us and it was fine. It was also fine on Monday. Tuesday and Wednesday, we had these issues.

AP Invoice Entry
Part Tran History
Job Tracker
2 custom screens

We’ve imported from pilot two days in a row now.

I started a case as well.

I don’t like reading these cases about cloud upgrades. Sure these things will happen, but you guys even tested it and there were no issues.

1 Like

Utah, we have a very elaborate testing plan for upgrades as we just moved to the cloud in March. We test every customization to make sure there are no issues. Customizations that were there on Monday were missing on Tuesday, installed from Pilot, then missing again on Wednesday. I don’t think this is common, but I think they screwed some things up with this weekend’s upgrade and will not take the blame for it. There is maintenance this weekend and I’m hoping it’s to address the issues we are experiencing.

2 Likes

We had customizations that went missing as well. No C# Code - however out PartEntry and QuoteEntry customizations reverted to base Epicor.

Like you i imported from Pilot and it worked and then had to do the same thing one week later.

So far the customizations are hanging in there - however i agree they are fiddling with something and not sharing :frowning:

DaveO

1 Like

What has your uptime been since you have been on SaaS?

Our uptime has been EXCELLENT - no complaints.

The very few critical issues we have had - Epicor has been very responsive.

I know I sound like a postcard advertisement for SaaS - and let me tell you (and many will confirm) that i am not always complimentary of Epicor Support.

Howver, the SaaS service has been VERY good.

IMO - DaveO

We document any outages. We’ve only had a few hiccups with the taskagent, but no application downtime outside of planned maintenance. Even then, we barely noticed it was down. We are a 24/6 shop and not heavily using Epicor during the planned outages. I think the challenge with the cloud is they won’t admit fault is something did not go wrong. We also have an agreement with them that allows us some enhancements above the avg multi-tenant user and we have to remind them after every upgrade to put those back in place.

1 Like

We have been monitoring this also. Noticed that employees with just personalization level security were changing the customization when they saved. This jacked up the customization, the system would then load the base. It helps to turn off personalization for all users. We have an closed case but it was assigned a problem number PRB0236706.
Even with personalization’s removed this can still happen doing a right click save layouts.

2 Likes

Thanks for the feedback Eric.

How long have you been on multi-tenant? How many licensed users do you have?

-Utah

We migrated in March from a hosted environment. Roughly 75 daily users, not including MES.

@Patrick and everyone. Just checking in with a status. We are tracking an issue where saving a personalization appears to damage a customization. This is not a SaaS specific issue. We have been working internally but have been unable to recreate it more than once on our systems. We are working with support and after a review of the code have provided them with a test assembly that we think may resolve the issue. However as we have been unable to recreate it in house we are still in trouble shooting mode.

All evidence points to it being personalization related so year term disabling of personalization rights may help. We are actively working this issue and hope to get feedback from support and the customers they are working with soon.

3 Likes

Just noticed two customizations I’ve done recently to Job Entry and PO Entry (Sheet Wizard) are gone. Oddly enough, the Customization Name/layer still exists when I go to developer mode.

I had noticed this in PILOT a couple of weeks ago and thought it was odd that after working on something for a week, when I went to present it, it was gone. Couldn’t identify why, so I assumed it might have gotten deleted somehow.

Just checked our LIVE environment for something else and same thing. The tab I added via Sheet Wizard is gone.

We are SaaS just like OP and I’d be curious to know if this will be fixed before the next update is released. I can always reupload my changes but that will be unmanageable as a long term solution.

We are on prem, and didn’t notice anything in our tests we ran with 2021.1.7, but I doubt anybody changed a personalization in there.

Since we’re on prem we can choose to not go to .7 and only go to .X when they fix this. Since SaaS upgrades monthly, is .5 a safe bet to go to?

I personally only noticed it on .7 but that’s a great question.

We went ahead with .8 because of other items that have been fixed. I disabled all the personalization permissions until this gets fixed. So far, no issues here.

But I have the old version spun up just in case we need any exports/imports from that database.

1 Like

Appreciate this. We have been losing a few customizations daily, sometimes twice a day. And in some cases, no personalizations were made. We just had one of our environments refreshed to test and cannot recreate the issue.

1 Like