🫂 Kinetic - Americas - Extended Maintenance / Outage

I’d say the good news is, at least that’s out of the way for another six years!

Hold on, when is the migration to linux servers?

An improvement would be a parallel migration: Stand up the destination alongside production. Have all the time in the world to robustly configure the server. Move client databases and files as fast as capacity allows. Switch which server their endpoint directs to when they’re moved and initial index maintenance has completed. Repeat until everyone’s migrated.

Or just hail mary again?

Recovering a database doesn’t build its indexes. Note, the indexes exist, their reference structure doesn’t until it’s built. Rebuilding indexes would add additional downtime. I think it’s reasonable to assume there was enough shouting at support to push clients online without a rebuild. Reorganizing can be done on live data but needs an existing structure to reorganize. Statistics and maintenance jobs will catch up eventually, or a rebuild might get done during future maintenance downtime.

2 Likes

We are having issues with our read only database too, which I am assuming what parttrap is connected to. Put in a request to have the statistics error fixed but no response on that either.

I am trying to solution out report elements from our Pilot database and I am taking the site down.

During our outage I notice something new, “Test Mode”.
Instead of 502

For a while I got “Test Mode”

I have not seen this before.
Would be nice if it said, “Sorry, were off line now performing maintenance. We are scheduled to be back up and running at 12:45 PM server time.”

Yeah. Cloud SLA gets reduced from 99.7% to 99.3% uptime, so the time doesn’t count!

1 Like

That’s terrible.

Might wanna check the decimal point position there. :sob:

3 Likes

We too have been masively impacted by this event. I am getting zero information out of Epicor. I have created multiple service tickets only to be told that they are working on the issue and they dont have any updates. I have checked status.epicor.com throughout this event and that is completely worthless as the events have all been closed when they are clearly not.

I have reached out to our account manager, tried different avenues of approach with the various Cloud Services groups, no one will say anything. This is bigger than we think. Epicor really messed this upgrade up bad and is doing everything they can to not say anything.

This is reflecting exceptionally poorly on them as an organization and SaaS provider.

7 Likes

Here’s what we figured out at least with sql
Azure Studio/SSMS
adding the SAASXXXX_SiteID into the database field allowed us to connect and query.
image

1 Like

My support case that I entered this morning before 9:00 am, now says it has been linked to a “major case” with a case number listed, but no other details listed and no way for my to see the major case. :face_with_diagonal_mouth:

5 Likes

We are finally back up and running. I have no idea if they fixed it for everyone or they are going one by one.

3 Likes

Major case that isn’t mentioned on their status subsite…either directly under Kinetic (all show “Normal”) or as an Incident on the main page. Really scares the hell outta me if there’s an insistance to uplift to the cloud when (if?) we upgrade. I don’t want our company to suffer through this when the on-prem (albeit 3rd party hosted) setup has worked well for us.

5 Likes

We have also gotten confirmation that the major issue is resolved. All of our systems are now functioning optimally. I can only ask that anyone at Epicor that reviews these messages learns that better client communication and disclosure is a must.

4 Likes

Good to hear…now the recovery begins. Best of luck to all.

1 Like

And this just in…

Kinetic - Americas on cadence customers are scheduled for maintenance:

From: Wednesday, April 30, 3:00 AM Local Datacenter Time
To: Wednesday, April 30, 4:00 AM Local Datacenter Time

Well that should include some index rebuilding. An hour seems optimistic, but at least it’s possible to bail out on a rebuild.

4 Likes

We are in the Eastern time zone and that is the window when our 1st shift starts. I have a ticket into Epicor to find out what we are supposed to do with so little notice. fun! Fun!!!

3 Likes

This is during business hours for us as well. I can only assume they have determined it can’t wait until the actual maintenance window on Saturday.

2 Likes

And on Month End, no worries.

4 Likes

Reading through this thread… wow, just wow… :face_with_raised_eyebrow:

2 Likes

How long were you down/unresponsive for?

1 Like