So, we are a few months now into going through the motions of an upgrade from Epicor 10.0.700.4 to Epicor 10.1.500.14 - this includes roughly 200-400 employee-hours of sitting through Conference Room Pilots (CRPs) testing E 10.1.500.14.
But now we’ve run into an issue that we’re on the fence about being able to live with or not - I think it may be SCR 198842, which seems to prevent us from using Start / End Labor activity to mark our serial-tracked finished goods as complete. (Now, there may be a workaround in using Report Quantity, but it seems like we had determined in the past that Report Quantity had it’s own issues with marking serial tracked parts complete?)
So, I would like to ask the group this - if you guys were all 80% into your CRP testing of an upgrade to 10.1.500.14, and then you decided you really wanted to apply patch 10.1.500.15, would you completely start over on CRP testing, possibly delaying your go-live and adding significant person-hour cost to the upgrade project? I appreciate the new development / testing / quality initiative by Epicor and understand where they are coming from, but I guess one “benefit” (it had it’s own drawbacks) of the old way was you could get a one-off fix (potentially) to apply that - in theory - only affected one little area of Epicor, thus not requiring resetting your end-to-end testing. And worse case scenario if the one-off fix ended up having a fatal flaw, you could roll it back off and not be doomed to wait weeks (or months) for a patch that fixes that fatal flaw you just patched in to your ERP.
I’m with you. Manual testing is just not going to cut it in this new fast release world. ERP software is very complex and not every company uses it in the same way. Add in the multiple optional modules and the amount of testing becomes unreasonable very quickly. There is no way for a software company to test every single combination.
I have been thinking about this a lot lately. What users need is a way not to test the software but to test how OUR company uses the software with OUR business processes and rules. Maybe this can be done with Epicor’s automated testing tool (ATE), we don’t have it so I’m not sure. In the meantime, I have been looking at the some of the other .Net Testing Frameworks out there. Most of them are Unit testing frameworks for coders but I stumbled across the world of BDD (Behavior Driven Development) where one describes the behaviors of the software and then writes tests to check the behavior. One such platform is called Cucumber (https://cucumber.io/docs/reference) which comes out of the Ruby world but there is .Net versions too. Technically, a company should be able to write specifications on how they user Epicor and then run automated tests. One of the interesting benefits of this process is the ability to create “living documentation”, so as you update your specifications, your documentation is updated at the same time. Unless your documentation is isn’t pushed to later…
I haven’t spent too much time on this project yet but if we (the users) don’t have some kind of automated testing tool then I don’t see us upgrading as often as Epicor would like. We would find a nice stable place and sit until the pressure was too high and then we would do the same type of manual exhaustive testing that you’re currently doing.
Mark W.
So I take it your answer, Mark, would be yes, you would not go live on patch 15, having done most of your CRP testing on 14, unless you were to completely reset and do all of your CRP testing over again on patch 15?
This is an interesting discussion to follow. My team owns the ‘delta checking’ when moving from .14 to .15 on releases to ensure we don’t do anything to break a customization, integration, etc. Changes are reviewed pretty thoroughly and if there is a breaking change it raises alarms up and down the line.
We are also doing more and more automation constantly as it is critical in the SaaS world - lots of small changes instead of slow huge changes. Just follow any discussions on dev in the new world and you’ll see that as the #1 priority.
You are correct that ERP is complex though and of course it only takes one instance such as you mentioned to have a scar you will carry forever. Trust is a pain to regain but we are working on deserving it.
It’s always interesting to listen to you all and how you receive and perceive the system…
Thanks for the insight (no pun intended) @Bart_Elia, I always find it
interesting and helpful to understand where Epicor stands on a given
topic. This topic being a very touchy one…
I had high hopes, after Insights last year, when I heard that Epicor was
doing away with hotfixes, because we got stuck in that nightmare where new
hotfixes were breaking existing hotfixes… So the promise of that going
away and that the bi-weekly point releases would fix known issues, not
contain any new features, wouldn’t break anything and were
thoroughly tested sound great.
In the end, it ended up being too good to be true; I traded one nightmare
for another…
When we updated from 400.14 to 400.19, Calculate Global Scheduling Order
stopped working (1820278PSC), it took almost four months to get it fixed
(400.26). That was four months without being able to run the scheduler,
which directly affected our daily operations.
Now I’m extremely gun-shy about installing point fixes unless I
absolutely need a fix they contain and then I have weeks worth of testing
to do first.
I understand the Epicor10 is complicated and that there is virtually no way
to test every customer scenario. I’m also not looking for you to address
this issue specifically. I’m only sharing to give you more knowledge about
how the point fixes are working in the real world.
Thanks again for all your posts, keep them coming!
Norman Hutchins
System Administrator
Howell Laboratories, Inc.
The epicor automated testing tool is a macro software which allows you to record how you use the software and then play it back automatically for testing
Let me be clear, the Hotfix system was not sustainable. I completely agree with Epicor here and I think rolling updates IS the right way to go. The difference between an Office365 continuous update and Epicor ERP is the vast numbers of modules/Country Specific Features/supported versions. I do not envy what Sales asks Development to do. And even if Development could meet the Sales Team’s demands, we Customers find behaviors we rely on that Development or Sales ever dreamed.
The only way I could see this working is by pushing more automated testing to the customer. That testing would test the behavior expected out of the software as that company uses it. The user would get more benefits (by producing Live Documentation/written implementation plans/etc) than just simple unit testing. Think of a tool that combined ATE with XSOL.
Maybe a halfway point between all Bugfix packages and all hotfixes would be the Microsoft model where between service packs, they publish some hotfixes for specific versions. If a fix is truly standalone, maybe it could be published on EpicWeb and be user-installable? Just a thought.
I certainly appreciate the dialogue and will keep thinking about potential solutions that makes the product AND our lives better.
Thanks for your thoughts. As to the automated testing mentioned above, there is another effort started recently as the old automated suite was not scalable to where we want to go…
I agree the change we made for delivery is the correct direction. The internet pace is here to stay for all of us in IT for good and bad. The processes to ensure the quality of that pace is something we all are faced with. If you work in an environment where your boss does not want solutions faster, let me know
One of my personal passions not speaking on behalf of the company here, just a geek … is telemetry. How do you folks feel about ‘phone home’ failures to us? If we stood up some monster data warehouse to collect ‘Dr. Watsons’, would you opt in?
My dream goal would be to have support login in the morning and email you that your crash yesterday is fixed in v 1.2.x or is scheduled for vNext due to 37 reports of the same crash, etc.
What would be your resistances to that approach? I have a list I am keeping on this and would value your input
I LOVE the idea of a Dr. Watson capability! My suggestion is to have a local Event Manager for each installation which gives a view of the state of the system to the local admin and then the ability to send selected telemetry to Epicor (Application Logs/MRP logs/Task Agent Logs/Printing Logs/Disc Usage/CPU Usage) in exchange for notices on patches.
