JobEntrySvc For REST needs documentation

Bart_Elia · May 18, 2019, 7:22pm

Wow, take a couple of days off from e10Help for 400 go live and bam.

First, Stephen I am not sure how I stated incorrectly that the trace is all your need but rather it’s more like the best we have and usually does the job for folks reverse engineering the core processes in ERP. For that, my apologies for being incomplete.

The real struggle most have with ERP understanding is we (including myself) parachuting into the middle of a 100 Million+ line code base of 1700+ services and 200 odd modules with roots in the early 90s and feel overwhelmed by the tons and tons of functionality of the system with massive customization possibilities (Too many more than too few most would state).

Insert the adage of the how to eat an elephant - one bite at a time and you have my basic response. Anyone new to anything will go thru the aggressive push, the horror or overwhelmed anger, the imposter syndrome, the lights starting to go off and finally recognition and a bit of self confidence at the comprehension.

The services in general try to tackle the simple things simply. Unfortunately as many have found out internally as well as externally, certain processes have gained popularity and the flow in the system is not as easy as any internal or external would prefer. The easy reaction is to toss baby and bath water out and break every existing customer and partner and integration. That is something we will not do.

What we will do is build new services to simplify certain recognized flows. We are looking at new patterns that I hope to be able to have us show off over the next cycles but #SafeHarbor as always. REST is an obvious go forward and REST patterns are slightly different than the existing core methods especially to restafarians such as myself. Options are being reviewed that take legacy, existing an future needs into account.

As the majority of our new customers are cloud based and the new Kinetic UX is trickling out to market, that area is becoming more and more important to us and all will benefit from a cleaned up API surface. Stay tuned but in the meanwhile, plenty of good anecdotes up here.

Yechezkal_Gutfreund · May 19, 2019, 4:22pm

@Bart_Elia, @bconner Thank you so MUCH for your informative comments. (and @hkeric.wci , your comments about serviceConnect also are very timely, we will look more carefully at the pros/cons).

I had no idea I was opening such a hot topic. It seemed a simple technical issue, needing some information. More likely, I should have realized that humor rarely works except in person. I think some people misread this as anger. It is the opposite, Rather:

I agree with @Bart_Elia that we are really excited about what the REST interface enables. And I think our team will soon be able to show how this really improves the Epicor ecosystem, and we are looking forward to contributing.

As for the value of looking at traces. Yes, that is a technique. But probably one that should be the last resort, rather than the first.

The value and beauty of was has been done here is that by having all the information in the DB, it is a place for mining meta-information about all sorts of causal links about dependencies.

Let me sketch out an idea, that will need fleshing out, and would need some work, but it might get folks thinking about what is possible). [I will be out of the country for 2 weeks, so I cannot follow up on this]

For example, there are lots of chains of DB state changes triggered by BPMs , etc. One thought I have one could have a graph database (e.g. the free Neo4j) with attributes related to data and action chains. This would operate in parallel with the MS SQL relational DB.

One can then construct and interactive query interface to:

look for cyclical paths
look for race conditions
discover conflicts in update between transactions, processes, etc.
find spanning node paths to discover all touched BOs, or tables,
explore timing and latency for hidden execution paths

No, not a week project, but the meta-data about these causal chains already are in the DB,

If you want, we have sponsors (or epicor might have customers) that would be interested in an R&D project to exploit reinforcement learning, gradient analysis and hill climbing to do process optimization (think of discovering business rules that actually are working as bottlenecks to efficient tool and personal use).

In any case, opening the ERP system to a wide ecosystem of toolware, is a great business value for EPICOR,

(as for WCF, well I have been a Microsoft tech for many many years. Do folks here still remember DirectAnimation, DirectShow, JET, ODBC, OLEDB, and lots of other casualties. REST also has it’s issues, but I applaud the move, mostly because it allows better leverage of a large ecosystem).

Stephen_Booth · May 20, 2019, 2:25pm

Traces are always my first step if I’m automating or side-stepping/replacing an Epicor process. I try to understand what Epicor planned the software to do, and then I decide what our business wants and how much of that I need to tweak. Traces have saved thousands of hours of digging through disassembled code, and I’ve done my fair share of that as well.

I’ve also never seen this much customization ability, this much access to code, this much access to the developers and designers who built this product, on any other platform. Epicor is worlds beyond everyone in this category, and the developers have a great willingness to help.

I’m pretty sure OLEDB and ODBC are still around, and the others are still around, but under a different name (e.g. Extensible Storage Engine - Wikipedia). That’s what we do as humans…learn and grow, which requires change. Part of that change is making things simpler, faster, or smarter. Yesterday is was SOA, today it’s Cloud Computing, tomorrow it will be Exploding Space Modulators (once we make it to Mars). Same idea, simpler means, faster to the destination.

If there is something that helps your business, Epicor allows you to use that if you have the knowledge and determination to use that. Almost all technology can be attached onto the product and used alongside, so if you want to use GraphQL, Epicor isn’t preventing it. What I’ve found is there is usually someone in the community who is in the same boat and are investigating these technologies as well. The flexibility of this system incredible and you will often see encouragement to utilize that flexibility for your own processes.

I second (or third, or fourth) Haso. Read the docs, blow up a server or two (test of course). The more you do it, the easier it is to understand what is noise and what is music, especially when it comes to the traces.

Yechezkal_Gutfreund · June 4, 2019, 9:18pm

@Bart_Elia, @bconner

I thought I would fill you in on our progress. Before I left for a 10 day overseas trip, I hired a Epicor consultant. In 3.5 days he had a working Job Creation, using CURL he could

Create Job header
create Demand link (stock)
assign human resources
Add operations
Engineer, release and schedule.

All by ODATA methods (and I added the only custom method GetNextJobNumber).

The guy was:

Excited by this project
Had no REST experience
Not a programmer
But had a great attitude
NO whining about needed to spend 60+ days learning fundamentals
DID not use Trace
NO comments about C#, WPF, BLTester, (i.e. he is a non-programmer)
Did not make stupid comments like: navigate via .NET Reflector, dotPeek, dnSpy. and spending 80+ hours a week tinkering.

All he did was come with a exciting, positive attitude. And in the two day that I have been back I have slapped on a React/Redux Forms driven interface with a very powerful material-ui table/pivotchart toolware (not Kendo, which I know but one I think is even better).

@bconner, you have a great system. Strongly encourage you to follow up on what you are said above about having better documentation for the REST API. If you get that, you will get a much larger talent pool building and expanding your product via REST.

We are excited about showing and sharing this at the next EPIC or other forums. Here is my 2-cents on the value proposition for what you guys have done:

Truly RAD development times
Modern clean and custom interface
Rapid iteration for co-design of workflows with customers (we can rebuild the UX experience in hours).
Full spectrum access at the BO level to Epicor
Non-opinionated UX experience (WPF was forcing some really poor choices)
Access to powerful web based Visual Analytics tools (D3, DevExtreme Tables, Pivot, Charts, etc.)

I could go on.

Chris_Conn · June 4, 2019, 10:09pm

Well shit, sounds like he needs a better paying job.

hkeric.wci · June 4, 2019, 10:28pm

If people can do this with a blindfold and have no corruption then there is no excuse for anyone.

Especially not for those who have 7yrs of Epicor Experience or more. I should ask for money back from Codabears, Pronto Progress, cre8tive, Epicor CSG, SixS Partners for having decades of experience - Gold and Platinum Partners hmmm., probably a shit-ton of pre-made snippets, yet still taking their time and even sometimes when it takes 15 days, its buggy as hell. If anyone is reading it from any Consulting Companies, step-up your game. New Kids on the block.

Yes that means you too @Chris_Conn - spending a week Freighting and Unfreighting a Shipment, with your employer having access to the SDK with all the glory of Development Comments, SCR Numbers, Full Schema in Plain Text, Client and Server-Side Code

What if Epicor University and what is thought at Insights is, simply… wrong. That Epicor even advocating BL Tester, Tracing and other things is just a conspiracy to fill your mind with… garbage? To occupy our minds so we dont investigate Operation Paperclip… Food for thought.

When Epicor sends its “Certified” Consultants with 20yrs of experience, they cant even touch your guy, because most of the SMEs that Epicor sent, struggled, desperately, provided misinformation and had to call in for additional help… Sometimes they even give-up and call it a bug and fly back home, send us a bill and expect us to pay in full… Someone tag Epicor’s Head of Consultancy, to learn from the master.

Bart_Elia · June 4, 2019, 10:59pm

I appreciate the comments on the technology approach being more reasonable to understand and on board than alternatives we have provided in the past. There is a lot of value in being able to treat the system as a black box and not having to know all the details during your initial understanding and approach to the system.

Looking at some of this thread though I think you can see some alarms being raised about learning the details as a valuable step in adopting the product. Sooner or later, something will be wrong and having that increased level of understanding will help future you

bconner · June 5, 2019, 12:29am

@Yechezkal_Gutfreund This is absolutely the story we want to tell on what REST can do for you.
We might like to have this person out to insights next year to tell it!

Now as you dig into deeper into other BOs the above conversation still applies you might end up tracing or just trial and error figuring out what’s required in what circumstance on different BO services, but (depending on the service quite a lot of CRUD activities are as easy as described).

on the documentation - we continue to have this discussion wrt how to expose more information and make it easier to reason about. There are some obvious low hanging fruit items we’ll do like exposing more $metadata we have at hand, but the real hard part is complex contextual business rules and multi-service call flows where the exact rules really only exist in code. We have some ideas about how to generate something useful from the information we do have (telemetry, inspect-able code) - docs or even friendlier api facades. In a similar direction to a comment you made above wrt process inspection machine learning. One of those weekend research projects I’ve been meaning to try for some time now.

Yechezkal_Gutfreund · June 5, 2019, 1:40am

I think you are on the right track with these comments. A lot will have to do with how much automated reflection and introspection you can do at the source level, how much will have to be done by automated tracing, ML, and other techniques.,

Ideally, good interfaces provide nearly opaque abstractions that cover the internals. Look at the MS .NET docs for hastable: https://docs.microsoft.com/en-us/dotnet/api/system.collections.hashtable?view=netframework-4.8,

Lots of examples, remarks, see also, caveats, etc.

Yes, in my time I have done 100K+ lines of assembly code (I worked at DEC and was part of RSTS/e OS development), I have done CUDA, FPGA, etc. but we don’t want to expose those lower level machinery when we can create a BO layer over the DB. This is application work, not programming naked iron.

I do appreciate the challenges of your problem. If one was nasty, one would say: “what a ridiculous spider web of causal semantic links of data-flows across BO domains”. But, that is so unfair. You are modeling the real world, and that his the nature of real world systems. They are deeply inter-related and linked.

You might consider a GraphDB for some type for this meta-data, so that you can do 1st person projections and queries from a particular node of activity and trim the nature of the causation network.

I think the $$ spent on producing documentation will make good business sense. It will widen the user base (less dependence on pricey consultants), increase market reach (and hopefully add to the # of seats sold, and increase deep integration into business processes (customer loyalty) by allowing easier specialization to unique business needs.

Banderson · June 5, 2019, 3:15am

So it took this guy 3.5 days to do what should have taken 10 minutes?

Chris_Conn · June 5, 2019, 1:26pm

In his defense, he wasnt a developer, didnt know REST, and was very spunky.

nhutchins · June 5, 2019, 1:39pm

Well that was an entertaining and informative thread to read during my morning coffee.

Yechezkal_Gutfreund · June 5, 2019, 2:13pm

@Banderson EXCELLENT, you are making my point for me.

For a person inside of EPICOR with the inside knowledge of JobEntry, this would have been a 10 minute question. Or if the documentation was up to the level of the Microsoft .NET framework docs, it would have also have been that way.

But instead we had to hire a consultant who had to spend 3.5 days by iterative detective work to get the solution.

I did not mention this, but I also filed a tech support request with Epicare, and got back this reply:

I'm with the Tech Support side and not on the Application Support side, so I do not know the exact payload that is required for every API/BO call.  This is not knowledge that we are expected to have (and I do not).  If I knew I'd be more than happy to share that with you.  All I can do is offer advice.
Creating a Job is a complex operation.

Having said that, I spent time trying to create a Job via the REST API in my Demo DB.  This is the minimum payload that worked for me in creating a Job.

So it seems that Tech support also does not have the documentation, and has to waste time/money doing random walk detective work.

So, you help make the value proposition for Epicor to improve the REST API doc even more clear:

You save time & money for your own tech support team
You will radically save time and effort for your customers (3.5 days / 10 minutes) = (168x speedup)
You can increase market share (I don’t think SAP, or Oracle) has anything near this end-user friendly system.
We can retire consultants who approach devops with a 1970s mentality, and allow more customers to do their own customizations
Is anyone from Epicor Management listening? Yes, you have to put out some $ initially, but I think @Banderson is making the case this makes financial sense

jgiese.wci · June 5, 2019, 2:14pm

I think this topic has been fully covered and we can move on from it.

Readers Digest version everyone agrees:
Documentation needs work, but it is also understood that there is an immense level of difficulty in documenting something worked on over decades by hundreds of people with thousands of BOs.