Iodrives : RE: Vantage getting slower by the week

system · February 3, 2010, 4:32am

I found one reason for slowness was due to BAQ 'leaking'. When I was testing BAQs they were taking longer and longer to generate.

In the end, the system started reading more and more from disk which in turn slows other users accessing the disks.

Sometimes the BAQ would cause Epicor to crash (even though it was written correctly) soÂ the Apps were still working away in the background but I had to log back in.

That was the major cause of our slowness.

We also found defragging several times helped and also putting allocating 6 x 1gb blocks for the DB.

Our DB is getting bigger all the time due to new data.

I do find that sometimes on Monday's it seems really slow and restarting the server seems to fix it. Obviously you shouldnt have to do that.

Also the max no of apps can assist in the speed issue.

Â

________________________________
From: Rupert <rupert@...>
To: vantage@yahoogroups.com
Sent: Wed, 3 February, 2010 7:53:09
Subject: [Vantage] Re: Vantage getting slower by the week.

Â
Are you using MES or Office MES for production.

If so, make sure that users are clocking out every night or when necessary - If this is not done it ccan impact the speed significantly.

you can also run 8920 and 8930 in admin to see the current usage within jobs and see if peeps are actually clocking out.

>

[Non-text portions of this message have been removed]

system · February 3, 2010, 4:32am

Another good one is a dump and load.

________________________________
From: Rupert <rupert@...>
To: vantage@yahoogroups.com
Sent: Wed, 3 February, 2010 7:53:09
Subject: [Vantage] Re: Vantage getting slower by the week.

Â
Are you using MES or Office MES for production.

If so, make sure that users are clocking out every night or when necessary - If this is not done it ccan impact the speed significantly.

you can also run 8920 and 8930 in admin to see the current usage within jobs and see if peeps are actually clocking out.

>

[Non-text portions of this message have been removed]

system · February 3, 2010, 9:17am

You have a terrible cache rate hit - don't look at the percentage but rather the actual numbers.

For 198,000 records - the system loaded 115,000 DB blocks. That is plain terrible. The entire problem probably lies here - although RAID configurations and memory may be related - if your buffer hits are bad - then the system will crawl.

How to fix:

1) Increase the -B parameter (# of Block in the DB Buffer on the configuration screen in Progress Explorer). Ideally this should be at least 10-20% of the total DB size. This is sometimes impractical - but generally as the DB grow very large - there is a lot of history - which if not cached is generally not a problem. You list 20,000 DB buffers @ 8k per block you have allocated a whopping 163MB (mega bytes, not giga bytes) to you buffer. This is all that Progress will use - does not matter if you had a terabyte of memory in your system.

2) Look at what specifically is running when the slowdowns occur. A poorly written report can easily be asking Progress for the entire table (or worse an entire table iteratively), with a bad cache hit rate the reason you don't see the problem (on CPU meters) is because the bottle neck is the disk bandwidth. Again faster disk will help - but first check with above - and then fix the bad report queries.

3) DB Dump and load. This aligns similar records together. As the DB populates you may have various tables stored in one records block - after a D&L when a block is retrieved from disk it is chock full of only the records your are looking for.

Thank you,

Justin C. Muller
President, Managing Partner
Velocity Development Group, LLC
jmuller@...<mailto:jmuller@...>
www.velocitydev.com<http://www.velocitydev.com>

From: vantage@yahoogroups.com [mailto:vantage@yahoogroups.com] On Behalf Of mmcwilliams22
Sent: Tuesday, February 02, 2010 4:55 PM
To: vantage@yahoogroups.com
Subject: [Vantage] Re: Vantage getting slower by the week.

We are progress.
Sorry to take up so much of the forum today.
I notice at points stats like this in promon:
I wonder if these reads are really killing me.
Record Updates 9
Record Creates 2
DB Writes 91
BI Writes 3
Record Locks 1022
Record Reads 198728
DBReads 115146
Writes by APW 100%
Buffer Hits 72%
Writes by BIW 33%
DB size 977MB

--- In vantage@yahoogroups.com<mailto:vantage%40yahoogroups.com>, Jared Allmond <jallmond@...> wrote:
>
> I believe you mentioned you are on SQL version, and you are backing up several times per day, are these full backups, or log backups? Are you sure the degraded performance does not coincide with a backup? I have found Volume Shadow Service helpful, but someone else mentioned to disable it. Are you in Simple or Full Recovery mode? Is your log file really bloated? How large is your data file?
>
> Perfmon is going to be your friend in narrowing down any physical bottlenecks like disk I/O, CPU, memory. Offhand I would suggest that 4GB is pretty small by today's standards. If you have more slots, you can upgrade fairly cheaply.
>
> Are you comfortable running SQL queries using SQL Server Management Studio (SSMS)? Try this script to see what SQL statement is currently executing on the server. Might help you determine what user activities are the dog.
>
> USE [Mfgsys803]
> GO
> BEGIN
> -- Do not lock anything, and do not get held up by any locks.
> SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED
>
> -- What SQL Statements Are Currently Running?
> SELECT [Spid] = session_Id
> , ecid
> , [Database] = DB_NAME(sp.dbid)
> , [User] = nt_username
> , [Status] = er.status
> , [Wait] = wait_type
> , [Individual Query] = SUBSTRING (qt.text,
> er.statement_start_offset/2,
> (CASE WHEN er.statement_end_offset = -1
> THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2
> ELSE er.statement_end_offset END -
> er.statement_start_offset)/2)
> ,[Parent Query] = qt.text
> , Program = program_name
> , Hostname
> , nt_domain
> , start_time
> FROM sys.dm_exec_requests er
> INNER JOIN sys.sysprocesses sp ON er.session_id = sp.spid
> CROSS APPLY sys.dm_exec_sql_text(er.sql_handle)as qt
> WHERE session_Id > 50 -- Ignore system spids.
> AND session_Id NOT IN (@@SPID) -- Ignore this current statement.
> ORDER BY 1, 2
> END
>
> Jared
> _______________________
> Jared Allmond
> IT Systems Engineer
> Wright Coating Technologies
> jallmond@...<mailto:jallmond@...>
> voice: 269.344.8195
> direct: 269.341.4353
> fax: 269.344.3007
>
>
>
> [Non-text portions of this message have been removed]
>

[Non-text portions of this message have been removed]

system · February 3, 2010, 11:20am

Good Day Ron:

I could not come up with a good term for the Iodrive - Solid state
flash memory is good.

You may well be OK with that since one vendor said the Iodrive is
the equivalent of 100-500 "Spindles"; whereas a 24-disc Array is just 24
spindles.

Do the Iodrives still cost $15,000?

Len

________________________________

From: vantage@yahoogroups.com [mailto:vantage@yahoogroups.com] On Behalf
Of IT Support
Sent: Tuesday, February 02, 2010 3:47 PM
To: vantage@yahoogroups.com
Subject: RE: [Vantage] Re: Vantage getting slower by the week.

Len,

We currently only have a PE2900 @ 2x 2.0GHz with 24GB memory.

We are only a 30 user company, with 20 consistent users thru out the
day.

Vantage 803.408B is running quite well for us now (performance wise) &
we did a pretty good analysis of our server when we had the performance
issues.

After every test we did, no matter what time of day or how many users
were on the system, the only taxed part of the server was the disk IO.

I know we are not optimal in this area as well... we really should have
another RAID 10 volume, but things are running well.

We are considered a "small" Vantage customer I guess & separating
Progress off onto another server would be overkill for us.

When we move the R710, we will be gaining quite a bit; 1 thousand MHz
per CPU, the new 1333 memory architecture, the ioDrive, & 64bit server
OS.

The ioDrive is not SSD, but solid-state flash memory. It's really quite
impressive.

I guess if we are not satisfied after the move to 9.05 with that setup,
we'll just wait for 10.0 & try again... J

Ron Flynn

Cubic Designs, Inc.

CUSTOM MEZZANINE SYSTEMS

16770 West Victor Road

New Berlin, WI 53151

262.789.1966 xt. 124

262.789.1970 fax

From: vantage@yahoogroups.com <mailto:vantage%40yahoogroups.com>
[mailto:vantage@yahoogroups.com <mailto:vantage%40yahoogroups.com> ] On
Behalf
Of Len Hartka
Sent: Tuesday, February 02, 2010 1:40 PM
To: vantage@yahoogroups.com <mailto:vantage%40yahoogroups.com>
Subject: RE: [Vantage] Re: Vantage getting slower by the week.

Good Day Ron:

A R710?
For 75-100 users, we have three R710's plus a 24 drive array.
Of course, we did NOT get the iodrivecard - I assume that is the
"SSD unit" that plugs right into the PCI slot on the motherboard. I
know it as an IOmega device.

len.hartka@... <mailto:len.hartka%40sunautomation.com>
<mailto:len.hartka%40sunautomation.com>

________________________________

From: vantage@yahoogroups.com <mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com>
[mailto:vantage@yahoogroups.com <mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com> ] On
Behalf
Of IT Support
Sent: Tuesday, February 02, 2010 1:27 PM
To: vantage@yahoogroups.com <mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com>
Subject: RE: [Vantage] Re: Vantage getting slower by the week.

Charlie,

I never had the opportunity to attend that class, however even though it
was after the fact,

I attended the admin/performance class at Perspectives & that's all they
stated as well.

When we go to Epicor 9.05, we are going a different route yet again.

We will be going with a R710 maxed out and using a ioDrive160 card to
handle the DB's.

Due to 2009 being such a poor year, we were not able to make any
hardware improvements, but things are getting better to the point

That we do not want to make that mistake again in 2010 for Epicor 9.

A fellow group member brought up the card & after thorough conversations
with our consultants & ioFusion we feel for the money there isn't a
better solution.

1 caveat though is that the card is not currently bootable, so we will
have the OS on a RAID 1 volume & everything else on the card.

Ron Flynn

Cubic Designs, Inc.

CUSTOM MEZZANINE SYSTEMS

16770 West Victor Road

New Berlin, WI 53151

262.789.1966 xt. 124

262.789.1970 fax

From: vantage@yahoogroups.com <mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com>
[mailto:vantage@yahoogroups.com <mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com> ] On
Behalf
Of Charlie Wilson
Sent: Tuesday, February 02, 2010 12:15 PM
To: vantage@yahoogroups.com <mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com>
Subject: RE: [Vantage] Re: Vantage getting slower by the week.

Depending on how many drives are in the array of a raid 10 you can
double
and quadrpel read and write rates. This is due to the fact that the raid
is
a stripe across multiple mirrors which means you can write a string
sectors
at once basically. There are 1000 and white paper reports about raid
configurations you can go read. But I can't believe that was epicors
recommendation. I went to the admin class and it was raid 10 raid 10
raid 10
I had to make sure it wasn't tattooed on me somewhere before I left.

~Charlie

_____

From: vantage@yahoogroups.com <mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com>
[mailto:vantage@yahoogroups.com <mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com> ] On
Behalf Of
IT Support
Sent: Tuesday, February 02, 2010 1:02 PM
To: vantage@yahoogroups.com <mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com>
<mailto:vantage%40yahoogroups.com>
Subject: RE: [Vantage] Re: Vantage getting slower by the week.

I'm with Charlie if that's true...

We had severe performance issues & installed a RAID 10 volume for the
DB & have seen a much improved performance increase.

People actually comment on how well Vantage is running, which was NEVER
the case before.

We were only able to add a single 3x3 RAID 10 (15K Drives) due to
chassis limitations, otherwise I would have added a second set like
suggested for the transaction logs...

Also, our consultants did a very thorough tuning of the entire server...

We are a 30 user SQL environment.

Ron Flynn

Cubic Designs, Inc.

CUSTOM MEZZANINE SYSTEMS

16770 West Victor Road

New Berlin, WI 53151

262.789.1966 xt. 124

262.789.1970 fax

From: vantage@yahoogroups <mailto:vantage%40yahoogroups.com> .com
[mailto:vantage@yahoogroups <mailto:vantage%40yahoogroups.com> .com] On
Behalf
Of Charlie Wilson
Sent: Tuesday, February 02, 2010 11:53 AM
To: vantage@yahoogroups <mailto:vantage%40yahoogroups.com> .com
Subject: RE: [Vantage] Re: Vantage getting slower by the week.

And everything is on this raid 1?

~Charlie

_____

From: vantage@yahoogroups <mailto:vantage%40yahoogroups.com> .com
<mailto:vantage%40yahoogroups.com>
[mailto:vantage@yahoogroups <mailto:vantage%40yahoogroups.com> .com
<mailto:vantage%40yahoogroups.com> ] On
Behalf Of
mmcwilliams22
Sent: Tuesday, February 02, 2010 12:42 PM
To: vantage@yahoogroups <mailto:vantage%40yahoogroups.com> .com
<mailto:vantage%40yahoogroups.com>
Subject: [Vantage] Re: Vantage getting slower by the week.

Dual Xeon 2.8 with 4GB of RAM 15k disks
Windows 2003 Server
15 Shop Floor
25 Office users
Antivirus is ESET with all the recommended exlusions.
Volume shadow copy is not enabled.
Indexing is disabled.
I defragged the server on Sunday yet still have 60% file & 30% total.
Vantage is backed up at 2a, 6a, 10a, 4p & then also uploaded to an
offsite
location.

--- In vantage@yahoogroups <mailto:vantage%40yahoogroups.com> .com,
Michael
Barry <mbarry@...> wrote:

>
> Sorry, I left out a couple of additional questions. How is your server

configured? I got the RAID 1 but what about CPU/RAM etc? How many
concurrent
users including shop floor? I'm assuming that you are running some
variant
of Windows Server so do you have things like volume shadow copy or the
indexing service running? Is there a periodic disk defragmenter or
online
backup process running?

>
> Regards,
>
> Michael
>
> Michael Barry
> Aspacia Systems Inc
> 866.566.9600
> 312.803.0730 fax
> http://www.aspacia <http://www.aspacia <http://www.aspacia.com/

<http://www.aspacia.com/>
<http://www.aspacia.com/ <http://www.aspacia.com/> > >
com/>
com/

>
> This email, and any attachments thereto, is intended only for use by

the
addressee(s) named herein and may contain legally privileged and/or
confidential information. If you are not the intended recipient of this
email, you are hereby notified that any dissemination, distribution or
copying of this email, and any attachments thereto, is strictly
prohibited.
If you have received this email in error, please immediately notify me
by
telephone and permanently delete the original and any copy of any email
and
any printout thereof.

>
>
>
>
> On Feb 2, 2010, at 9:17 AM, mmcwilliams22 wrote:
>
> > Yes. The thing is that it is not only ODBC, it is the whole system.

A
user may be in AP invoice entry and all the sudden they start getting
"Not
Responding" and it takes a minute for a new line to appear and then 30
minutes later it is just cruising along. We are using 15k drives in RAID
1,
I am sure that has some to do with it. But will RAID 10 really make that
much of a difference & if that is the case then maybe SSD is really
worth
looking into. If I knew where to pinpoint I could throw the resources at
it.

> >
> > --- In vantage@yahoogroups <mailto:vantage%40yahoogroups.com> .com,

"Blake Clemens" <blake.clemens@> wrote:

> > >
> > > Is your DSN setup for "Read Uncommitted"?
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Blake Clemens
> > >
> > > IT Systems Engineer
> > >
> > > Delmarva Millwork Corporation
> > >
> > > (800) 360-2364 x132
> > >
> > > From: vantage@yahoogroups <mailto:vantage%40yahoogroups.com> .com

[mailto:vantage@yahoogroups <mailto:vantage%40yahoogroups.com> .com] On
Behalf

> > > Of mmcwilliams22
> > > Sent: Tuesday, February 02, 2010 12:00 PM
> > > To: vantage@yahoogroups <mailto:vantage%40yahoogroups.com> .com
> > > Subject: [Vantage] Re: Vantage getting slower by the week.
> > >
> > >
> > >
> > >
> > >
> > > 11.5.3
> > >
> > > --- In vantage@yahoogroups <mailto:vantage%40yahoogroups.com> .com

<mailto:vantage%40yahoogroups.com> ,

> > > "Blake Clemens" <blake.clemens@> wrote:
> > > >
> > > > What version of Crystal are you using to design your reports?

I'm
not

> > > > talking about the Crystal version on each client machine.
> > > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Blake Clemens
> > > >
> > > > IT Systems Engineer
> > > >
> > > > Delmarva Millwork Corporation
> > > >
> > > > (800) 360-2364 x132
> > > >
> > > > From: vantage@yahoogroups <mailto:vantage%40yahoogroups.com>