Epicor 9 on MS SQL performance (that old chestnut !)

While it may sound completely counter-intuitive, try turning off the teamed NICs and just use a single NIC card.

 

I have run into situations on more than one occasion where a single NIC is better performance than teamed NICs, and it’s usually in a situation where there isn’t enough data to actually warrant the teamed NICs, if you don’t even use the full bandwidth of a single NIC then I think you just add latency with the teaming because of the extra layer for the data to go through in the system.

 

From: vantage@yahoogroups.com [mailto:vantage@yahoogroups.com] On Behalf Of clive.1972@...
Sent: Tuesday, September 17, 2013 7:55 AM
To: vantage@yahoogroups.com
Subject: [Vantage] RE: Epicor 9 on MS SQL performance (that old chestnut !)

 




To reply to other comments...

 

Network Diagnostics: .16 to .19 for server and .52 to .68 for network, so with tolerances ? (maybe network could be quicker but not without great expenditure, we already use Cisco 3000 series switches which aren't cheap).

 

Config Check: This is happy, nothing to worry about there.

 

AppServer and MS SQL 2008 R2: All of this runs on the same server which should not be an issue considering the hardware we've chosen to use. MS SQL is never stressed and we never run out of AppServers, in fact it's rare to see more than 4 or so being used at a time (compared to V8 where it would almost always be in the teens).

 

NIC: Teamed, 2x 1GB. If you look in Task Manager you need a magnifying glass to see the line for utilisation it's that low.

 

Server / Windows: This is configured as to be expected, High Performance etc.

 

... I think it's this sort of thing that I need...

 

I was able to talk to Nathan Anderson at Epicor support.  He was really helpful, he requested our complete setup and suggested a lot of tuning tips(I sent him 4 gigs worth of files).  I used the performance tuning guide with no noticeable difference but the stuff he sent over made a difference.

 

... I'm trying through my CAM to get hold of a guy in the UK called Neil McLachlan, he's the one in the UK who'll know exactly what needs to be done, as I know that like the Epicor guy above there are priceless nuggest of info in his head.

 

Thanks.



--- In vantage@yahoogroups.com, <don.n.doan@...> wrote:

 I was able to talk to Nathan Anderson at Epicor support.  He was really helpful, he requested our complete setup and suggested a lot of tuning tips(I sent him 4 gigs worth of files).  I used the performance tuning guide with no noticeable difference but the stuff he sent over made a difference.  We're a little under hardware recommendations for 701 so we beefed up some of our memory, but our processors and drives are a little dated.  Together with the tips from support and the performance tuning sessions at insights we're a lot closer to what we want performance wise without changing too much hardware.  I also went through customizations that our consulting company put together for us and cleaned up a lot of code which sped up our sales orders/projects/scheduling by more than double. 

 

Look up those documents from insights if you can find them and contact support to check all of your settings.

Also try posting your recorded times for processes.  Maybe some people can compare to see if there's a big difference or you are in the same boat as everyone else.

 

If your company has the cash you can also buy i think 50 hours from Epicor just for tuning.  I don't think thats the way to go though unless its really that bad.

 


--- In vantage@yahoogroups.com, <vantage@yahoogroups.com> wrote:

I don’t see it as a progress or sql issue. it is tweaking your environment to get the best you can out of it. All setups can be faster or slower depending on how much time you spend on them. hardware cost is not going to solve network or setup issues.

 

What are your Network Diagnostics numbers from the Performance diagnostic Tools? This will give you a benchmark to work from and us an idea of how good or bad your speed is.

 

Mine are consistent at .18 - .23 server and .31 to .33 network.  with those I have most happy and some not happy users.

 

Have you run the config check and addressed any issues?

 

How do you have the SQL server connected to the App server. Are you using teamed 1 GB or 10 GB Nic cards? Direct Ethernet connections either copper or fiber?

I have 4 1 GB ports teamed thru a dedicated Cisco switch for our Epicor servers.

 

Are all of the power performance settings done on both servers? in the bios and the OS.

This made a world of difference for us.

 

We are currently changing any clients with a 100 nic up to a gig port to squeeze more out of them.

 

I look at the activity monitor for long running queries. I got a script from support to detect locking and blocking issues in sql. This has helped me identify reports or processes that are hogging resources.  I got it since I was running out of app servers, but it has yielded a lot of correctable issues.

 

Greg Payne

 

 

From: vantage@yahoogroups.com [mailto:vantage@yahoogroups.com] On Behalf Of clive.1972@...
Sent: Friday, September 13, 2013 11:33 AM
To: vantage@yahoogroups.com
Subject: [Vantage] Epicor 9 on MS SQL performance (that old chestnut !)

 

 

So, where I currently work we went from Vantage 8.03.407 on Progress over Epicor 9.05.700C MS SQL just under three months ago (it took a few years to get there). I knew from the start that even with the montrous server hardware we have in place performance of the Epicor application would degrade a little (Progress to MS SQL) but our users are finding it incredibly slow and at times very painful to use.

 

As mentioned above the server hardware is total overkill, there are no issues there in that it never breaks into a sweat at all, but using the Epicor 9 client (quotes, orders, jobs etc) is very noticeably slower.

 

All performance tuning guides have been read, and applied if required, and I've used all the SQL skills I have but I don't believe that there's anything MS SQL wise to worry about (server has 64 GB RAM, almost all of the DB is in RAM, SQL transactions are lightning fast). I'm getting stumped as to why E9 is performing like a pig.

 

It may well just be it is what it is, we've already achieved the best perfoamnce possible. Has anyone here gone from Progress to MS SQL (with aroudn 55 concurrent users), had a server worth well over $15k, tuned the life out of it and still had crap performance ?. If so what did you do ?, eventualy manage to get it running better or just accept it ?.

 

Thanks.

 


CONFIDENTIALITY NOTICE

The information contained in this communication, including attachments, is privileged and confidential. It is intended only for the exclusive use of the addressee. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us at 727-578-6280 and immediately delete the communication.

"This (document/presentation) may contain technical data as defined in the International Traffic In Arms Regulations (ITAR) 22 CFR 120.10. Export of this material is restricted by the Arms Export Control Act (22 U.S.C. 2751 et seq.) and may not be exported to foreign persons without prior approval form the U.S. Department of State."




So, where I currently work we went from Vantage 8.03.407 on Progress over Epicor 9.05.700C MS SQL just under three months ago (it took a few years to get there). I knew from the start that even with the montrous server hardware we have in place performance of the Epicor application would degrade a little (Progress to MS SQL) but our users are finding it incredibly slow and at times very painful to use.

 

As mentioned above the server hardware is total overkill, there are no issues there in that it never breaks into a sweat at all, but using the Epicor 9 client (quotes, orders, jobs etc) is very noticeably slower.

 

All performance tuning guides have been read, and applied if required, and I've used all the SQL skills I have but I don't believe that there's anything MS SQL wise to worry about (server has 64 GB RAM, almost all of the DB is in RAM, SQL transactions are lightning fast). I'm getting stumped as to why E9 is performing like a pig.

 

It may well just be it is what it is, we've already achieved the best perfoamnce possible. Has anyone here gone from Progress to MS SQL (with aroudn 55 concurrent users), had a server worth well over $15k, tuned the life out of it and still had crap performance ?. If so what did you do ?, eventualy manage to get it running better or just accept it ?.

 

Thanks.

There are many possible variable (unfortunately!).  It would be helpful to understand how bad the performance is in measurement terms. This would let us give you an idea if it is as good as it gets. Also, where is it bad – in opening screens, processing reports, both, something else?

 

When chasing down bad performance for users opening screens (for example), I found it important to actually time it. I used a stopwatch app on my phone to measure from click to open, with results like these (the numbers are avg seconds for five openings):

 

3Â Job Entry

4Â Order Entry

1Â Cash Receipt

3Â Invoice Entry

2Â Part Tracker

4Â Job Tracker

3Â Receipt Entry

4Â Cust Tracker

4Â Order Tracker

 

This gave me a baseline to know if changes made any real difference.

 

Some possible considerations:

- How many appservers are you running? Too few can slow things down.

- Is it a virtual environment? The listserv has lots of discussions that might help.

- Putting the report server on a different box can speed things up.

- Depending on where users are located, using terminal services can help, although that adds one more thing to manage as well as additional cost.

- Are you running any of the 3rd party bolt-on apps? Sometimes they need to be put on a different server to increase performance.

 

These are just a few thoughts. Hope it helps.

 

Regards,

 

Jeff

 

 

Jeff Greenaway

Consulting CIO/ERP Consultant

jeffgreenaway@...

www.cio-ondemand.com

 

From: vantage@yahoogroups.com [mailto:vantage@yahoogroups.com] On Behalf Of clive.1972@...
Sent: Friday, September 13, 2013 8:33 AM
To: vantage@yahoogroups.com
Subject: [Vantage] Epicor 9 on MS SQL performance (that old chestnut !)

 

 

So, where I currently work we went from Vantage 8.03.407 on Progress over Epicor 9.05.700C MS SQL just under three months ago (it took a few years to get there). I knew from the start that even with the montrous server hardware we have in place performance of the Epicor application would degrade a little (Progress to MS SQL) but our users are finding it incredibly slow and at times very painful to use.

 

As mentioned above the server hardware is total overkill, there are no issues there in that it never breaks into a sweat at all, but using the Epicor 9 client (quotes, orders, jobs etc) is very noticeably slower.

 

All performance tuning guides have been read, and applied if required, and I've used all the SQL skills I have but I don't believe that there's anything MS SQL wise to worry about (server has 64 GB RAM, almost all of the DB is in RAM, SQL transactions are lightning fast). I'm getting stumped as to why E9 is performing like a pig.

 

It may well just be it is what it is, we've already achieved the best perfoamnce possible. Has anyone here gone from Progress to MS SQL (with aroudn 55 concurrent users), had a server worth well over $15k, tuned the life out of it and still had crap performance ?. If so what did you do ?, eventualy manage to get it running better or just accept it ?.

 

Thanks.

Hi Clive,

 

If you are using BPM’s take a look at them.  That was a major issue for us, as we had checked customizations.  You may want to test, by turning them all off on a test system and see if the performance is better.  You may have no choice but to test on a live system, but we have caught issues with BPM’s where updating fields in other tables were causing delays on the system performance at the client level, but the servers were not having any performance issues.  Also, are the BPM’s doing heavy logging? Sometimes logging is specified during testing of BPM’s and the programmer might forget to remove it.  This will cause delays and the need for the system to be rebooted often (more than once a week).

 

Hope this helps…

 

Regards,

Sigfredo.

 

From: vantage@yahoogroups.com [mailto:vantage@yahoogroups.com] On Behalf Of clive.1972@...
Sent: Friday, September 13, 2013 11:33 AM
To: vantage@yahoogroups.com
Subject: [Vantage] Epicor 9 on MS SQL performance (that old chestnut !)

 

 

So, where I currently work we went from Vantage 8.03.407 on Progress over Epicor 9.05.700C MS SQL just under three months ago (it took a few years to get there). I knew from the start that even with the montrous server hardware we have in place performance of the Epicor application would degrade a little (Progress to MS SQL) but our users are finding it incredibly slow and at times very painful to use.

 

As mentioned above the server hardware is total overkill, there are no issues there in that it never breaks into a sweat at all, but using the Epicor 9 client (quotes, orders, jobs etc) is very noticeably slower.

 

All performance tuning guides have been read, and applied if required, and I've used all the SQL skills I have but I don't believe that there's anything MS SQL wise to worry about (server has 64 GB RAM, almost all of the DB is in RAM, SQL transactions are lightning fast). I'm getting stumped as to why E9 is performing like a pig.

 

It may well just be it is what it is, we've already achieved the best perfoamnce possible. Has anyone here gone from Progress to MS SQL (with aroudn 55 concurrent users), had a server worth well over $15k, tuned the life out of it and still had crap performance ?. If so what did you do ?, eventualy manage to get it running better or just accept it ?.

 

Thanks.

I don’t see it as a progress or sql issue. it is tweaking your environment to get the best you can out of it. All setups can be faster or slower depending on how much time you spend on them. hardware cost is not going to solve network or setup issues.

 

What are your Network Diagnostics numbers from the Performance diagnostic Tools? This will give you a benchmark to work from and us an idea of how good or bad your speed is.

 

Mine are consistent at .18 - .23 server and .31 to .33 network.  with those I have most happy and some not happy users.

 

Have you run the config check and addressed any issues?

 

How do you have the SQL server connected to the App server. Are you using teamed 1 GB or 10 GB Nic cards? Direct Ethernet connections either copper or fiber?

I have 4 1 GB ports teamed thru a dedicated Cisco switch for our Epicor servers.

 

Are all of the power performance settings done on both servers? in the bios and the OS.

This made a world of difference for us.

 

We are currently changing any clients with a 100 nic up to a gig port to squeeze more out of them.

 

I look at the activity monitor for long running queries. I got a script from support to detect locking and blocking issues in sql. This has helped me identify reports or processes that are hogging resources.  I got it since I was running out of app servers, but it has yielded a lot of correctable issues.

 

Greg Payne

 

 

From: vantage@yahoogroups.com [mailto:vantage@yahoogroups.com] On Behalf Of clive.1972@...
Sent: Friday, September 13, 2013 11:33 AM
To: vantage@yahoogroups.com
Subject: [Vantage] Epicor 9 on MS SQL performance (that old chestnut !)

 

 

So, where I currently work we went from Vantage 8.03.407 on Progress over Epicor 9.05.700C MS SQL just under three months ago (it took a few years to get there). I knew from the start that even with the montrous server hardware we have in place performance of the Epicor application would degrade a little (Progress to MS SQL) but our users are finding it incredibly slow and at times very painful to use.

 

As mentioned above the server hardware is total overkill, there are no issues there in that it never breaks into a sweat at all, but using the Epicor 9 client (quotes, orders, jobs etc) is very noticeably slower.

 

All performance tuning guides have been read, and applied if required, and I've used all the SQL skills I have but I don't believe that there's anything MS SQL wise to worry about (server has 64 GB RAM, almost all of the DB is in RAM, SQL transactions are lightning fast). I'm getting stumped as to why E9 is performing like a pig.

 

It may well just be it is what it is, we've already achieved the best perfoamnce possible. Has anyone here gone from Progress to MS SQL (with aroudn 55 concurrent users), had a server worth well over $15k, tuned the life out of it and still had crap performance ?. If so what did you do ?, eventualy manage to get it running better or just accept it ?.

 

Thanks.



CONFIDENTIALITY NOTICE

The information contained in this communication, including attachments, is privileged and confidential. It is intended only for the exclusive use of the addressee. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us at 727-578-6280 and immediately delete the communication.

"This (document/presentation) may contain technical data as defined in the International Traffic In Arms Regulations (ITAR) 22 CFR 120.10. Export of this material is restricted by the Arms Export Control Act (22 U.S.C. 2751 et seq.) and may not be exported to foreign persons without prior approval form the U.S. Department of State."

 I was able to talk to Nathan Anderson at Epicor support.  He was really helpful, he requested our complete setup and suggested a lot of tuning tips(I sent him 4 gigs worth of files).  I used the performance tuning guide with no noticeable difference but the stuff he sent over made a difference.  We're a little under hardware recommendations for 701 so we beefed up some of our memory, but our processors and drives are a little dated.  Together with the tips from support and the performance tuning sessions at insights we're a lot closer to what we want performance wise without changing too much hardware.  I also went through customizations that our consulting company put together for us and cleaned up a lot of code which sped up our sales orders/projects/scheduling by more than double. 


Look up those documents from insights if you can find them and contact support to check all of your settings.

Also try posting your recorded times for processes.  Maybe some people can compare to see if there's a big difference or you are in the same boat as everyone else.


If your company has the cash you can also buy i think 50 hours from Epicor just for tuning.  I don't think thats the way to go though unless its really that bad.



--- In vantage@yahoogroups.com, <vantage@yahoogroups.com> wrote:

I don’t see it as a progress or sql issue. it is tweaking your environment to get the best you can out of it. All setups can be faster or slower depending on how much time you spend on them. hardware cost is not going to solve network or setup issues.

 

What are your Network Diagnostics numbers from the Performance diagnostic Tools? This will give you a benchmark to work from and us an idea of how good or bad your speed is.

 

Mine are consistent at .18 - .23 server and .31 to .33 network.  with those I have most happy and some not happy users.

 

Have you run the config check and addressed any issues?

 

How do you have the SQL server connected to the App server. Are you using teamed 1 GB or 10 GB Nic cards? Direct Ethernet connections either copper or fiber?

I have 4 1 GB ports teamed thru a dedicated Cisco switch for our Epicor servers.

 

Are all of the power performance settings done on both servers? in the bios and the OS.

This made a world of difference for us.

 

We are currently changing any clients with a 100 nic up to a gig port to squeeze more out of them.

 

I look at the activity monitor for long running queries. I got a script from support to detect locking and blocking issues in sql. This has helped me identify reports or processes that are hogging resources.  I got it since I was running out of app servers, but it has yielded a lot of correctable issues.

 

Greg Payne

 

 

From: vantage@yahoogroups.com [mailto:vantage@yahoogroups.com] On Behalf Of clive.1972@...
Sent: Friday, September 13, 2013 11:33 AM
To: vantage@yahoogroups.com
Subject: [Vantage] Epicor 9 on MS SQL performance (that old chestnut !)

 

 

So, where I currently work we went from Vantage 8.03.407 on Progress over Epicor 9.05.700C MS SQL just under three months ago (it took a few years to get there). I knew from the start that even with the montrous server hardware we have in place performance of the Epicor application would degrade a little (Progress to MS SQL) but our users are finding it incredibly slow and at times very painful to use.

 

As mentioned above the server hardware is total overkill, there are no issues there in that it never breaks into a sweat at all, but using the Epicor 9 client (quotes, orders, jobs etc) is very noticeably slower.

 

All performance tuning guides have been read, and applied if required, and I've used all the SQL skills I have but I don't believe that there's anything MS SQL wise to worry about (server has 64 GB RAM, almost all of the DB is in RAM, SQL transactions are lightning fast). I'm getting stumped as to why E9 is performing like a pig.

 

It may well just be it is what it is, we've already achieved the best perfoamnce possible. Has anyone here gone from Progress to MS SQL (with aroudn 55 concurrent users), had a server worth well over $15k, tuned the life out of it and still had crap performance ?. If so what did you do ?, eventualy manage to get it running better or just accept it ?.

 

Thanks.



CONFIDENTIALITY NOTICE

The information contained in this communication, including attachments, is privileged and confidential. It is intended only for the exclusive use of the addressee. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us at 727-578-6280 and immediately delete the communication.

"This (document/presentation) may contain technical data as defined in the International Traffic In Arms Regulations (ITAR) 22 CFR 120.10. Export of this material is restricted by the Arms Export Control Act (22 U.S.C. 2751 et seq.) and may not be exported to foreign persons without prior approval form the U.S. Department of State."

To reply to other comments...


Network Diagnostics: .16 to .19 for server and .52 to .68 for network, so with tolerances ? (maybe network could be quicker but not without great expenditure, we already use Cisco 3000 series switches which aren't cheap).


Config Check: This is happy, nothing to worry about there.


AppServer and MS SQL 2008 R2: All of this runs on the same server which should not be an issue considering the hardware we've chosen to use. MS SQL is never stressed and we never run out of AppServers, in fact it's rare to see more than 4 or so being used at a time (compared to V8 where it would almost always be in the teens).


NIC: Teamed, 2x 1GB. If you look in Task Manager you need a magnifying glass to see the line for utilisation it's that low.


Server / Windows: This is configured as to be expected, High Performance etc.


... I think it's this sort of thing that I need...


I was able to talk to Nathan Anderson at Epicor support.  He was really helpful, he requested our complete setup and suggested a lot of tuning tips(I sent him 4 gigs worth of files).  I used the performance tuning guide with no noticeable difference but the stuff he sent over made a difference.


... I'm trying through my CAM to get hold of a guy in the UK called Neil McLachlan, he's the one in the UK who'll know exactly what needs to be done, as I know that like the Epicor guy above there are priceless nuggest of info in his head.


Thanks.



--- In vantage@yahoogroups.com, <don.n.doan@...> wrote:

 I was able to talk to Nathan Anderson at Epicor support.  He was really helpful, he requested our complete setup and suggested a lot of tuning tips(I sent him 4 gigs worth of files).  I used the performance tuning guide with no noticeable difference but the stuff he sent over made a difference.  We're a little under hardware recommendations for 701 so we beefed up some of our memory, but our processors and drives are a little dated.  Together with the tips from support and the performance tuning sessions at insights we're a lot closer to what we want performance wise without changing too much hardware.  I also went through customizations that our consulting company put together for us and cleaned up a lot of code which sped up our sales orders/projects/scheduling by more than double. 


Look up those documents from insights if you can find them and contact support to check all of your settings.

Also try posting your recorded times for processes.  Maybe some people can compare to see if there's a big difference or you are in the same boat as everyone else.


If your company has the cash you can also buy i think 50 hours from Epicor just for tuning.  I don't think thats the way to go though unless its really that bad.



--- In vantage@yahoogroups.com, <vantage@yahoogroups.com> wrote:

I don’t see it as a progress or sql issue. it is tweaking your environment to get the best you can out of it. All setups can be faster or slower depending on how much time you spend on them. hardware cost is not going to solve network or setup issues.

 

What are your Network Diagnostics numbers from the Performance diagnostic Tools? This will give you a benchmark to work from and us an idea of how good or bad your speed is.

 

Mine are consistent at .18 - .23 server and .31 to .33 network.  with those I have most happy and some not happy users.

 

Have you run the config check and addressed any issues?

 

How do you have the SQL server connected to the App server. Are you using teamed 1 GB or 10 GB Nic cards? Direct Ethernet connections either copper or fiber?

I have 4 1 GB ports teamed thru a dedicated Cisco switch for our Epicor servers.

 

Are all of the power performance settings done on both servers? in the bios and the OS.

This made a world of difference for us.

 

We are currently changing any clients with a 100 nic up to a gig port to squeeze more out of them.

 

I look at the activity monitor for long running queries. I got a script from support to detect locking and blocking issues in sql. This has helped me identify reports or processes that are hogging resources.  I got it since I was running out of app servers, but it has yielded a lot of correctable issues.

 

Greg Payne

 

 

From: vantage@yahoogroups.com [mailto:vantage@yahoogroups.com] On Behalf Of clive.1972@...
Sent: Friday, September 13, 2013 11:33 AM
To: vantage@yahoogroups.com
Subject: [Vantage] Epicor 9 on MS SQL performance (that old chestnut !)

 

 

So, where I currently work we went from Vantage 8.03.407 on Progress over Epicor 9.05.700C MS SQL just under three months ago (it took a few years to get there). I knew from the start that even with the montrous server hardware we have in place performance of the Epicor application would degrade a little (Progress to MS SQL) but our users are finding it incredibly slow and at times very painful to use.

 

As mentioned above the server hardware is total overkill, there are no issues there in that it never breaks into a sweat at all, but using the Epicor 9 client (quotes, orders, jobs etc) is very noticeably slower.

 

All performance tuning guides have been read, and applied if required, and I've used all the SQL skills I have but I don't believe that there's anything MS SQL wise to worry about (server has 64 GB RAM, almost all of the DB is in RAM, SQL transactions are lightning fast). I'm getting stumped as to why E9 is performing like a pig.

 

It may well just be it is what it is, we've already achieved the best perfoamnce possible. Has anyone here gone from Progress to MS SQL (with aroudn 55 concurrent users), had a server worth well over $15k, tuned the life out of it and still had crap performance ?. If so what did you do ?, eventualy manage to get it running better or just accept it ?.

 

Thanks.



CONFIDENTIALITY NOTICE

The information contained in this communication, including attachments, is privileged and confidential. It is intended only for the exclusive use of the addressee. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us at 727-578-6280 and immediately delete the communication.

"This (document/presentation) may contain technical data as defined in the International Traffic In Arms Regulations (ITAR) 22 CFR 120.10. Export of this material is restricted by the Arms Export Control Act (22 U.S.C. 2751 et seq.) and may not be exported to foreign persons without prior approval form the U.S. Department of State."