DMT max rows question

Hello,
Just a quick question that I hope someone can answer. What is the maximum rows that the DMT GUI can handle before it throws an error? I do know it’s less the 980k.

I don’t know the answer, but I would normally suggest breaking up your input table into multiple smaller tables just for faster processing. DMT can import more than one at a time. Try breaking up into 50,000 per, which would make 20 tables… but would be loaded 10-20 times faster if loaded all at once (multi-threaded load).

2 Likes

Great idea Tim thanks. I forgot about multi-thread loading.

Multii-threading DMT in action :slight_smile:

The real question is will it really take 1/17th of the time? When I tried the single file it was going to take about 13hrs… Yikes!.

It usually starts to slow down once you get so many tabs going. I think it’s about 6 or 8 for us, but I would guess it depends on your server setup.

I’m assuming that you found the tool to split the file for you?

splitting files is not something I do on a regular basis and this was a test, so I rolled it myself. I’ll probably use powershell for the final solution. Feel free to post your suggested tool, it will always help someone perusing the posts in future.

It’s right down in the corner, you just tell it which file to split and how many files you want. It takes the file name and duplicates it with a 1 2 3 etc, on the end in the same folder the original file it. It works pretty slick and saves a lot of time.

image

2 Likes

WOW! and Thanks.
There you go, pays to read the documentation a bit more clearly. Thanks that’t the tip of the week. @edge had a hand in that handy feature no doubt.

1 Like

and if you have to do it a lot, you can set up playlists to load all of the tabs for you. If you always split to say 10 files, you can make a play list to load and play those 10 files.

I used the playlist for that test. I actually create the playlist with 1 row, then edited the CSV file. Would have been nice to have a copy row button…Hmmm there’s another one for the enhancements list

3 Likes

Just for some geekiness. All imports started within 15 minutes of each other. Compared from the estimate of the single file 12 hrs or so, a big time saving.

File hours min rpm rows
1 4 10 200 49999
2 4 11 199 49999
3 4 13 197 49998
4 4 9 201 49999
5 4 13 197 49999
6 4 14 197 49999
7 4 9 201 49999
8 4 13 198 49999
9 4 12 199 49999
10 4 13 197 49999
11 4 11 199 49999
12 4 13 197 49999
13 4 11 199 49999
14 4 12 198 49999
15 4 12 199 50000
16 4 12 198 49999
17 2 23 190 27185
1 Like

OK… so now back to the original question… what is the maximum number of rows?.. It really doesn’t matter because large files are just not that practical. It is always better to do this in small chunks.
So glad that this worked out for you!

I have experienced errors such as Cannot access a disposed object.
Object name: ‘System.ServiceModel.Channels.ServiceChannel’. When using the multiple file method. But this might be just a result of previous errors.

In our latest test with open sales orders I split the file into 12 files and ran them all at once. reduced the time from 7 hrs to 2, but as I said I got these errors. Running a single file I don’t get them.