Max # of Records per CSV Upload

We have moved to a support ticketing system and our forums are now closed.

Open Support Ticket

Tagged: csv upload, database, maximum

Author

Posts
January 21, 2015 at 5:19 pm #28332

Paolo
Site Admin

Post count: 31206

Hi Headaway,

the code behind the CSV upload script is something that we developed over 3 years ago and it has been working perfectly ever since for thousands of clients.

However, the larger the file, the larger is the CPU and the RAM consumption. A shared server provides very little CPU and RAM resources, because most of the time you share the server with thousands of other websites, making it a poor choice for any website that aims to receive any traffic in the first place.

The cause of the problem here is in the server not the script.

If you try to test on a decent hosting provider, there are plenty giving you 1 month free like cloudways.com, you will immediately notice that CSV upload works flowlessly.

In case your priority is working on a low level shared hosting, than yes GeoDirectory may not be the right tool for you.

Thank you

January 21, 2015 at 8:53 pm #28379

headway
Free User

Post count: 49

Another message from server support:

I can assure you that our “low level shared hosting” is one of the fastest in the world.

..this plugin can’t work correctly because it uses enormous amount of time and resources. It’s not serious to say that the reason is in the server because this script needs several gigabytes of RAM and few CPU cores to run. Such server including the support (you will need sys admin to administrate it) will cost you several hundred dollars per month. Not to mention that if the script is written correctly it can import that information for several minutes without maximum execution time to expire using the resources that it has.

Is the CSV importer supposed to need several gigabytes of RAM and few CPU cores to run? If not, then what could be causing this problem? Does it have something to do with the CSV file? A recent update to the core? Other plugins that could be causing a conflict? Please advise.

January 21, 2015 at 10:20 pm #28399

Guust
Moderator

Post count: 29970

Headway, I personally have uploaded CSV files for some sites that are hosted on below $10 a month shared servers, without any problems. None of these files would have had more than 1000 listings, but some had at least a few hundred.
And the only reason you would need sys admin if they restrict your account below the standards that most other hosts use.
Just my 2 cents worth.

January 21, 2015 at 10:33 pm #28403

Paolo
Site Admin

Post count: 31206

Hi Again,

I’ve just tested to double check and I do see some problem with your file.

Region should be Massachusetts and not MA (google maps API requires the region to be the full name), but even changing that it doesn’t appear to allow to upload all +2000 listings.

I’ve asked to Stiofan to investigate.

I should have done that after your 1st post and I apologize for this.

We’ll keep you posted.

Thx

January 23, 2015 at 1:50 pm #28617

headway
Free User

Post count: 49

Thanks. Please let me know if there is any additional information you need me to provide.

January 23, 2015 at 2:02 pm #28620

Guust
Moderator

Post count: 29970

Having MA instead of Massachusetts should not stop the upload though.
It will create a region called MA, and when adding other listings from the Google API, you might get a duplicate region in the database (MA and Massachusetts), but I have never known it to interfere with the upload.

January 23, 2015 at 3:14 pm #28623

Stiofan O’Connor
Site Admin

Post count: 22956

Hi,

I have added this as a task to improve, it could be the memory or the processor, there seems to be a bottle neck somewhere i will think of a better solution to this running all at the one time and implement it.

I will keep u posted on progress.

Thanks,

Stiofan

January 25, 2015 at 8:20 pm #28798

headway
Free User

Post count: 49

Thank you. Would this be something that would be released in a new update to the core plugin?

January 26, 2015 at 5:25 pm #28850

Paolo
Site Admin

Post count: 31206

Yes it would.

February 10, 2015 at 4:06 pm #30565

headway
Free User

Post count: 49

The new code works great! The progress bar is a nice touch.

I have been able to import up to 5,000 records at a time now. I am sure I would be able to do more, but the “Select & Upload CSV” part of the process seems to only allow me to use files of less than 1.5 MB. Is there any way for me to upload even larger file sizes?

February 10, 2015 at 4:18 pm #30568

Paolo
Site Admin

Post count: 31206

Hi,

that could be due to your php setting for uploads. Please see http://stackoverflow.com/questions/2184513/php-change-the-maximum-upload-file-size

Let us know,

Thx

February 10, 2015 at 4:19 pm #30569

Stiofan O’Connor
Site Admin

Post count: 22956

if that is not the problem u could try posting the file in a zip file here and we can test.

Stiofan

February 11, 2015 at 2:06 pm #30679

headway
Free User

Post count: 49

This reply has been marked as private.

February 11, 2015 at 3:49 pm #30690

Paolo
Site Admin

Post count: 31206

Hi Headway,

tested and I can confirm is a bug. Flagged it for developers, who will fix asap.

We’ll report here once fixed.

Thx again for your patience!

February 17, 2015 at 5:09 pm #31180

Stiofan O’Connor
Site Admin

Post count: 22956

this has been fixed and will be in the next release.
Author

Posts