Max # of Records per CSV Upload

We have moved to a support ticketing system and our forums are now closed.

Open Support Ticket

Tagged: csv upload, database, maximum

Author

Posts
January 18, 2015 at 1:25 am #27849

headway
Free User

Post count: 49

I have a massive database that I am looking to get uploaded to my site. The CSV file is correctly formatted and everything works when I try a small sample of data (around 250 entries or so) but fails to complete when I try anything much larger (above 1,000 entries or so).

Is there a maximum limit to how many entries can be added at one time? If so, is there some way to override this?

January 18, 2015 at 2:11 am #27851

Guust
Moderator

Post count: 29970

There is no limitation to the number of listing you can upload via CSV. The only limit would be in case of hundreds of thousands of listings and a very limited execution time in your server.

There is most likely an error somewhere in the file, tripping things up.
It only needs one disallowed character to trip up.

Easiest solution is to split your file in let’s say a 100 records at the time, and keep splitting until you find the error.

You can also post your file here in a private message and we will have a look to see whether there is an obvious error.

http://docs.wpgeodirectory.com/core-general/#csv

January 18, 2015 at 3:32 pm #27877

headway
Free User

Post count: 49

This reply has been marked as private.

January 18, 2015 at 3:34 pm #27878

headway
Free User

Post count: 49

This reply has been marked as private.

January 18, 2015 at 9:37 pm #27902

Guust
Moderator

Post count: 29970

Have you created all your categories already? They have to be existing.
Does that sample file import? I cannot directly see anything wrong.

All the content has to be “plain text”, no special characters can be used, they have to be HTML ready.
http://www.ascii.cl/htmlcodes.htm
I could not see any of them in your sample file.

January 19, 2015 at 8:13 pm #28055

headway
Free User

Post count: 49

In order to better identify the problem, I have changed all of the categories in the CSV file to the same thing, which has definitely been added to the database already.

The file uploads and then the “import data now” button appears, but after it is clicked, the job never finishes and lets me know how many have been successfully imported.

However, some of the listings do go appear to go through and are visible on the site. It just stops midway through.

Thanks for your help.

January 19, 2015 at 8:22 pm #28059

headway
Free User

Post count: 49

Could there be something that needs to be adjusted on my server to allow for a large amount of data to be imported at once?

January 19, 2015 at 8:35 pm #28061

Paolo
Site Admin

Post count: 31206

You can try increasing timeout limit in your server or break down the file in 2 or 3 smaller ones.

Let us know,

Thx

January 20, 2015 at 12:40 am #28104

headway
Free User

Post count: 49

I asked my hosting company to increase the timeout limit, which they did a number of times, but still to no avail.

Here is what was sent to me by the support technician:

I’m increasing the limit to 120s but this is the maximum that I can give you. If this will not help consider contacting the developer of this plugin because it’s not normal a PHP script to run for so much time without any output.

Here is the error from the last time:

[Mon Jan 19 17:24:59 2015] [warn] [client 97.104.173.120] mod_fcgid: read data timeout in 91 seconds, referer: /wp-admin/admin.php?page=geodirectory
[Mon Jan 19 17:24:59 2015] [error] [client 97.104.173.120] Premature end of script headers: admin-ajax.php, referer: /wp-admin/admin.php?page=geodirectory

January 20, 2015 at 12:48 am #28105

Paolo
Site Admin

Post count: 31206

Hi,

it depends how many rows you are trying to add and how long it takes to that server to add each of them.

If the number of row multiplicated by the time needed to enter each row is higher than 120s than it will fail…

What kind of server is hosting your website? Dedicated, VPS or shared?

Have you considered breaking down the CSV files?

Let us know,

January 20, 2015 at 12:59 am #28106

headway
Free User

Post count: 49

I have tried varying numbers of rows and found that I can only add around 270 or so at a time and have everything successfully complete. I have “elite” hosting at trafficplanethosting, but it is shared.

I am trying to add in hundreds of thousands of records, so this is obviously not going work. Is there any other way to get the data in there?

January 20, 2015 at 1:08 am #28107

Paolo
Site Admin

Post count: 31206

Please tell to your hosting provider that: selling a server that in 120 seconds is only able to enter 270 rows from a CSV into a database table is not normal… especially if the name of such server plan is “elite”.

I warmely suggest to change to at least a decent VPS, possibly by someone that doesn’t ripp off their customers.

Thanks

January 20, 2015 at 1:12 am #28108

headway
Free User

Post count: 49

Could this possibly be caused by an error in one of the rows that is tripping things up? Or some other problem with the file?

January 20, 2015 at 1:49 am #28110

headway
Free User

Post count: 49

They have escalated me to level 2 support. Maybe there will be a solution coming from the hosting side.

January 21, 2015 at 1:16 pm #28275

headway
Free User

Post count: 49

Here is the latest message I have received from my hosting company’s support:

Unfortunately the script that is doing the import takes a lot of time to complete without output. This kind of operations need to be done with scheduled tasks, not like that.

I tested also with 20 minutes time out and it didn’t go through, so I recommend you contact the plugin developer to check, and preferably to find a way this process can be started as a cron job.

Please advise what this might mean and how I can resolve this issue. Your directory software is great, but it’s useless to me unless I can upload all of these listings. I may just have to start over from scratch if there isn’t a solution.
Author

Posts