Max # of Records per CSV Upload

This topic contains 49 replies, has 4 voices, and was last updated by  headway 9 years, 8 months ago.

We have moved to a support ticketing system and our forums are now closed.

Open Support Ticket

Tagged: , ,

  • Author
    Posts
  • January 21, 2015 at 5:19 pm #28332

    Paolo
    Site Admin
    Post count: 31206

    Hi Headaway,

    the code behind the CSV upload script is something that we developed over 3 years ago and it has been working perfectly ever since for thousands of clients.

    However, the larger the file, the larger is the CPU and the RAM consumption. A shared server provides very little CPU and RAM resources, because most of the time you share the server with thousands of other websites, making it a poor choice for any website that aims to receive any traffic in the first place.

    The cause of the problem here is in the server not the script.

    If you try to test on a decent hosting provider, there are plenty giving you 1 month free like cloudways.com, you will immediately notice that CSV upload works flowlessly.

    In case your priority is working on a low level shared hosting, than yes GeoDirectory may not be the right tool for you.

    Thank you

    January 21, 2015 at 8:53 pm #28379

    headway
    Free User
    Post count: 49

    Another message from server support:

    I can assure you that our “low level shared hosting” is one of the fastest in the world.

    ..this plugin can’t work correctly because it uses enormous amount of time and resources. It’s not serious to say that the reason is in the server because this script needs several gigabytes of RAM and few CPU cores to run. Such server including the support (you will need sys admin to administrate it) will cost you several hundred dollars per month. Not to mention that if the script is written correctly it can import that information for several minutes without maximum execution time to expire using the resources that it has.

    Is the CSV importer supposed to need several gigabytes of RAM and few CPU cores to run? If not, then what could be causing this problem? Does it have something to do with the CSV file? A recent update to the core? Other plugins that could be causing a conflict? Please advise.

    January 21, 2015 at 10:20 pm #28399

    Guust
    Moderator
    Post count: 29970

    Headway, I personally have uploaded CSV files for some sites that are hosted on below $10 a month shared servers, without any problems. None of these files would have had more than 1000 listings, but some had at least a few hundred.
    And the only reason you would need sys admin if they restrict your account below the standards that most other hosts use.
    Just my 2 cents worth.

    January 21, 2015 at 10:33 pm #28403

    Paolo
    Site Admin
    Post count: 31206

    Hi Again,

    I’ve just tested to double check and I do see some problem with your file.

    Region should be Massachusetts and not MA (google maps API requires the region to be the full name), but even changing that it doesn’t appear to allow to upload all +2000 listings.

    I’ve asked to Stiofan to investigate.

    I should have done that after your 1st post and I apologize for this.

    We’ll keep you posted.

    Thx

    January 23, 2015 at 1:50 pm #28617

    headway
    Free User
    Post count: 49

    Thanks. Please let me know if there is any additional information you need me to provide.

    January 23, 2015 at 2:02 pm #28620

    Guust
    Moderator
    Post count: 29970

    Having MA instead of Massachusetts should not stop the upload though.
    It will create a region called MA, and when adding other listings from the Google API, you might get a duplicate region in the database (MA and Massachusetts), but I have never known it to interfere with the upload.

    January 23, 2015 at 3:14 pm #28623

    Stiofan O’Connor
    Site Admin
    Post count: 22956

    Hi,

    I have added this as a task to improve, it could be the memory or the processor, there seems to be a bottle neck somewhere i will think of a better solution to this running all at the one time and implement it.

    I will keep u posted on progress.

    Thanks,

    Stiofan

    January 25, 2015 at 8:20 pm #28798

    headway
    Free User
    Post count: 49

    Thank you. Would this be something that would be released in a new update to the core plugin?

    January 26, 2015 at 5:25 pm #28850

    Paolo
    Site Admin
    Post count: 31206

    Yes it would.

    February 10, 2015 at 4:06 pm #30565

    headway
    Free User
    Post count: 49

    The new code works great! The progress bar is a nice touch.

    I have been able to import up to 5,000 records at a time now. I am sure I would be able to do more, but the “Select & Upload CSV” part of the process seems to only allow me to use files of less than 1.5 MB. Is there any way for me to upload even larger file sizes?

    February 10, 2015 at 4:18 pm #30568

    Paolo
    Site Admin
    Post count: 31206

    Hi,

    that could be due to your php setting for uploads. Please see http://stackoverflow.com/questions/2184513/php-change-the-maximum-upload-file-size

    Let us know,

    Thx

    February 10, 2015 at 4:19 pm #30569

    Stiofan O’Connor
    Site Admin
    Post count: 22956

    if that is not the problem u could try posting the file in a zip file here and we can test.

    Stiofan

    February 11, 2015 at 2:06 pm #30679

    headway
    Free User
    Post count: 49
    This reply has been marked as private.
    February 11, 2015 at 3:49 pm #30690

    Paolo
    Site Admin
    Post count: 31206

    Hi Headway,

    tested and I can confirm is a bug. Flagged it for developers, who will fix asap.

    We’ll report here once fixed.

    Thx again for your patience!

    February 17, 2015 at 5:09 pm #31180

    Stiofan O’Connor
    Site Admin
    Post count: 22956

    this has been fixed and will be in the next release.

  • Author
    Posts
Viewing 15 posts - 16 through 30 (of 50 total)

We have moved to a support ticketing system and our forums are now closed.

Open Support Ticket
20% Discount Offer
Hurry! Get your 20% discount before it expires. Get 20% Discount