Import stops

This topic contains 14 replies, has 3 voices, and was last updated by  Paolo 5 years, 8 months ago.

We have moved to a support ticketing system and our forums are now closed.

Open Support Ticket

Tagged: , , ,

  • Author
    Posts
  • August 15, 2018 at 7:14 pm #442550

    Iain Brewster
    Expired Member
    Post count: 74

    Hi,
    I am trying to import around 290k records.

    I have started doing this in sprints of 10k but the import will stop part way through.
    It doesn’t give the option to continue import and if I start import again then I get duplicate records.

    The import is quite slow so starting over again is becoming quite frustrating.

    Is there something I am doing wrong or is there a faster way to do this ?

    Thanks

    August 15, 2018 at 10:08 pm #442581

    Paolo
    Site Admin
    Post count: 31206

    Hi,

    What kind of server are we talking about? Because if it is very slow and getting stuck, most likely it is not fast enough and it is timing out.

    Try increasing the PHP execution timeout.

    Another option is to try to break the files into smaller lists (5000 for example).

    The only faster way to do this, would be to get a faster server. (for 290k listings, you should get at least a decent dedicated server)

    Let us know how it goes,

    Thank you

    August 18, 2018 at 12:06 pm #442966

    Iain Brewster
    Expired Member
    Post count: 74

    Thanks Paolo. I do need a dedicated server but am not in a position for that yet and my PHP is currently at max.
    I had a thought though.

    Can I give the listings their post_ids before upload and that way if it stopped and I restarted there would not be duplicate entries?

    August 18, 2018 at 12:28 pm #442971

    Alex Rollin
    Moderator
    Post count: 27815

    You could import X dummy listings and get X post ids, which you could then merge into your sheet.

    August 18, 2018 at 3:41 pm #442986

    Iain Brewster
    Expired Member
    Post count: 74

    Could I do that if the issue is importing and exporting taking too long considering the volume of records I have?

    Is there another way to prevent duplicate records in import other than post_id

    For example don’t allow listings with the same address to be imported?

    August 18, 2018 at 4:24 pm #442988

    Paolo
    Site Admin
    Post count: 31206

    No Ian,

    similar operations would slow down the import process even more.

    I believe the only way to import so many listings on an underpowered server, is to do a couple of tests to verify on average how many listings it is able to import.

    Once you know that number, delete anything you have imported so far and break down the files to a slightly lower number than the average you were able to import during the tests. I

    It is faster to do 100+ successful imports, than doing what you are trying to do now.

    I know it is a tedious job, but that’s the price to pay for using a server that is not fit for purpose.

    August 18, 2018 at 4:24 pm #442989

    Paolo
    Site Admin
    Post count: 31206

    No Ian,

    similar operations would slow down the import process even more.

    I believe the only way to import so many listings on an underpowered server, is to do a couple of tests to verify on average how many listings it is able to import.

    Once you know that number, delete anything you have imported so far and break down the files to a slightly lower number than the average you were able to import during the tests. I

    It is faster to do 100+ successful imports, than doing what you are trying to do now.

    I know it is a tedious job, but that’s the price to pay for using a server that is not fit for purpose.

    August 28, 2018 at 6:55 am #444275

    Iain Brewster
    Expired Member
    Post count: 74

    Thanks paolo.i have tried reducing the import volume to smaller amounts but the import stopped after 600 records then when trying the same import stopped after 1.5 k but another import was successful with 5k records.

    Can you advise any other issues maybe with data you know that could be causing this.

    Thanks for your support.

    August 28, 2018 at 7:05 am #444277

    Iain Brewster
    Expired Member
    Post count: 74

    Or any logs that say why the import terminated ?

    August 28, 2018 at 2:01 pm #444347

    Paolo
    Site Admin
    Post count: 31206

    Probably you are on a shared server and when other sites on that server have low traffic it can import the entire file, while when traffic is higher, only partial imports are possible.

    On a dedicated server this would not happen.

    You should really look into better hosting or this could drive you crazy before you are able to import the entire db without faults…

    We can’t do miracles unfortunately.

    August 30, 2018 at 10:38 am #444643

    Iain Brewster
    Expired Member
    Post count: 74

    Thanks is Paolo,

    I am looking at dedicated servers cheaper than my current host. Host gator. Can you recommend any?
    Also host gator have advised I could import faster by FTP directly into the files. Is this an option?

    Thanks for your patience with this as I am new to all of this and learning.

    August 30, 2018 at 4:43 pm #444707

    Paolo
    Site Admin
    Post count: 31206

    Finding a dedicated server cheaper than a shared hosting is like finding a sport car that costs less than a city car…

    If you find one let us know. 🙂

    We use one of the cheapest we could find and it cost us per month $56 for the actual server and $60 for the server management guy.

    If you don’t have the budget to create this website, you should consider a project with less listings that can be imported and managed by a smaller server.

    Thanks

    August 30, 2018 at 6:20 pm #444736

    Iain Brewster
    Expired Member
    Post count: 74

    Sorry paolo. I think you misunderstood me. I meant I am looking at dedicated servers cheaper than the ones my current host can offer not cheaper than my current sharing plan. So was looking for some guidance in that respect.

    August 30, 2018 at 6:58 pm #444742

    Paolo
    Site Admin
    Post count: 31206

    ops, sorry about that…

    http://www.nocix.net/ has the best prices on the market by far, but they are not managed servers, you must do the server configuration on your own or have someone do it for you.

    Hope this helps,

    August 30, 2018 at 7:02 pm #444744

    Paolo
    Site Admin
    Post count: 31206

    Another option could be to contact https://stghc.com/

    We host our demos with them and we are considering building a GeoDirectory hosted solution with them. They are long time customers of ours and I’m sure they can built a custom solution for your needs at a reasonable price.

    Thanks and sorry again for misunderstanding your question.

  • Author
    Posts
Viewing 15 posts - 1 through 15 (of 15 total)

We have moved to a support ticketing system and our forums are now closed.

Open Support Ticket