CSV Import Problems

This topic contains 57 replies, has 2 voices, and was last updated by  Alistair Williamson 7 years, 10 months ago.

We have moved to a support ticketing system and our forums are now closed.

Open Support Ticket
  • Author
    Posts
  • #371157

    Alistair Williamson
    Full Member
    Post count: 415

    ‘There is no such thing as criticism just feedback’

    Ps. I am not a developer just a brave user who is generally good at following instructions.

    At the risk of jumping on the ‘I don’t like OpenOffice bandwagon…..’

    Having invested a colossal amount of time (months) and effort producing a CSV file with 75,000 records I experienced a variety of problems:

    1) I understand the reason why it doesn’t make sense to build massive data management functionality into GD

    2) I am an experienced Excel user – along with a few others – having to learn OpenOffice as well is simply annoying

    2) It appears you not only need OpenOffice but the GD documentation also describes functions from Libreoffice – this is not made clear

    3) OpenOffice is nowhere near as good as Excel for handling large file sizes. 75,000 rows caused problems in OpenOffice. For example I sorted data by a column and noticed later that my data columns no longer had the original integrity – names and addresses no longer aligned – caused me weeks of work. Also cutting and pasting columns of data from Excel to OpenOffice doesn’t work – I started with 75,000 rows in Excel and ended up with 65,000 in OpenOffice. My confidence in OpenOffice is non-existent. Excel is simply a better spreadsheet. Fact. The fact that Excel doesn’t work with GD is a limitation.

    4) In the end I prepared all of my data in Excel, saved it as a CSV and then opened it in Libreoffice and followed the ‘save a copy’ routine described in ‘tips and tricks’ and the file Import has worked.

    5) I limited my Import to a 2Mb file size – I didn’t trust what would happen if I tried to Import a 40mb file – thus allowing me to Import c. 4,000 records at a time. I have a 58 Mb/s upload speed and a JaguarPC VPS server with 2 CPU’s and 2Mb memory. The Import has taken 4 hours – very time consuming. During this time I had to leave my PC running and couldn’t do any work on the website.

    7) If a 40Mb Import were supported by GD and my server it would have taken 80 hours before I found out there was a problem.

    6) During the Import I have not been able to do any other work in the backend otherwise the Import stops. It would be far better if the Import could run in the background.

    7) I have realised during the Import that I need to change one field. I am expecting to have to export all of the data again – this is very quick 🙂 Make the change – also very quick 🙂 – and then reimport the data again – horribly slow 🙁 plus the uncertainty of success…..Not being able to make simple adjustments is a major problem. I have tried WP Search and Replace and crashed everything; I have to say that I am not keen to learn yet another tool outside GD.

    8) Early experiments with importing my data caused duplication problems – I misinterpreted ‘if pos ID exists…..’ and I had to delete records en masse from GD – this took hours. This is also very challenging and time consuming because the facility to select a group of records is extremely limited. For example you can only sort records by title or date published or select by name. Furthermore GD export only allows selection by published dates. This puts a major emphasis on getting the Import right in the first place. Any improvement in the selection at GD export would help hugely .e.g. Selection by ‘post ID’ from xxxx to xxxx.

    Just feedback.

    Bests

    Al

    #371171

    Stiofan O’Connor
    Site Admin
    Post count: 22956

    Hi Al,

    I’ll try and answer your questions in turn.

    1. I guess this was rhetorical? 🙂

    2. Excel has “features” that simply can’t be turned off and it will start reformatting and breaking anything with numbers… I would say complain to Microsoft but they would probably just not listen and make it worse…

    3. Try smaller batches. Its not that Excel does not just work with GD its that it does not work with numbers, ironic i know 🙂

    4. Your brave, i would check all GPS and phone numbers still match.

    5. I’ll try and think of ways to improve things.

    7.1? Yes we recommend smaller batches.

    6. That all depends on the resources of your server.

    7.2? Is it mass replace all in a row?

    8. Yes deleting in bulk is a notably slow process on WP, i am not sure there is much we can do about that.

    Thanks for the feedback Al,

    Stiofan

    #371173

    Alistair Williamson
    Full Member
    Post count: 415

    4. I had to rebuild the whole data set 🙁

    6. Does this mean I can leave the chrome tab with the import running, open another tab and still work in GD backend?

    7.2 unfortunately no. I have to edit 2 fields. I think I will just export batches, fix and reimport.

    8. Would it be possible to add a post_id range as well as date range? In that way you can identify small batches imported on the same date?

    Thanks

    Al

    #371185

    Stiofan O’Connor
    Site Admin
    Post count: 22956

    6. Yes should should be able to if your server can handle it.

    8. In the next ver you will be able to change the published date, i guess that will help.

    Stiofan

    #371186

    Alistair Williamson
    Full Member
    Post count: 415

    Thanks again 🙂

    #371362

    Alistair Williamson
    Full Member
    Post count: 415

    Hi,

    1. Even though I changed the import file size in GD>General>Import file size. I am getting this error?

    Array( [error] => The uploaded file exceeds the upload_max_filesize directive in php.ini.)x

    2. Also is there anyway a non-imported listing can be identified apart from reexporting the data and checking on Openoffice or libreoffice?

    Bests

    Al

    #371510

    Stiofan O’Connor
    Site Admin
    Post count: 22956

    Hi AL,

    1. That is a server setting, you would need to ask your server admin to change that.

    2. Not sure i know what you mean, can’t you just search for it with the search box?

    Stiofan

    #371539

    Alistair Williamson
    Full Member
    Post count: 415

    Hi Stiofan,

    1. I will speak to the server guys

    2. I was commenting on the message received when an import is completed. If there is a record the system rejects, a message is received, ‘999 records imported successfully and 1 could not be added due to : blank/invalid address(city, region, country, latitude, longitude).

    The problem is:

    – you have to wait for the import to complete to find out what worked
    – you don’t know which 1 out of the 1,000 records was wrong – not specific enough. record 789 didnt import would be better…
    – the message – blank/invalid address(city, region, country, latitude, longitude) isn’t specific enough

    I have exported 10,000 records in one hit without a problem then made some minor changes and had to break the file into smaller sizes and then eventually get the messages:

    Total 1000 item(s) found.
    1000 / 1000 item(s) could not be added due to blank/invalid address(city, region, country, latitude, longitude).
    1000 / 1000 item(s) could not be added due to blank title/invalid post type/invalid characters used in data.

    At this point I have no idea what is wrong and have no choice but to revisit/recreate my import file.

    Again only feedback but very time consuming and frustrating.

    If I had millions of listings this would be impossible. How many of your customers are trying to manage really large databases? How are they doing it? Simply saying it all works in OpenOffice – I have read this response in the Forum a lot – isn’t really true I am afraid.

    Any thoughts or improvements now or for the future very much appreciated.

    Bests

    Al 🙂

    #371545

    Alistair Williamson
    Full Member
    Post count: 415

    Incidentally, I test any import with a handful of listings before attempting a larger import – 1,000 listings max. 2Mb limit from my server writing to JaguarPc – and have the waited up to 1 hour for the import to complete before finding there was an unforseen/unclear import problem.

    #371574

    Stiofan O’Connor
    Site Admin
    Post count: 22956

    Again thanks for the feedback, i’ll need to take a day and brainstorm it when i find time.
    If you think of any solutions let me know 🙂

    Stiofan

    #371687

    Alistair Williamson
    Full Member
    Post count: 415
    This reply has been marked as private.
    #371709

    Alistair Williamson
    Full Member
    Post count: 415
    This reply has been marked as private.
    #371710

    Alistair Williamson
    Full Member
    Post count: 415
    This reply has been marked as private.
    #371785

    Stiofan O’Connor
    Site Admin
    Post count: 22956

    I would suggest getting it right the first time 🙂

    WP is slow at importing, we do it via ajax so that an unlimited amount can be ported in but it can slow it down a little.

    What sort of massive bulk changes are you making?

    Stiofan

    #371797

    Alistair Williamson
    Full Member
    Post count: 415

    Hi Stiofan,

    🙂 Agreed 🙂

    I changed an image on all records, so changed the URL in one column.

    I added another custom field and introduced text data akin to the post_title

    Cheers

    Al

Viewing 15 posts - 1 through 15 (of 58 total)

We have moved to a support ticketing system and our forums are now closed.

Open Support Ticket
20% Discount Offer
Hurry! Get your 20% discount before it expires. Get 20% Discount