CSV Import Problems
This topic contains 57 replies, has 2 voices, and was last updated by Alistair Williamson 7 years, 10 months ago.
We have moved to a support ticketing system and our forums are now closed.
Open Support TicketTagged: CSV Import Problems
-
AuthorPosts
-
April 1, 2017 at 7:19 am #371157
‘There is no such thing as criticism just feedback’
Ps. I am not a developer just a brave user who is generally good at following instructions.
At the risk of jumping on the ‘I don’t like OpenOffice bandwagon…..’
Having invested a colossal amount of time (months) and effort producing a CSV file with 75,000 records I experienced a variety of problems:
1) I understand the reason why it doesn’t make sense to build massive data management functionality into GD
2) I am an experienced Excel user – along with a few others – having to learn OpenOffice as well is simply annoying
2) It appears you not only need OpenOffice but the GD documentation also describes functions from Libreoffice – this is not made clear
3) OpenOffice is nowhere near as good as Excel for handling large file sizes. 75,000 rows caused problems in OpenOffice. For example I sorted data by a column and noticed later that my data columns no longer had the original integrity – names and addresses no longer aligned – caused me weeks of work. Also cutting and pasting columns of data from Excel to OpenOffice doesn’t work – I started with 75,000 rows in Excel and ended up with 65,000 in OpenOffice. My confidence in OpenOffice is non-existent. Excel is simply a better spreadsheet. Fact. The fact that Excel doesn’t work with GD is a limitation.
4) In the end I prepared all of my data in Excel, saved it as a CSV and then opened it in Libreoffice and followed the ‘save a copy’ routine described in ‘tips and tricks’ and the file Import has worked.
5) I limited my Import to a 2Mb file size – I didn’t trust what would happen if I tried to Import a 40mb file – thus allowing me to Import c. 4,000 records at a time. I have a 58 Mb/s upload speed and a JaguarPC VPS server with 2 CPU’s and 2Mb memory. The Import has taken 4 hours – very time consuming. During this time I had to leave my PC running and couldn’t do any work on the website.
7) If a 40Mb Import were supported by GD and my server it would have taken 80 hours before I found out there was a problem.
6) During the Import I have not been able to do any other work in the backend otherwise the Import stops. It would be far better if the Import could run in the background.
7) I have realised during the Import that I need to change one field. I am expecting to have to export all of the data again – this is very quick 🙂 Make the change – also very quick 🙂 – and then reimport the data again – horribly slow 🙁 plus the uncertainty of success…..Not being able to make simple adjustments is a major problem. I have tried WP Search and Replace and crashed everything; I have to say that I am not keen to learn yet another tool outside GD.
8) Early experiments with importing my data caused duplication problems – I misinterpreted ‘if pos ID exists…..’ and I had to delete records en masse from GD – this took hours. This is also very challenging and time consuming because the facility to select a group of records is extremely limited. For example you can only sort records by title or date published or select by name. Furthermore GD export only allows selection by published dates. This puts a major emphasis on getting the Import right in the first place. Any improvement in the selection at GD export would help hugely .e.g. Selection by ‘post ID’ from xxxx to xxxx.
Just feedback.
Bests
Al
April 1, 2017 at 11:39 am #371171Hi Al,
I’ll try and answer your questions in turn.
1. I guess this was rhetorical? 🙂
2. Excel has “features” that simply can’t be turned off and it will start reformatting and breaking anything with numbers… I would say complain to Microsoft but they would probably just not listen and make it worse…
3. Try smaller batches. Its not that Excel does not just work with GD its that it does not work with numbers, ironic i know 🙂
4. Your brave, i would check all GPS and phone numbers still match.
5. I’ll try and think of ways to improve things.
7.1? Yes we recommend smaller batches.
6. That all depends on the resources of your server.
7.2? Is it mass replace all in a row?
8. Yes deleting in bulk is a notably slow process on WP, i am not sure there is much we can do about that.
Thanks for the feedback Al,
Stiofan
April 1, 2017 at 11:49 am #3711734. I had to rebuild the whole data set 🙁
6. Does this mean I can leave the chrome tab with the import running, open another tab and still work in GD backend?
7.2 unfortunately no. I have to edit 2 fields. I think I will just export batches, fix and reimport.
8. Would it be possible to add a post_id range as well as date range? In that way you can identify small batches imported on the same date?
Thanks
Al
April 1, 2017 at 12:45 pm #3711856. Yes should should be able to if your server can handle it.
8. In the next ver you will be able to change the published date, i guess that will help.
Stiofan
April 1, 2017 at 12:52 pm #371186Thanks again 🙂
April 2, 2017 at 7:31 pm #371362Hi,
1. Even though I changed the import file size in GD>General>Import file size. I am getting this error?
Array( [error] => The uploaded file exceeds the upload_max_filesize directive in php.ini.)x
2. Also is there anyway a non-imported listing can be identified apart from reexporting the data and checking on Openoffice or libreoffice?
Bests
Al
April 3, 2017 at 3:34 pm #371510Hi AL,
1. That is a server setting, you would need to ask your server admin to change that.
2. Not sure i know what you mean, can’t you just search for it with the search box?
Stiofan
April 3, 2017 at 4:54 pm #371539Hi Stiofan,
1. I will speak to the server guys
2. I was commenting on the message received when an import is completed. If there is a record the system rejects, a message is received, ‘999 records imported successfully and 1 could not be added due to : blank/invalid address(city, region, country, latitude, longitude).
The problem is:
– you have to wait for the import to complete to find out what worked
– you don’t know which 1 out of the 1,000 records was wrong – not specific enough. record 789 didnt import would be better…
– the message – blank/invalid address(city, region, country, latitude, longitude) isn’t specific enoughI have exported 10,000 records in one hit without a problem then made some minor changes and had to break the file into smaller sizes and then eventually get the messages:
Total 1000 item(s) found.
1000 / 1000 item(s) could not be added due to blank/invalid address(city, region, country, latitude, longitude).
1000 / 1000 item(s) could not be added due to blank title/invalid post type/invalid characters used in data.At this point I have no idea what is wrong and have no choice but to revisit/recreate my import file.
Again only feedback but very time consuming and frustrating.
If I had millions of listings this would be impossible. How many of your customers are trying to manage really large databases? How are they doing it? Simply saying it all works in OpenOffice – I have read this response in the Forum a lot – isn’t really true I am afraid.
Any thoughts or improvements now or for the future very much appreciated.
Bests
Al 🙂
April 3, 2017 at 5:04 pm #371545Incidentally, I test any import with a handful of listings before attempting a larger import – 1,000 listings max. 2Mb limit from my server writing to JaguarPc – and have the waited up to 1 hour for the import to complete before finding there was an unforseen/unclear import problem.
April 3, 2017 at 5:49 pm #371574Again thanks for the feedback, i’ll need to take a day and brainstorm it when i find time.
If you think of any solutions let me know 🙂Stiofan
April 4, 2017 at 11:49 am #371687This reply has been marked as private.April 4, 2017 at 1:14 pm #371709This reply has been marked as private.April 4, 2017 at 1:21 pm #371710This reply has been marked as private.April 4, 2017 at 5:59 pm #371785I would suggest getting it right the first time 🙂
WP is slow at importing, we do it via ajax so that an unlimited amount can be ported in but it can slow it down a little.
What sort of massive bulk changes are you making?
Stiofan
April 4, 2017 at 6:17 pm #371797Hi Stiofan,
🙂 Agreed 🙂
I changed an image on all records, so changed the URL in one column.
I added another custom field and introduced text data akin to the post_title
Cheers
Al
-
AuthorPosts
We have moved to a support ticketing system and our forums are now closed.
Open Support Ticket