Data management

This topic contains 6 replies, has 2 voices, and was last updated by  Kai Liu 5 years, 2 months ago.

We have moved to a support ticketing system and our forums are now closed.

Open Support Ticket
  • Author
    Posts
  • #464676

    Kai Liu
    Expired Member
    Post count: 368

    I am trying to load a bigger pool of data to one CPT. However, new CSV could have duplicate records of existing listings in GD. What is the best approach to load new CSV without creating thousands of duplicate listing?

    I was thinking using new cat name in new CSV, import them, then delete the old categories, and rename the new cat back to old cat names. However, removing old cat only making existing listings become orphan listings, it is even harder to remove.

    I also thought of working in DB by making address field as Unique, but this will have other side effect.

    Please tell me an efficient way to achieve my purpose. I have projects on both V1 and V2, both facing the same problem.

    Also, will you consider to build in features to eliminate duplicated while importing by comparing multiple fields (like Remove Duplicate Data feature in Excel where I can specify multiple fields to compare, if all fields are identical, count it as duplicate and ignore when import, more advanced if allowing user to pre-define which version to keep in case of duplicate, later or existing). With this feature, will make continuous data update much easier.

    thanks

    Sam

    #464683

    Guust
    Moderator
    Post count: 29970

    You can export all the listings you have and merge with the new CSV. Then you can look for your duplicates before reimporting, using the LibreOffice tools you suggested.
    If you have a duplicate, delete one of the two but keep the post ID in the first column.
    Then upload and choose to overwrite existing post IDs.

    Thanks

    #464712

    Kai Liu
    Expired Member
    Post count: 368

    Thanks Guust,

    I think your suggestion is the quickest way to eliminate duplicates with least process. And will maintain existing google index of the listings

    Cheers.

    Sam

    #464783

    Guust
    Moderator
    Post count: 29970

    You’re welcome

    #465528

    Kai Liu
    Expired Member
    Post count: 368

    Is it possible to add parameters to import function, eg allow user to define what is duplicate by ticking available table fields, such as post_title + post_zip + phone (same as Remove Duplicate Data function in Excel), plus option if duplicate occurs, Skip or Overwrite.

    During importing and processing, if match of defined fields are found, skip or overwrite that row of record according to user selection.

    This will save hours, even days of work to try to dig out duplicates, even tho it will take longer to import, still worth it.

    #465530

    Guust
    Moderator
    Post count: 29970

    That is a customisation you would have to organise yourself.

    #465547

    Kai Liu
    Expired Member
    Post count: 368

    got it. thanks

Viewing 7 posts - 1 through 7 (of 7 total)

We have moved to a support ticketing system and our forums are now closed.

Open Support Ticket