Import CSV To Database.

Hi,

In my CSV, i have records with "id" field as a Primary Key and again i Import the same CSV with new values then it will override the existing records.

1. I don't want to insert the records those are already exist.

2. Or I don't want to insert Duplicate records.

  Discussion posts and replies are publicly visible

Parents
  • Hi,

    What i understand is that you dont want to over ride existing rows when a new CSV is imported at the same time you dont want to have duplicates. See if the following will workout for you,

    Lets assume your CSV contains columns as id,name,dob .

    Create your DB table (staging table) that has columns as follows
    uid(Primary Key),
    id,
    name,
    dob,
    batchID,
    createdDateTime.

    You can populate the Additional columns batchID (autoincrement this value for each file import ),createdDateTime from the smartService . This way you will have visibility to identify imported rows by batchID. Build a stored Proc in Mysql using INSERT () ON DUPLICATE KEY UPDATE() and populate your actual table with data for a specific batch ID.
Reply
  • Hi,

    What i understand is that you dont want to over ride existing rows when a new CSV is imported at the same time you dont want to have duplicates. See if the following will workout for you,

    Lets assume your CSV contains columns as id,name,dob .

    Create your DB table (staging table) that has columns as follows
    uid(Primary Key),
    id,
    name,
    dob,
    batchID,
    createdDateTime.

    You can populate the Additional columns batchID (autoincrement this value for each file import ),createdDateTime from the smartService . This way you will have visibility to identify imported rows by batchID. Build a stored Proc in Mysql using INSERT () ON DUPLICATE KEY UPDATE() and populate your actual table with data for a specific batch ID.
Children