Migration

Is it possible to migrate for instance a CDT but change the values to hide the real values?

  Discussion posts and replies are publicly visible

Parents Reply Children
  • I got more input on what is asked, so, basically, I need to export from a DB all the data to an exel, then from the excel i need to import those data to another DB but im the middle of that import/export the data needs to change so I dont upload the real values to the second DB, for instance Roger should be Baman and 14/03/1993 should be 22/10/1889

  • So for this you will have 2 CDTs, same fields in each but different names/tables.  One for your original table (A) and the manipulated table (B).  Utilize the Export DSE to Excel node to populate CDT A.  Create an expression rule with an input of CDT A which will have an output of CDT B.  This rule will utilize a!forEach() over CDT A, creating rows for CDT B such as the example below using a!map().  Note in here we are creating a random value for salary for CDT B, in this area you can manipulate any of the data points you like.

    a!localVariables(
      local!data: {
        a!map(id: 1, firstName: "John", lastName: "Doe", salary: 50000),
        a!map(id: 2, firstName: "Jane", lastName: "Doe", salary: 55000)
      },
      
      a!forEach(
        items: local!data,
        expression: a!map(
          id: fv!item.id,
          firstName: fv!item.firstName,
          lastName: fv!item.lastName,
          salary: fixed(rand()*100000,0)
        )
      )
    )

    Once this is done, utilize the Write to DSE service to persist CDT B to the new table.

  • and how do I modify the text values? being that it must contain the same number of letters and capitalisation 

  • To randomize strings as such (and also apply the same to numeric values), create 2 helper rules:

    rule!chris_test_randomize_numeric(numeric)

    tointeger(
      joinarray(
        a!forEach(
          items: 1+enumerate(len(ri!numeric)),
          expression: {
            a!localVariables(
              local!random: tointeger(rand()*9-1)+1,
              if(
                and(
                  fv!isFirst,
                  local!random=0
                ),
                1, /* change to 1 to maintain length */
                local!random
              )
            )
          }
        )
      )
    )

    rule!chris_test_randomize_string(text)

    a!localVariables(
      local!alpha: "abcdefghijklmnopqrstuvwxyz",
      joinarray(
        a!forEach(
          items: code(ri!text),
          expression: {
            a!localVariables(
              local!random: index(local!alpha,tointeger(rand()*len(local!alpha)-1)+1,null),
              if(
                fv!item<97,
                upper(local!random),
                local!random
              )
            )
          }
        )
      )
    )

    Then call your new rules when converting your CDT:

    a!localVariables(
      local!data: {
        a!map(id: 1, firstName: "John", lastName: "Doe", salary: 50000),
        a!map(id: 2, firstName: "Jane", lastName: "Doe", salary: 55000),
        a!map(id: 3, firstName: "Timothy", lastName: "McDonald", salary: 45000)
      },
    
      a!forEach(
        items: local!data,
        expression: a!map(
          id: fv!item.id,
          firstName: rule!chris_test_randomize_string(text: fv!item.firstName),
          lastName: rule!chris_test_randomize_string(text: fv!item.lastName),
          salary: rule!chris_test_randomize_numeric(numeric: fv!item.salary)
        )
      )
    )