Migration

Is it possible to migrate for instance a CDT but change the values to hide the real values?

  Discussion posts and replies are publicly visible

Parents Reply Children
  • What I mean is, imagine you have bank account data about a number if clients and you want to migrate that data in order for other people to work on it, but you dont want them to work on the real data, just some transformed data of the real values

  • Sure, if you want to migrate a static set of the data, create a new DB table and insert only the values you want over there, then create a new version of the CDT to point to the new table.  Similarly, you can create a view on the primary table which contains only columns/values or manipulated values from the underlying table.

  • And how can I “encrypt” the data? Chance for instance  183 to 230(random number that can be reverted back to normal)? Thats my main doubt

  • If you are supplying a view for users to work with the data but not manipulate, use this in place of your actual number column:

    SELECT CAST(RAND()*1000 as INT) as 'Random_Number_Column'

    If you are moving this data to a separate table for stale edits, use the same logic in your INSERT script.

    What is the use case?  Are developers working with this data in /design, or is this for end users to view and edit, etc?

  • My tutor asked us to learn about migration and how to change data in order to use it without knowing their real values but i dont find any information about it online

  • I would have questions for your tutor as far as:

    1) Use it how?  Read-only or editable?

    2) Does it need to be real-time, or will a snapshot suffice?

    3) Is this question related specifically to Appian or in general design theory?

    As database values can be masked in a number of ways, few methods already identified in this thread.

  • I got more input on what is asked, so, basically, I need to export from a DB all the data to an exel, then from the excel i need to import those data to another DB but im the middle of that import/export the data needs to change so I dont upload the real values to the second DB, for instance Roger should be Baman and 14/03/1993 should be 22/10/1889

  • So for this you will have 2 CDTs, same fields in each but different names/tables.  One for your original table (A) and the manipulated table (B).  Utilize the Export DSE to Excel node to populate CDT A.  Create an expression rule with an input of CDT A which will have an output of CDT B.  This rule will utilize a!forEach() over CDT A, creating rows for CDT B such as the example below using a!map().  Note in here we are creating a random value for salary for CDT B, in this area you can manipulate any of the data points you like.

    a!localVariables(
      local!data: {
        a!map(id: 1, firstName: "John", lastName: "Doe", salary: 50000),
        a!map(id: 2, firstName: "Jane", lastName: "Doe", salary: 55000)
      },
      
      a!forEach(
        items: local!data,
        expression: a!map(
          id: fv!item.id,
          firstName: fv!item.firstName,
          lastName: fv!item.lastName,
          salary: fixed(rand()*100000,0)
        )
      )
    )

    Once this is done, utilize the Write to DSE service to persist CDT B to the new table.

  • and how do I modify the text values? being that it must contain the same number of letters and capitalisation 

  • To randomize strings as such (and also apply the same to numeric values), create 2 helper rules:

    rule!chris_test_randomize_numeric(numeric)

    tointeger(
      joinarray(
        a!forEach(
          items: 1+enumerate(len(ri!numeric)),
          expression: {
            a!localVariables(
              local!random: tointeger(rand()*9-1)+1,
              if(
                and(
                  fv!isFirst,
                  local!random=0
                ),
                1, /* change to 1 to maintain length */
                local!random
              )
            )
          }
        )
      )
    )

    rule!chris_test_randomize_string(text)

    a!localVariables(
      local!alpha: "abcdefghijklmnopqrstuvwxyz",
      joinarray(
        a!forEach(
          items: code(ri!text),
          expression: {
            a!localVariables(
              local!random: index(local!alpha,tointeger(rand()*len(local!alpha)-1)+1,null),
              if(
                fv!item<97,
                upper(local!random),
                local!random
              )
            )
          }
        )
      )
    )

    Then call your new rules when converting your CDT:

    a!localVariables(
      local!data: {
        a!map(id: 1, firstName: "John", lastName: "Doe", salary: 50000),
        a!map(id: 2, firstName: "Jane", lastName: "Doe", salary: 55000),
        a!map(id: 3, firstName: "Timothy", lastName: "McDonald", salary: 45000)
      },
    
      a!forEach(
        items: local!data,
        expression: a!map(
          id: fv!item.id,
          firstName: rule!chris_test_randomize_string(text: fv!item.firstName),
          lastName: rule!chris_test_randomize_string(text: fv!item.lastName),
          salary: rule!chris_test_randomize_numeric(numeric: fv!item.salary)
        )
      )
    )