Most efficient way to remove duplicates in an array of CDT or Map based on a specific key?

I'd try to extract the email values from all items and then use the union() function to reduce duplicates.

union(local!data.email, local!data.email)

That would return a list of unique email values, but not the list of cdt. What would be the best way to return the list of cdt where any duplicates on the email field have been removed?

Hi,
Following Stefan's suggestion, I have employed the 'unique' function and 'forEach' loop to retrieve the CDTs. Please refer to the code snippet below:

a!localVariables(
  local!testData: {
    {
      email: "hello@gmail.com",
      type: 1,
      date: today()
    },
    {
      email: "hello11@gmail.com",
      type: 1,
      date: today()
    },
    {
      email: "hello@gmail.com",
      type: 2,
      date: today()
    }
  },
  local!uniqueEmails: union(
    touniformstring(local!testData.email),
    touniformstring({})
  ),
  a!forEach(
    items: local!uniqueEmails,
    expression: index(
      index(
        local!testData,
        wherecontains(
          tostring(fv!item),
          touniformstring(index(local!testData, "email", {}))
        ),
        {}
      ),
      1,
      {}
    )
  )
)

walkers said:

What would be the best way to return the list of cdt where any duplicates on the email field have been removed?

What exactly are you expecting as the result of this?  If the emails are duplicate, but the other items in each CDT are not necessarily the same, what is the expected behavior?  Would it just keep the first copy of any given unique email and discard the data contained in any duplicates even if it's different?  Or some other potential approach, which you haven't specified?

Hi Mike,

This is the result I want: "just keep the first copy of any given unique email and discard the data contained in any duplicates even if it's different".

walkers said:

This is the result I want: "just keep the first copy of any given unique email

That's good news then, because it should be fairly simple to implement.

In pseudocode:

1. get unique emails from all entries
2. cycle through unique emails
3. grab the array of original entries matching the current unique email
4. return the first such array match

This is basically what Mathurambika M has already written above, FWIW.

Thank you Mathurambika, that worked well.

Instead of the index&wherecontains approach, we can also use the displayvalue() function

a!localVariables(
  local!testData: {
    {
      email: "hello@gmail.com",
      type: 1,
      date: today()
    },
    {
      email: "hello11@gmail.com",
      type: 1,
      date: today()
    },
    {
      email: "hello@gmail.com",
      type: 2,
      date: today()
    }
  },
  local!uniqueEmails: union(
    touniformstring(local!testData.email),
    touniformstring({})
  ),
  a!forEach(
    items: local!uniqueEmails,
    expression: displayvalue(
      tostring(fv!item),
      touniformstring(index(local!testData, "email", {})),
      local!testData,
      {}
    )
  )
)

Instead of the index&wherecontains approach, we can also use the displayvalue() function

a!localVariables(
  local!testData: {
    {
      email: "hello@gmail.com",
      type: 1,
      date: today()
    },
    {
      email: "hello11@gmail.com",
      type: 1,
      date: today()
    },
    {
      email: "hello@gmail.com",
      type: 2,
      date: today()
    }
  },
  local!uniqueEmails: union(
    touniformstring(local!testData.email),
    touniformstring({})
  ),
  a!forEach(
    items: local!uniqueEmails,
    expression: displayvalue(
      tostring(fv!item),
      touniformstring(index(local!testData, "email", {})),
      local!testData,
      {}
    )
  )
)