Nightly process for a dataset of 150k entries


I have a dataset of 150k record that needs to be updated nightly. We get this data through an API. From the payload we will parse the data and map it to 5 individual CDT's and then write to the db.

How should I design this whole process?

  Discussion posts and replies are publicly visible