Writing large amounts of data to MariaDB Database

Certified Senior Developer

Hi,

We have a requirement to write over 50 000 rows daily to our MariaDB database.

I want to know what is the best way to get this done. We tried using a Record Type but the data was too large for it to handle and a breaker cancelled the synchronisation.

I have seen that creating Multiple node instances is not recommended as we will create an excessive connexion pool with the DB.

The only solution that comes to mind is to have a subprocess that gets fed smaller batches (100 or 1000 rows) looping through the data for insertion (sequential).

  Discussion posts and replies are publicly visible

Parents Reply Children