I have a lot of data in which is to be written to Database. It can be like an ar

chetany

A Score Level 1

over 7 years ago

I have a lot of data in which is to be written to Database. It can be like an array of 24,000 CDT variables. I want to write it to DB. Now, I have two approaches for this. I can A) Either Write to Database in batches - I can decide a batchSize of 200 and write 200 rows at a time sequentially. This approach will take some time to completely write to DB B) I can use MNI in process models. I will still use batchSize, but now more than one Write to DS smart service will execute simultaneously. This will increase the parallel writes to DB and will also complete the process model quickly.

My question: Will the parallel writes to DB using MNI affect the performance? Also I think it will open too many DB connections to database, so is it a good idea?

Discussion posts and replies are publicly visible

Parents

0 sikhivahans over 7 years ago

Be it a RDBMS CSV Import Export or Execute Stored Procedure, if they are used to process the large data sets in a single shots without batch processing, they may cause performance bottle necks.

As far as my knowledge is considered, there is a limit on the response duration of the smart service (I guess it's 4000 ms) and if this limit is exceeded, we need to revisit our design and again MNI might be one of the effective solutions. So I would suggest keeping an eye on the response duration of the any plugin prior to making it deal with large data sets. We have had this issue earlier and Health Check has categorised the operation as 'High Risk' stating that the response duration has exceeded the limits.
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

Reply

0 sikhivahans over 7 years ago

Be it a RDBMS CSV Import Export or Execute Stored Procedure, if they are used to process the large data sets in a single shots without batch processing, they may cause performance bottle necks.

As far as my knowledge is considered, there is a limit on the response duration of the smart service (I guess it's 4000 ms) and if this limit is exceeded, we need to revisit our design and again MNI might be one of the effective solutions. So I would suggest keeping an eye on the response duration of the any plugin prior to making it deal with large data sets. We have had this issue earlier and Health Check has categorised the operation as 'High Risk' stating that the response duration has exceeded the limits.
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

Children

No Data