• State Not Answered
  • +1 person also asked this people also asked this
  • Replies 14 replies
  • Subscribers 6 subscribers
  • Views 4646 views
  • Users 0 members are here
Related Discussions
Home » Discussions » Process

I have a lot of data in which is to be written to Database. It can be like an ar

chetany
A Score Level 1
I have a lot of data in which is to be written to Database. It can be like an array of 24,000 CDT variables. I want to write it to DB. Now, I have two approaches for this. I can A) Either Write to Database in batches - I can decide a batchSize of 200 and write 200 rows at a time sequentially. This approach will take some time to completely write to DB B) I can use MNI in process models. I will still use batchSize, but now more than one Write to DS smart service will execute simultaneously. This will increase the parallel writes to DB and will also complete the process model quickly.

My question: Will the parallel writes to DB using MNI affect the performance? Also I think it will open too many DB connections to database, so is it a good idea?

OriginalPostID-215023

OriginalPostID-215023

  Discussion posts and replies are publicly visible

Parents
  • 0
    A Score Level 2
    What's the source of your data that you are trying to write? 24000 records in a CDT at any point in time in a process is huge and will consume significant memory in Appian engine server. Depending on how you are sourcing and feeding the Data to the process that's supposed to write, you might want to consider using the "RDBMS CSV Import Export" smart service. Although I've not used this smart service, but using this will make sure that you are not flooding your process with huge amount of data in the form of process variables as the parsing and writing the data will be done in the JVM. But again you need to be mindful of the JVM memory issue processing large set of data and you should do it only during off hours.
Reply
  • 0
    A Score Level 2
    What's the source of your data that you are trying to write? 24000 records in a CDT at any point in time in a process is huge and will consume significant memory in Appian engine server. Depending on how you are sourcing and feeding the Data to the process that's supposed to write, you might want to consider using the "RDBMS CSV Import Export" smart service. Although I've not used this smart service, but using this will make sure that you are not flooding your process with huge amount of data in the form of process variables as the parsing and writing the data will be done in the JVM. But again you need to be mindful of the JVM memory issue processing large set of data and you should do it only during off hours.
Children
No Data

© 2024 Appian. All rights reserved.