Skip to main content
jyotik5834
jyotik5834
September 12, 2024
Question

Import huge(~35 MB) CSV file into Database

  • September 12, 2024
  • 6 replies
  • 0 views

I have a file that have around 80000+ records and size is around 35MB.

I would like to understand the way to read the data in it but need paging to be part of the process as i want this to be inserted into DB.

6 replies

somasundaram.d
somasundaram.d
September 12, 2024

Try the "Import CSV to Database" smart service in the excel tools plugin

community.appian.com/.../excel-tools

It has the reputation to work well even for bigger files

jyotik5834
jyotik5834Author
September 12, 2024

Hi Soma,

Tried it, but it has the file size limitation.

somasundaram.d
somasundaram.d
September 12, 2024

Oh! Then can you also give a try using the "Load CSV to Database" smart service which is part of the same plugin

What is the maximum file size that smart service is accepting?

You can also try using readexcelsheetpaging(), if it is possible to convert the csv to excel

B
bajib0001
September 13, 2024

Hi [mention:57fc948ec4c44d7fbda6e5509555d197:e9ed411860ed4f2ba0265705b8793d05] 

while using Load CSV to Database we have to check this

 

somasundaram.d
somasundaram.d
September 14, 2024

I tried uploading a CSV file with 640000 rows of data with the file size of 62 MB. I did not face any challenge using Import CSV to Database smart service.

Can you please clarify what is the difference we are having, so that we can try to solve your issue?

karumurua531442
karumurua531442
September 16, 2024

Hi[mention:f2e446085ae74ff1abaff016e3349b60:e9ed411860ed4f2ba0265705b8793d05]  I would not recommend pushing larger data files into the database in one go. You can split those into smaller chunks and push them to the database, which would also help to run queries faster with minimal load.

Terms & ConditionsAccessibility statement