Skip to main content
New Participant
Aug 21, 2023 8:45 PM UTC
Question

Parsing a DAT file into SQL database

I'm trying to parse a large data file into our sql database.  I have found the import csv to database smart service which would work with the "|" delimiter of the dat file except it has metadata as its first and last lines (not a csv header).  I have uploaded the file into an appian document object, but I can't figure out how to remove those lines so it works with the smart service.

I should also note the dat file is coming from an external datasouce and cannot be altered prior to ingestion.

5 replies

mathieud0001
Genius
Aug 21, 2023 10:58 PM UTC

1st thing I would try is to load the file as is in the DB and just delete/ignore the 1st and last rows.

Otherwise, you might be able to do something with this plugin:

[View:https://community.appian.com/b/appmarket/posts/text-file-reader]

erikw0003Author
New Participant
Aug 31, 2023 2:21 PM UTC

Hi,

Thank you for the suggestion.  It does seem to be a way to accomplish what I would like to do.  However I am running into hitting max heap usage.  What is the correct way to increase heap size in appian?  I've tried looking and there isn't a lot of directly helpful information.

mathieud0001
Genius
Aug 31, 2023 2:29 PM UTC

You probably want to find a way to split this in multiple files.