Skip to main content
August 21, 2023
Question

Parsing a DAT file into SQL database

  • August 21, 2023
  • 5 replies
  • 0 views

I'm trying to parse a large data file into our sql database.  I have found the import csv to database smart service which would work with the "|" delimiter of the dat file except it has metadata as its first and last lines (not a csv header).  I have uploaded the file into an appian document object, but I can't figure out how to remove those lines so it works with the smart service.

I should also note the dat file is coming from an external datasouce and cannot be altered prior to ingestion.

5 replies

mathieud0001
Brainy
August 21, 2023

1st thing I would try is to load the file as is in the DB and just delete/ignore the 1st and last rows.

Otherwise, you might be able to do something with this plugin:

[View:https://community.appian.com/b/appmarket/posts/text-file-reader]

erikw0003Author
August 31, 2023

Hi,

Thank you for the suggestion.  It does seem to be a way to accomplish what I would like to do.  However I am running into hitting max heap usage.  What is the correct way to increase heap size in appian?  I've tried looking and there isn't a lot of directly helpful information.

mathieud0001
Brainy
August 31, 2023

You probably want to find a way to split this in multiple files.