Memory Threshold Issue

RE: Memory Threshold Issue

Peter Lewis — Tue, 29 Jun 2021 13:35:29 GMT

Agreed with the solutions suggested for truncating a table, but we still haven't gotten to the fundamental problem: why do you need to delete all of this data? Is this a temporary table that you're using for ETL purposes? How is the data getting populated in this table?

I'm concerned that this is an XY problem where we're trying to solve the wrong problem.

RE: Memory Threshold Issue

Mike Schmitt — Tue, 29 Jun 2021 13:17:05 GMT

Overall I agree with you guys about how best to truncate a table (assuming they want to make their PK IDs to start back at zero, unless I'm mistaken about what TRUNCATE does). But this inefficient querying is something I see across all sorts of use cases so I always try to point it out anyway ;-)

RE: Memory Threshold Issue

davel001150 — Tue, 29 Jun 2021 13:07:28 GMT

I agree with Stefan. You're making this about two hundred thousand times harder than it needs to be. You're gathering 100,000 ids, storing 100,000 IDs in a PV, then doing 100 subprocesses that each handle a Delete from Datastore node that is MNI 1000 times. Or maybe trying a forEach with a delete from datastore rule that can be cloned up to a million times. For what? Since you're not filtering, your ultimate result will be an empty table.

Just use the TRUNCATE command. One step. No Appian, no PV with mountain of data eating your RAM, no looping. Just a few seconds and empty table. Just get a DBA to TRUNCATE the table if you want it to have 0 rows. Or "Execute Stored Procedure" and the stored procedure in question has only 1 line of code: TRUNCATE TABLE <name>;

Good point Mike, but we're in the realm of optimizing code before removing it.

RE: Memory Threshold Issue

Mike Schmitt — Tue, 29 Jun 2021 13:04:24 GMT

In addition to what Stefan said - in your first query, there is no need to pull back all rows of a query ("batchSize: -1") when you're just checking that the totalCount is zero. Instead set your batchsize to Zero, which makes the query pull back metadata (including TotalCount) but not pull back any rows of data.

RE: Memory Threshold Issue

Stefan Helzle — Tue, 29 Jun 2021 09:31:15 GMT

https://docs.appian.com/suite/help/21.2/Local_Variables.html

academy.appian.com/

But, again, this will not solve your root problem. Trying to delete 100ks of rows in DB with that method is no good idea.

RE: Memory Threshold Issue

ayushimittal — Tue, 29 Jun 2021 08:51:51 GMT

Hi Stefan,

How can I store the single column of data store entity in local variable?

RE: Memory Threshold Issue

Stefan Helzle — Tue, 29 Jun 2021 08:12:18 GMT

First, I strongly suggest to not do that query two times in the if() statement. Create a local variable and store the data there.

Next. This seems like you want to delete all rows in that table. MariaDB has a special comment for this. It is called "truncate". I suggest to write a small stored procedure and use that.

RE: Memory Threshold Issue

ayushimittal — Tue, 29 Jun 2021 08:08:11 GMT

Hi Stewart,

We are fetching a single column (ID) data from one database table so will it be helpful to create a view in this case?

RE: Memory Threshold Issue

ayushimittal — Tue, 29 Jun 2021 08:06:46 GMT

Hi Stefen,

Please find the expression rule code, we are using to fetch the data from database table:

a!entityDataIdentifiers(
  entity: ri!Entity,
  identifiers: {
    if(
      a!queryEntity(
        entity: ri!Entity,
        query: a!query(
          selection: a!querySelection(
            columns: {
              a!queryColumn(
                field: "id"
              )
            }
          ),
          pagingInfo: a!pagingInfo(
            startIndex: 1,
            batchSize: - 1
          )
        ),
        fetchTotalCount: true
      ).Totalcount = 0,
      "",
      a!queryEntity(
        entity: ri!Entity,
        query: a!query(
          selection: a!querySelection(
            columns: {
              a!queryColumn(
                field: "id"
              )
            }
          ),
          pagingInfo: a!pagingInfo(
            startIndex: 1,
            batchSize: - 1
          )
        ),
        fetchTotalCount: false
      ).data.id
    )
  }
)

Please let me know how can I optimize it as it is fetching all the IDs need to be deleted.

RE: Memory Threshold Issue

Stefan Helzle — Tue, 29 Jun 2021 06:12:53 GMT

Is this part of a larger DB cleanup process? Will it run periodically? Why is there so much to clean up? Could it be prevented?

Is the query to fetch the IDs a simple one? Like "select id from table where tobedeleted=1"? If yes, did you consider to use the Query Database Node and a statement like "delete from table where tobedeleted=1". There is some drawbacks going that way, but it might be worth it.

RE: Memory Threshold Issue

ayushimittal — Tue, 29 Jun 2021 05:56:14 GMT

Hi Peter,

Using expression rule, We are fetching all the IDs from the database table and passing it to script task in the process model as the 'IDs to be deleted'.

While we are executing this process model, it is failing on this script task and throwing the above error.

RE: Memory Threshold Issue

Peter Lewis — Mon, 28 Jun 2021 17:05:07 GMT

I think we need to take a step back - what is your use case? There might be a completely different query you could run depending on what you're trying to do. How are you using this data? What is the goal?

RE: Memory Threshold Issue

Stewart Burchell — Mon, 28 Jun 2021 16:59:27 GMT

Could you do the "heavy lifting" in the database itself? That is, use the database aggregation functions to generate a view (virtual or materialised) and then point Appian at the result table in order to present the data.

RE: Memory Threshold Issue

Mike Schmitt — Mon, 28 Jun 2021 16:33:31 GMT

Then I defer to 's original suggestion of employing Batching somehow within your process.

RE: Memory Threshold Issue

ayushimittal — Mon, 28 Jun 2021 16:29:10 GMT

We are fetching only a single column (ID) of the database table but there are over 2 Lakh rows.

RE: Memory Threshold Issue

Mike Schmitt — Mon, 28 Jun 2021 15:01:26 GMT

Considering that you seem to be fetching "listofIDtodelete" - how many columns are you returning in your query?

RE: Memory Threshold Issue

davel001150 — Mon, 28 Jun 2021 11:51:58 GMT

One possibility might be Execute Stored Procedure. That can also be called upon to run a database FUNCTION, which can return simply the results of your aggregation. I assume you just want things like COUNT, AVE, MIN, MAX. You can have the DB run all the aggregation calculations and have it return only the few integers you need, if what you need is a few integers. That way, you don't have to bring the whole dataset to bear for Appian to perform aggregation, but let the DB do it.

RE: Memory Threshold Issue

Stefan Helzle — Mon, 28 Jun 2021 11:26:07 GMT

Then you need to optimize your query. Do your really need all data returned? Is this about reporting into Excel or charts?

RE: Memory Threshold Issue

ayushimittal — Mon, 28 Jun 2021 11:23:15 GMT

Thanks Stefan for your prompt response.

But we are using this data for reporting purpose. The aggregation functions are applied on the whole data so how can we apply batching logic in it.

RE: Memory Threshold Issue

Stefan Helzle — Mon, 28 Jun 2021 11:15:59 GMT

The only sustainable solution is to refactor your design. Think of batching, looping etc.