Selecting a random sample from a large table

RE: Selecting a random sample from a large table

Soma — Fri, 11 Oct 2024 06:02:08 GMT

Since you are using stored procedure, you should be able to filter out using certain parameters to bring down the data size

RE: Selecting a random sample from a large table

kl0001 — Thu, 10 Oct 2024 18:55:55 GMT

gotcha, I'll do some more investigation into that then..thanks!

RE: Selecting a random sample from a large table

Mike Schmitt — Thu, 10 Oct 2024 17:01:56 GMT

True random values often appear "skewed" due to observer bias. If you want something that's evenly distributed instead of truly random, you'll probably need to determine your own algorithm for that (i.e. take indexes from 1 to N divided by samples desired, then randomize each sample index by taking some deviation based on a generated random). The DB software has loads of functions available and I don't even begin to claim to fathom them all. If you were working in MariaDB i would be more confident in sharing some of my stored proc tricks (disclaimering that i haven't done much play with random numbers in it), but you could probably discover the ones you need by googling the Oracle DB docs.

RE: Selecting a random sample from a large table

kl0001 — Thu, 10 Oct 2024 16:57:48 GMT

Do you have examples of random functions that can be done on the db side? The only ones I've been able to find are DBMS_RANDOM.VALUE and SAMPLE() for oracle, but I've heard DBMS_RANDOM.VALUE can be very slow on large tables, and SAMPLE() tends to return skewed data.

RE: Selecting a random sample from a large table

Mike Schmitt — Thu, 10 Oct 2024 16:20:06 GMT

The SP can recreate much of the view logic you're thinking of internally and do more efficient / more advanced calculations in advance (i assume up to and including using some db-internal logic to select some samples) - since it's all done internal to the DB, it'll process faster than trying to query through Appian.

RE: Selecting a random sample from a large table

kl0001 — Thu, 10 Oct 2024 16:17:27 GMT

What are you thinking of in terms of running it through an SP?

RE: Selecting a random sample from a large table

kl0001 — Thu, 10 Oct 2024 16:15:35 GMT

I've seen some comments that this is a very heavy operation on large tables. If even after the all the joins, the table is still 25 million rows, is DBMS_RANDOM.VALUE still possible performance wise?

RE: Selecting a random sample from a large table

Mike Schmitt — Thu, 10 Oct 2024 15:51:08 GMT

Youch. Then I must defer to prior suggestions to run it through a SP.

RE: Selecting a random sample from a large table

kl0001 — Thu, 10 Oct 2024 15:41:43 GMT

I tried that, but if the start index is big enough, Appian times out with this error:

I'll need to query from a view since I can make sure the view has the correct fields that I need to filter on.

RE: Selecting a random sample from a large table

Mike Schmitt — Thu, 10 Oct 2024 15:25:18 GMT

Why not use the TotalCount to determine the table's current size, then generate a random value between 1 and N, and query that using the generated value as the StartIndex (with a page size of 1)? You could do this for one or several queries into a random row.

a!localVariables(
  local!totalEntries: rule!ASDF_QRY_PersonDocuments(
    pagingInfo: a!pagingInfo(1, 0),
    fetchTotalCount: true()
  ).totalCount,
  
  local!totalSamples: 5,
  
  local!sampleIndices: a!forEach(
    enumerate(local!totalSamples),
    
    tointeger(rand()*local!totalEntries)
  ),
  
  local!sampleQueries: a!forEach(
    local!sampleIndices,
    index(rule!ASDF_QRY_PersonDocuments(
      pagingInfo: a!pagingInfo(
        startIndex: fv!item,
        batchSize: 1
      )
    ).data, 1)
  ),
  
  
  {}
)

This takes about 1 second to execute for a table with just shy of 700,000 entries. I imagine (depending on the number of samples desired) that it would scale up to your 25M scope decently(?) though i'd have a hard time testing that for you, as this is my prod system's largest table.

RE: Selecting a random sample from a large table

Soma — Thu, 10 Oct 2024 15:17:36 GMT

Try using DBMS_RANDOM.VALUE

SELECT *
     FROM (SELECT t1.column1, t2.column2, t3.column3, t4.column4, -- Specify the columns you need
                  DBMS_RANDOM.VALUE AS random_val -- Generate a random number for each row
           FROM table1 t1
           JOIN table2 t2 ON t1.common_key = t2.common_key -- Adjust this join condition
           JOIN table3 t3 ON t2.common_key = t3.common_key -- Adjust this join condition
           JOIN table4 t4 ON t3.common_key = t4.common_key -- Adjust this join condition
           ORDER BY DBMS_RANDOM.VALUE) -- Order rows randomly
     WHERE ROWNUM <= :sample_size';

RE: Selecting a random sample from a large table

kl0001 — Thu, 10 Oct 2024 12:52:05 GMT

Do you happen to have an example of how it can be done in a stored procedure? Do you use SAMPLE() in oracle and then filter on that or?

RE: Selecting a random sample from a large table

kl0001 — Thu, 10 Oct 2024 12:51:48 GMT

Do you happen to have an example of how it can be done in a stored procedure? Do you use SAMPLE() in oracle and then filter on that or?

RE: Selecting a random sample from a large table

Stefan Helzle — Thu, 10 Oct 2024 06:44:43 GMT

I suggest to implement this in a stored procedure and call it from Appian.

RE: Selecting a random sample from a large table

Soma — Thu, 10 Oct 2024 06:22:22 GMT

Please consider a stored procedure for calculation and logical manipulation. It gives more options and flexibility than a view. You can have the stored procedure to collect the sample and dump it in sample table. From which appian can take the ids to query and act upon.