using JSONpath in a query

RE: using JSONpath in a query

vishalk2742 — Tue, 05 Jul 2022 11:38:13 GMT

I am working on the similar use case, any luck with JSON_TABLE function. I see Appian does not have much to explore but we have some options from DB and Java API which can be useful for this particular use case.
Appreciate if you have share any tips / workaround you did for this use case

RE: using JSONpath in a query

Marco — Tue, 25 Jan 2022 18:18:50 GMT

I hadn't thought of that. Hmm... In this particular case the json objects are uniform so the table could be set up to take everything in, I suppose. I'd probably need to get permission to do things that way though. I'll take it up with them to see. Thanks for the idea.

RE: using JSONpath in a query

Stewart Burchell — Tue, 25 Jan 2022 17:03:53 GMT

Are you genuinely constrained in how the data is populated into the DB? For example: the current process that loads the data (as a PK, JKSON blob pair) - could that be expanded to extract the data from the incoming JSON and load into a normalized model that you could then implement as you were initially hoping to do so using standard query entity calls? In short: is there an opportunity to move the problem upstream to the database load process, rather than where you're currently trying to solve it?

RE: using JSONpath in a query

Marco — Tue, 25 Jan 2022 16:59:58 GMT

We're using MariaDB. We've been looking into using json_table, but haven't figured out how to get it to work yet. Thanks for the tip.

RE: using JSONpath in a query

Marco — Tue, 25 Jan 2022 16:59:28 GMT

Thanks, that's another option we've been looking into. Haven't gotten it to function yet, but it seems promising at least.

RE: using JSONpath in a query

Chris — Tue, 25 Jan 2022 16:32:42 GMT

I would lean toward the DB side as well as you may have functions available that could be used to create a view to make this much easier on the Appian side. What DB/version are you on? For example in MSSQL you may be able to take advantage of OPENJSON(), or JSON_VALUE / JSON_TABLE in Oracle, etc.

RE: using JSONpath in a query

Stewart Burchell — Tue, 25 Jan 2022 08:51:36 GMT

A possible alternative, depending on the database you're working with, is that the DB might support JSON functions and you could use that to conduct the relevant filtering. That would mean you calling a Stored Procedure and either designing it in such a way as to be generic to support any type of query, or having one Stored Proc per concrete query type you know it needs to support. But, as I said, entirely dependent upon your DB's support for JSON functions.

RE: using JSONpath in a query

Mike Schmitt — Mon, 24 Jan 2022 21:31:56 GMT

[quote userid="71398" url="~/discussions/f/plug-ins/23521/using-jsonpath-in-a-query/90116#90116"]As for filtering down to a more reasonable set of rows, there are no criteria that I can filter outside of the json stuff [/quote]

I was thinking along the lines of pre-filtering based on the JSON information (if there's anything you can directly filter on, like the "gender" example we did before). If not then you'll have to skip this part.

[quote userid="71398" url="~/discussions/f/plug-ins/23521/using-jsonpath-in-a-query/90116#90116"]read only grids seem to handle whatever logic is needed to bring in the data in smaller batches to see on individual pages[/quote]

This is all handled on the back end by a!queryEntity with paging info. Read-only grids are merely a visual manifestation of this. Problem being, if there's no "text includes" filtering you can do on your JSON column, and no way to otherwise break this data out to other proper columns, there's not much you can do that will allow you to do very much inside of a!queryEntity. At this point I'd be considering whether it's possible to build a view that breaks out some of your json data (at least for columns that will always exist) into discrete columns. MariaDB has various JSON functionality available but as far as this specific thing, I haven't necessairly tried this.

RE: using JSONpath in a query

Marco — Mon, 24 Jan 2022 20:42:20 GMT

Believe me, if I could store the data in traditional db columns, I would. It would make everything a lot easier. Unfortunately, the format of the data is out of my hands.

As for filtering down to a more reasonable set of rows, there are no criteria that I can filter outside of the json stuff as the only two fields on the table are the ID and the json object.

When you pass a query into them, read only grids seem to handle whatever logic is needed to bring in the data in smaller batches to see on individual pages. Is there some way to apply that sort of logic elsewhere? Loop through the entire data, but in smaller, more manageable batches?

RE: using JSONpath in a query

Mike Schmitt — Mon, 24 Jan 2022 20:15:17 GMT

For that sort of thing to work with a!queryEntity, you'd need to store that data in traditional DB columns, unless you're prepared to do something closer to what Stewart suggested and pre-query a bunch of rows and manually filter through them on the Appian side. One note here would be, depending on your use case you could potentially limit your pre-query to a more reasonable set of rows by filtering first on other criteria.

RE: using JSONpath in a query

Marco — Mon, 24 Jan 2022 19:51:07 GMT

I ran into an issue trying that. I may need to compare values as part of the filtering process. For example, I might want to see if their salary field is above a particular threshold. Includes wouldn't be sufficient in such a case.

RE: using JSONpath in a query

Marco — Mon, 24 Jan 2022 19:06:36 GMT

I suppose that's one way to do it. I was just being told to make use of JSONpath as a possibly more performant solution. But if it won't work, it won't work.

RE: using JSONpath in a query

Marco — Mon, 24 Jan 2022 19:04:53 GMT

The table only contains 2 fields at the moment: a primary key and a json object, so that's not necessary.

The problem is I can't seem to apply jsonpath without getting everything first and once I hit around 3,000 rows, I run out of memory.

RE: using JSONpath in a query

Stewart Burchell — Mon, 24 Jan 2022 18:04:39 GMT

Could your initial query ONLY return the JSON column and the Primary Key (and any other applicable selection criteria that will work with standard filters) , and then you filter out those rows that don't match your JSONpath expression, and then make a second query to get all of the other data you need? (yes, it's two separate queries, but might perform better than fetching the full length and breadth of the table and then filtering locally)

RE: using JSONpath in a query

Mike Schmitt — Mon, 24 Jan 2022 17:52:45 GMT

I don't believe query filters can make use of jsonPath whatsoever. Wouldn't it generally be sufficient to use an "includes" filter for the json attribute, i.e. where the json column includes `"gender":"female"` ?