Replace special characters in JSON response

Hi guys,

We have an issue where an API push is failing because we are receiving JSON data and then pushing this data straight back out to the API. The API can output special characters but won't accept them. To get around this, we need to find a way to replace certain special characters from value fields with encoded characters. Example:

Received JSON:

{test1: "this is a test with some valid characters",

test2: "this is a test with some invalid characters , ! £"}

Pushed JSON:

{test1: "this is a test with some valid characters",

test2: "this is a test with some invalid characters .co .ex .po"}

So in this instance the aim would be to replace those special characters at the end of test2. 

Is there a way to do this, potentially with REGEX or a replace string? It would need to check all the value fields and replace any of the special characters in there with some encoding we've defined.

Thanks for any help.

  Discussion posts and replies are publicly visible

Parents
  • REGEX would probably work, but you could also use out-of-the-box Appian functions.

    The stripwith() function will remove any characters you define as invalid. So if you wanted to specifically remove the £ symbol, you could use stripwith() like this:

    stripwith(
        [JSON Here],
        "£"
    )

    You can also use the cleanwith() function which does the reverse: it only keeps the characters you define as valid and removes all others. So for example, if you only want to allow letters and numbers, you could use cleanwith() like this:

    cleanwith(
        [JSON Here],
        "abcdefghijklmnopqrstuvwxyz0123456789"
    )

    Both of the methods described above remove the problem characters. If you want to replace them with something else, you can also try the replace function.

  • Thanks for the reply. That would be perfect, but we need to try and find a way to only apply these rules to the value fields of the JSON. One of the characters that can't be accepted by the RAML for values is colon, for example. Colons are obviously used as key value separators for JSON, which is fine, but it can't be included in any of the value fields in the JSON. So we only want it stripped inside value fields, not the JSON as a whole.

    Is there a way to index specific keys in the JSON, update the values for those keys, and then return the entire JSON body as a whole?

Reply
  • Thanks for the reply. That would be perfect, but we need to try and find a way to only apply these rules to the value fields of the JSON. One of the characters that can't be accepted by the RAML for values is colon, for example. Colons are obviously used as key value separators for JSON, which is fine, but it can't be included in any of the value fields in the JSON. So we only want it stripped inside value fields, not the JSON as a whole.

    Is there a way to index specific keys in the JSON, update the values for those keys, and then return the entire JSON body as a whole?

Children
  • Hmmm, do you know the structure of the response and know which fields you want to index? If you do, you can use a!fromJson() to convert your JSON to an Appian dictionary. Then, you could use the index() function to return certain fields that you can then run the cleanwith() or stripwith() on. Then, do the reverse - convert back to JSON and send out the response.

    You might have to weigh this against regex too. At some point, it may be easier to just use regex rather than all these manipulations in Appian. To use regex, you have to install a plugin: https://community.appian.com/b/appmarket/posts/regular-expression-functions

    Unfortunately, I'm not much of a regex coder so I'm not sure what syntax would be best, but someone else on here might be able to help you with that syntax Slight smile

  • Thanks for the helpful suggestions Peter, I went along the lines of what you mentioned here. In case anyone else comes across this in the future, this is what we have ended up going with:

    Created an expression rule that takes in the jsonBody and generates a local variable for each field we wanted to retrieve and encode. To populate the variables we went with a!jsonPath (similar to index but specific to JSON) which allows you to provide the field key and get the associated field value, converted this using a!fromJson, and then converted each variable to a uniform string (which allowed the next functions to manipulate the values while also maintaining the array structure of the retrieved fields, as there may be multiple instances of a single field key). We combined these local variable fields into a single array using union (within a reject function to account for fields not being present). Finally, we used the substitute function for each value in this combined array to find the value in the original JSON body and replace it with the encoded version (directly outputted into the substitute parameter from an expression rule we made for our encoding). We used this substitute within a reduce function so that it would apply recursively (as foreach outputs one instance of the entire JSON body for each substitute, with only the currently iterated value encoded). The end result is the entire JSON body, updated with each necessary value encoded.

    As a side note, we would definitely have went with either cleanwith() or stripwith() as suggested, but I forgot to say we need to retain the special characters but push them in an acceptable format. So instead of removing them, we replace them with different values which we've defined.

    Sorry that the above explanation is a little convoluted, if anyone does come across this and needs any clarification on any of the above in the future I'd be happy to do my best to answer.

    Thanks again ,  and  for your responses.