AWS Signature 4 SHA256 Key Hashing

RE: AWS Signature 4 SHA256 Key Hashing

jonbond0002 — Thu, 31 Dec 2020 08:13:38 GMT

Hi Tejas,

I am getting signature error, when trying to do PUT request from Appian HTTP connected systems, But same key is working for GET request. Also when I am testing the PUT request in POSTMAN, it is working. Looks like AWS Signature Version 4 authentication have some issue.

Can you please help me to resolve this issue.

Regards,

Jon

RE: AWS Signature 4 SHA256 Key Hashing

sanin — Fri, 20 Mar 2020 08:41:23 GMT

Thank you, now it works!

However, description for Service field in Connected System Properties is somewhat misleading. It would be good to add something like:

"The service namespace that identifies the AWS product. For example, s3 for Amazon S3 resources."

RE: AWS Signature 4 SHA256 Key Hashing

Tejas Sakhardande — Thu, 19 Mar 2020 15:54:12 GMT

Can you try using just "sqs" as the value of the service field?

RE: AWS Signature 4 SHA256 Key Hashing

sanin — Thu, 19 Mar 2020 15:49:33 GMT

Hi,

value is full ARN string: "arn:aws:sqs:eu-central-1:XXXXXXXXXXXX:sqsname.fifo"

I have replaced here my AWS account id with X's.

RE: AWS Signature 4 SHA256 Key Hashing

Tejas Sakhardande — Thu, 19 Mar 2020 13:37:55 GMT

Hi,
Can you let me know what is the value of the 'Service' field in your Connected System?

RE: AWS Signature 4 SHA256 Key Hashing

sanin — Thu, 19 Mar 2020 10:01:36 GMT

Hi Tejas Sakhardande,

I am experiencing some difficulties connecting to AWS SQS using new AWS Signature Version 4 authentication.

It looks like Service part of the signature (extracted from ARN) is not correct.

My ARN: arn:aws:sqs:eu-central-1:XXXXXXXXXXXX:sqsname.fifo

The error message: <Error><Type>Sender</Type><Code>SignatureDoesNotMatch</Code><Message>Credential should be scoped to correct service: 'sqs'. </Message><Detail/></Error>

RE: AWS Signature 4 SHA256 Key Hashing

Tejas Sakhardande — Mon, 18 Nov 2019 16:43:04 GMT

As an update to this post, we're excited to announce that we've added support for AWS Signature Version 4 authentication to HTTP connected systems in Appian. You should no longer require the plugins in order to connect Appian to an Amazon service which uses the AWS Signature Version 4 authentication.

Let us know if you have any questions or concerns!

RE: AWS Signature 4 SHA256 Key Hashing

Eliot Gerson — Wed, 22 May 2019 16:24:14 GMT

Glad to hear the outbound call is working!! What sort of function would you need in order to decrypt the response?

For the second part, I think you're correct that you'll still need authentication for any incoming calls to Appian that aren't a direct response to a synchronous call.

RE: AWS Signature 4 SHA256 Key Hashing

Rick Bekker — Wed, 22 May 2019 09:36:19 GMT

It works, thanks for you help! For the response back from Buckaroo I also need to decrypt (un-hash) the message. And the new challenge; they want to send a a-synchronized REST call back with the payment update without any basic authentication only with the hashed body, I'm trying to convince Buckaroo to use basic authentication.

RE: AWS Signature 4 SHA256 Key Hashing

Eliot Gerson — Mon, 15 Apr 2019 16:24:04 GMT

Sounds great! Version 2.3.0 should be available soon on the App Market and for deployment to cloud sites. That will have an "hmacsha256bytehash" method that will hopefully meet your needs (as well as a getAwsV4Signature, for those who needed an additional function to complete the aws workflow).

RE: AWS Signature 4 SHA256 Key Hashing

Rick Bekker — Mon, 15 Apr 2019 11:04:46 GMT

I think the steps that Josh describes for me are correct. If we can have an addition on the current plugin that would be great.

RE: AWS Signature 4 SHA256 Key Hashing

Eliot Gerson — Thu, 11 Apr 2019 20:39:33 GMT

I think I should be able to add that capability to the cryptography hash plugin. I don't want it to get too unwieldy, but at the same time, it seems nice to have these things in one centralized location, and by adding it to the app market plugin, it promotes easy reuse by other members of the community.

To that end, can you confirm that the four steps that Josh outlined sound like the right steps for you? If so, I'll go ahead and add a function that does what he described.

Also, I've updated the cryptography plugin with the last step of the AWS signature process, so if anyone is looking for that, keep an eye out for version 2.2.0 or above. (jeffreyl946 , that will be the version that you probably want. once it's ready, you can deploy it to your cloud sites via the admin console.) I wound up making the method ever so slightly different than the example Josh gave, so you may need to modify your own SAIL code slightly if you're using Josh's as a reference.

RE: AWS Signature 4 SHA256 Key Hashing

Josh — Thu, 11 Apr 2019 17:19:20 GMT

Looking at the code in the link Rick provided, it looks to be more simple than the AWS V4 signature. Here's the steps I see for the Buckaroo use case:

Convert the secretKey from string to byte array
Convert the dataToSign from string to byte array
Use secretKeyBytes to "sign" dataToSignBytes using hmacsha256 (output needs to be byte array).
Convert signatureBytes to base64 encoded string (not a hex encoded string like awsV4)

As for the AWS V4 Signature, here is the (verified working) class I came up with to produce the signature hex string. This string then gets added to the authorization header in the actual request:

package com.commute.aws.signature;

import java.util.Map;

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;

import org.apache.commons.lang.StringUtils;
import org.apache.log4j.Logger;

import com.appiancorp.exceptions.InsufficientPrivilegesException;
import com.appiancorp.services.ServiceContext;
import com.appiancorp.suiteapi.common.exceptions.AppianException;
import com.appiancorp.suiteapi.common.exceptions.ErrorCode;
import com.appiancorp.suiteapi.expression.annotations.Category;
import com.appiancorp.suiteapi.expression.annotations.Function;
import com.appiancorp.suiteapi.expression.annotations.Parameter;
import com.appiancorp.suiteapi.security.external.SecureCredentialsStore;

@Category("awsSignatureFunctionsCategory")
public class AwsSignatureGenerator {

	private static final Logger LOG = Logger.getLogger(AwsSignatureGenerator.class);

	@Function
	public static String getAWSV4Signature(ServiceContext sc, SecureCredentialsStore scs,
			@Parameter String scsExternalSystemKey, @Parameter String scsFieldKey, @Parameter String dateStamp,
			@Parameter String regionName, @Parameter String serviceName, @Parameter String stringToSign)
			throws Exception {

		String key = getCryptoKey(scs, scsExternalSystemKey, scsFieldKey);

		/* 1. Get signature key */
		byte[] signatureKey = getSignatureKey(key, dateStamp, regionName, serviceName);
		/* 2. Calculate signature */
		byte[] signature = hmacSHA256(stringToSign, signatureKey);
		/* 3. Encode signature to hex string and return value */
		String stringHexSignature = bytesToHexString(signature);
		return stringHexSignature;
	}

	private static byte[] hmacSHA256(String data, byte[] key) throws Exception {
		String algorithm = "HmacSHA256";
		Mac mac = Mac.getInstance(algorithm);
		mac.init(new SecretKeySpec(key, algorithm));
		return mac.doFinal(data.getBytes("UTF-8"));
	}
	
	private static byte[] getSignatureKey(String key, String dateStamp, String regionName, String serviceName)
			throws Exception {
		byte[] kSecret = ("AWS4" + key).getBytes("UTF-8");
		byte[] kDate = hmacSHA256(dateStamp, kSecret);
		byte[] kRegion = hmacSHA256(regionName, kDate);
		byte[] kService = hmacSHA256(serviceName, kRegion);
		byte[] kSigning = hmacSHA256("aws4_request", kService);

		return kSigning;
	}
	
	private static String bytesToHexString(byte[] bytes) {
		StringBuilder sb = new StringBuilder();
		for (byte b : bytes) {
			sb.append(String.format("%02x", b));
		}
		return sb.toString();
	}

	private static String getCryptoKey(SecureCredentialsStore scs, String scsExternalSystemKey, String scsFieldKey)
			throws AppianException {
		if (StringUtils.isNotEmpty(scsExternalSystemKey)) {
			try {
				// Get Secure Credential Store
				Map<String, String> credentials = scs.getSystemSecuredValues(scsExternalSystemKey);
				if (!credentials.containsKey(scsFieldKey)) {
					LOG.error("Field " + scsFieldKey + " does not exist in Secure Credential Store " + scsExternalSystemKey);
					throw new AppianException(ErrorCode.EXTERNAL_SYSTEM_CONFIGURATION_INVALID_ATTR_NAME, scsFieldKey);
				}
				// Return key
				return credentials.get(scsFieldKey);
			} catch (InsufficientPrivilegesException e) {
				throw new AppianException(ErrorCode.EXTERNAL_SYSTEM_NOT_FOUND_INSUFFICIENT_PRIVILEGES, e);
			}
		} else {
			throw new AppianException(ErrorCode.EXTERNAL_SYSTEM_NOT_FOUND_INSUFFICIENT_PRIVILEGES,
					scsExternalSystemKey);
		}
	}
}

For creating the "string to sign" here is the expression rule we are using:

/* 
  Steps for creating an AWS V4 Signature:
    1. Create a Canonical Request: https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
       Canonical Request Format = 
        [HTTPMethod]\n
        [CanonicalURI]\n
        [CanonicalQueryString]\n
        [CanonicalHeaders]\n
        [SignedHeaders]\n
        [HashedPayload]
        
    2. Create a String to Sign: https://docs.aws.amazon.com/general/latest/gr/sigv4-create-string-to-sign.html
       StringToSign =
        Algorithm]\n
        [RequestDateTime]\n
        [CredentialScope]\n
        [HashedCanonicalRequest]
        
    3. Calculate the Signature: https://docs.aws.amazon.com/general/latest/gr/sigv4-calculate-signature.html
    
    4. Create Authorization Header value: https://docs.aws.amazon.com/general/latest/gr/sigv4-add-signature-to-request.html
       [algorithm] Credential=[accessKeyId]/[credentialScope], SignedHeaders=[signedHeaders], Signature=[signature]
*/
with(
  local!dateStamp: rule!TRSY_formatXAmzDateTime(dateTime: ri!dateTime, isDateOnly: true),
  local!dateTimeStamp: rule!TRSY_formatXAmzDateTime(dateTime: ri!dateTime),
  /* CredentialScope = [dateStamp]/[aws-region]/[aws-service]/aws4_request */
  local!credentialScope: joinarray(
    {
      local!dateStamp,
      cons!TRSY_SQS_AWS_REGION,
      cons!TRSY_SQS_SERVICE_NAME,
      cons!TRSY_SQS_AWS_SIGNATURE_VERSION
    },
    "/"
  ),
  /* Step 1: Canonical request */
  local!canonicalRequest: with(
    local!hashedPayload: sha256hash(ri!body),
    local!headers: joinarray({
      "content-length:"&lenb(ri!body),
      "content-type:text/plain; charset=UTF-8",
      "host:"&cons!TRSY_SQS_HOST,
      "x-amz-content-sha256:" & local!hashedPayload,
      "x-amz-date:" & local!dateTimeStamp
    }, char(10)),
    joinarray(
      {
        "POST",
        cons!TRSY_SQS_URI,
        "Action="&ri!actionParameter&"&MessageGroupId="&ri!messageGroupIdParameter,
        local!headers&char(10),
        cons!TRSY_SQS_AWS_SIGNED_HEADERS,
        local!hashedPayload
      },
      char(10)
    )
  ),
  /* Step 2: String to sign */
  local!stringToSign: joinarray(
    {
      cons!TRSY_SQS_SIGNING_ALGORITHM,
      local!dateTimeStamp,
      local!credentialScope,
      sha256hash(local!canonicalRequest)
    },
    char(10)
  ),
  /* Step 3: Signature */
  local!signature: getawsv4signature(
    scsExternalSystemKey: cons!TRSY_SQS_SCSFIELD_EXTERNALFIELD, 
    scsFieldKey: cons!TRSY_SQS_SCSFIELD_FIELDNAME_SECRETACCESSKEY, 
    dateStamp: local!dateStamp, 
    regionName: cons!TRSY_SQS_AWS_REGION, 
    serviceName: cons!TRSY_SQS_SERVICE_NAME, 
    stringToSign:local!stringToSign
  ),
  /* Step 4: Authorization header value */
  concat(
    cons!TRSY_SQS_SIGNING_ALGORITHM,
    " ",
    joinarray(
      {
        "Credential="&cons!TRSY_SQS_SECRET_ACCESS_KEY_ID&"/"&local!credentialScope,
        "SignedHeaders="&cons!TRSY_SQS_AWS_SIGNED_HEADERS,
        "Signature="&local!signature
      },
      ", "
    )
  )
)

The TRSY_formatXAmzDateTime rule is just a helper to format the date/time into the format AWS requires:

//TRSY_formatXAmzDateTime
if( 
  or(
    rule!APN_isBlank(ri!isDateOnly),
    not(ri!isDateOnly)
  ),
  text(gmt(ri!dateTime), "yyyymmddThhmmss")&"Z",
  text(gmt(ri!dateTime), "yyyymmdd")
)

RE: AWS Signature 4 SHA256 Key Hashing

Eliot Gerson — Thu, 11 Apr 2019 16:00:08 GMT

Are you looking for a method that will take in a byte array (as the key) and a string (the value to sign), and return a hex string (using the byte array to sign the string)? Or are you also needing to generate the key itself?

Josh , I'm trying to figure out if this is the same problem you were having. Can you tell? It seems like one possible solution (if I can do it) would be to 1) give a snippet of SAIL that would take a hex string and convert it to binary 2) add a hmacsha256(bytes, string) method that you could then pass that binary string to. Does that sound right?

RE: AWS Signature 4 SHA256 Key Hashing

Rick Bekker — Thu, 11 Apr 2019 08:01:37 GMT

We are working on an integration with Buckaroo, Buckaroo needs a secret key in the header.

The key also need a hash on byte level, we are almost there in creating the expression but there is one problem.

The functionality that is missing is in the current crypto plugin is the hmcasha256 hash on byte level instead of string level.

Buckaroo documentation: https://dev.buckaroo.nl/Apis/Description/json#codeexampleincsharp

example code C# line 63:

var secretKeyByteArray = Encoding.UTF8.GetBytes(SecretKey)

Can you help us or give us pointers regarding the hash?

RE: AWS Signature 4 SHA256 Key Hashing

Eliot Gerson — Tue, 09 Apr 2019 18:53:59 GMT

That's great! One of the reasons we like to include the source code in our plugins is so that people can learn from them and build off them. I'm glad to hear you were able to do just that.

RE: AWS Signature 4 SHA256 Key Hashing

Josh — Tue, 09 Apr 2019 18:28:50 GMT

In all honesty, I was a bit impatient and wasn't sure if you would have time to make other changes, so I created my own plugin which basically does as you've described. I modified the getSignatureKey to be a private method that returns the byte array and then added a new method for creating the signature, returning it as a hex encoded string. I also added the use of secure credential store. I'm still waiting for it to be deployed in our dev environment so I can test it fully. Once I've tested it, I'd be happy to share the source code so you can implement it in the Cryptography Hash Functions plugin.

RE: AWS Signature 4 SHA256 Key Hashing

Eliot Gerson — Tue, 09 Apr 2019 18:13:06 GMT

Is there any point in the process where we get a result that isn't a byte array? For example, once the string is signed, is it a string, or a byte array?

Basically, the challenge is that we do technically have a "binary" type in Appian, but it's rather old, and we don't have a type converter for it (which is why you can't, for example, use it as the type for a rule input.) Adding support for that would require me to make a change at the product level, which would be a much more involved effort.

So, the goal is to do all the byte operations in the plugin, and then return some other type back, like a string.

With that in mind, do you think it would work for you if the method had you additionally pass the text you want to sign, signed it with the hmacsha256hash function, and then returned the signed text (in... hex encoded form, I guess?)? Or does the signed text also need to remain in binary?

RE: AWS Signature 4 SHA256 Key Hashing

Josh — Tue, 09 Apr 2019 14:52:19 GMT

Eliot,

Thank you so much! I'm using v2.0 of the plugin and the getSignatureKey function appears to work as expected. There is a slight challenge though. After numerous attempts, I was unable to produce a signature that matched AWS samples. After doing some more research, it appears the "string to sign" must be signed using the byte array of the signature key, not the hex string. Here's an example I found (look at the calculateSignature method): https://www.javaquery.com/2016/01/aws-version-4-signing-process-complete.html

Would you have time to update the plugin with that additional piece?

RE: AWS Signature 4 SHA256 Key Hashing

jeffreyl946 — Sun, 07 Apr 2019 22:56:24 GMT

Thanks Eliot! This is really appreciated!

RE: AWS Signature 4 SHA256 Key Hashing

Eliot Gerson — Fri, 05 Apr 2019 23:52:13 GMT

Alright, so, final update for now: the 2.0.0 version, with the getSignatureKey method, should be available within 24 hours or so for download from the App Market, and/or installation to Cloud Sites through the Admin Console.

The 2.1.0 release, which has support for the secure credentials store, will likely be available at some point on Monday or Tuesday.

Once the plugin is installed, the function tooltip (the function will be "getSignatureKey", under the cryptography functions category) should answer most questions, but if any questions or concerns do come up, just let me know. jeffreyl946, fyi as well.

RE: AWS Signature 4 SHA256 Key Hashing

Eliot Gerson — Fri, 05 Apr 2019 22:12:07 GMT

The review is complete, but it might take a day or two for it to be available. So I think it should show up for you at some point on Monday. If you have cloud sites, you'll be able to add it through the Admin Console. Otherwise, you'll be able to download it through the App Market, and then follow the normal instructions to install a plugin.

Also, I saw your comment on the plugin's app market page asking about the secure credentials store. That's good thinking, so I'm adding that capability to the plugin (just for the AWS signature function, for now, rather than for all the functions.) So, you'll actually want version 2.1.0 .

RE: AWS Signature 4 SHA256 Key Hashing

Eliot Gerson — Fri, 05 Apr 2019 21:20:00 GMT

ah, my apologies. I've edited my previous comment so as to not confuse any future viewers. The review process should be pretty quick, especially for something developed in-house, but I'll double check to see where things are at.

RE: AWS Signature 4 SHA256 Key Hashing

Josh — Fri, 05 Apr 2019 21:14:47 GMT

I'm not able to see any app submission. I get an error when I try the link you provided. My guess is I can only see those that I submit. How long does the review process take?

RE: AWS Signature 4 SHA256 Key Hashing

Eliot Gerson — Fri, 05 Apr 2019 20:49:12 GMT

I went ahead and did it as an update to the cryptography hash functions plugin. The official update is technically pending review, but should be available soon.