AWS Signature 4 SHA256 Key Hashing

I'm sorry to hear this has turned frustrating for you guys. To help get you started, I went ahead and put together a custom function plugin that will return the hex encoded signature key. I'll see if I can get that uploaded to the app market soon so that you can add it to your environments. What are you doing with the key once you generate it? Have you already determined whether, once the key is generated, you can then do what you need?

Eliot, that would be fantastic!  My plan is to use the key to create the signature that gets appended to the Authorization header.  To do that, I'll use the hmacsha256hash function from the Cryptographic Hash Functions plugin.  I've already created the rules to build the required string that gets signed with the signature key.

The review is complete, but it might take a day or two for it to be available. So I think it should show up for you at some point on Monday. If you have cloud sites, you'll be able to add it through the Admin Console. Otherwise, you'll be able to download it through the App Market, and then follow the normal instructions to install a plugin. 

Also, I saw your comment on the plugin's app market page asking about the secure credentials store. That's good thinking, so I'm adding that capability to the plugin (just for the AWS signature function, for now, rather than for all the functions.) So, you'll actually want version 2.1.0 .

Alright, so, final update for now: the 2.0.0 version, with the getSignatureKey method, should be available within 24 hours or so for download from the App Market, and/or installation to Cloud Sites through the Admin Console.

The 2.1.0 release, which has support for the secure credentials store, will likely be available at some point on Monday or Tuesday.

Once the plugin is installed, the function tooltip (the function will be "getSignatureKey", under the cryptography functions category) should answer most questions, but if any questions or concerns do come up, just let me know. jeffreyl946, fyi as well.  

Thanks Eliot! This is really appreciated!

Eliot,

Thank you so much!  I'm using v2.0 of the plugin and the getSignatureKey function appears to work as expected. There is a slight challenge though.  After numerous attempts, I was unable to produce a signature that matched AWS samples.  After doing some more research, it appears the "string to sign" must be signed using the byte array of the signature key, not the hex string.  Here's an example I found (look at the calculateSignature method): https://www.javaquery.com/2016/01/aws-version-4-signing-process-complete.html

Would you have time to update the plugin with that additional piece?

Is there any point in the process where we get a result that isn't a byte array? For example, once the string is signed, is it a string, or a byte array? 

Basically, the challenge is that we do technically have a "binary" type in Appian, but it's rather old, and we don't have a type converter for it (which is why you can't, for example, use it as the type for a rule input.) Adding support for that would require me to make a change at the product level, which would be a much more involved effort. 

So, the goal is to do all the byte operations in the plugin, and then return some other type back, like a string. 

With that in mind, do you think it would work for you if the method had you additionally pass the text you want to sign, signed it with the hmacsha256hash function, and then returned the signed text (in... hex encoded form, I guess?)? Or does the signed text also need to remain in binary?

In all honesty, I was a bit impatient and wasn't sure if you would have time to make other changes, so I created my own plugin which basically does as you've described.  I modified the getSignatureKey to be a private method that returns the byte array and then added a new method for creating the signature, returning it as a hex encoded string.  I also added the use of secure credential store.  I'm still waiting for it to be deployed in our dev environment so I can test it fully. Once I've tested it, I'd be happy to share the source code so you can implement it in the Cryptography  Hash Functions plugin.

That's great! One of the reasons we like to include the source code in our plugins is so that people can learn from them and build off them. I'm glad to hear you were able to do just that. 

We are working on an integration with Buckaroo, Buckaroo needs a secret key in the header.

The key also need a hash on byte level, we are almost there in creating the expression but there is one problem.

The functionality that is missing is in the current crypto plugin is the hmcasha256 hash on byte level instead of string level.

Buckaroo documentation: https://dev.buckaroo.nl/Apis/Description/json#codeexampleincsharp

example code C# line 63: 

var secretKeyByteArray = Encoding.UTF8.GetBytes(SecretKey)

Can you help us or give us pointers regarding the hash?

Are you looking for a method that will take in a byte array (as the key) and a string (the value to sign), and return a hex string (using the byte array to sign the string)? Or are you also needing to generate the key itself? 

Josh , I'm trying to figure out if this is the same problem you were having. Can you tell? It seems like one possible solution (if I can do it) would be to 1) give a snippet of SAIL that would take a hex string and convert it to binary 2) add a hmacsha256(bytes, string) method that you could then pass that binary string to. Does that sound right?

Looking at the code in the link Rick provided, it looks to be more simple than the AWS V4 signature.  Here's the steps I see for the Buckaroo use case:

1. Convert the secretKey from string to byte array
2. Convert the dataToSign from string to byte array
3. Use secretKeyBytes to "sign" dataToSignBytes using hmacsha256 (output needs to be byte array).
4. Convert signatureBytes to base64 encoded string (not a hex encoded string like awsV4)

As for the AWS V4 Signature, here is the (verified working) class I came up with to produce the signature hex string.  This string then gets added to the authorization header in the actual request:

package com.commute.aws.signature;

import java.util.Map;

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;

import org.apache.commons.lang.StringUtils;
import org.apache.log4j.Logger;

import com.appiancorp.exceptions.InsufficientPrivilegesException;
import com.appiancorp.services.ServiceContext;
import com.appiancorp.suiteapi.common.exceptions.AppianException;
import com.appiancorp.suiteapi.common.exceptions.ErrorCode;
import com.appiancorp.suiteapi.expression.annotations.Category;
import com.appiancorp.suiteapi.expression.annotations.Function;
import com.appiancorp.suiteapi.expression.annotations.Parameter;
import com.appiancorp.suiteapi.security.external.SecureCredentialsStore;

@Category("awsSignatureFunctionsCategory")
public class AwsSignatureGenerator {

	private static final Logger LOG = Logger.getLogger(AwsSignatureGenerator.class);

	@Function
	public static String getAWSV4Signature(ServiceContext sc, SecureCredentialsStore scs,
			@Parameter String scsExternalSystemKey, @Parameter String scsFieldKey, @Parameter String dateStamp,
			@Parameter String regionName, @Parameter String serviceName, @Parameter String stringToSign)
			throws Exception {

		String key = getCryptoKey(scs, scsExternalSystemKey, scsFieldKey);

		/* 1. Get signature key */
		byte[] signatureKey = getSignatureKey(key, dateStamp, regionName, serviceName);
		/* 2. Calculate signature */
		byte[] signature = hmacSHA256(stringToSign, signatureKey);
		/* 3. Encode signature to hex string and return value */
		String stringHexSignature = bytesToHexString(signature);
		return stringHexSignature;
	}

	private static byte[] hmacSHA256(String data, byte[] key) throws Exception {
		String algorithm = "HmacSHA256";
		Mac mac = Mac.getInstance(algorithm);
		mac.init(new SecretKeySpec(key, algorithm));
		return mac.doFinal(data.getBytes("UTF-8"));
	}
	
	private static byte[] getSignatureKey(String key, String dateStamp, String regionName, String serviceName)
			throws Exception {
		byte[] kSecret = ("AWS4" + key).getBytes("UTF-8");
		byte[] kDate = hmacSHA256(dateStamp, kSecret);
		byte[] kRegion = hmacSHA256(regionName, kDate);
		byte[] kService = hmacSHA256(serviceName, kRegion);
		byte[] kSigning = hmacSHA256("aws4_request", kService);

		return kSigning;
	}
	
	private static String bytesToHexString(byte[] bytes) {
		StringBuilder sb = new StringBuilder();
		for (byte b : bytes) {
			sb.append(String.format("%02x", b));
		}
		return sb.toString();
	}

	private static String getCryptoKey(SecureCredentialsStore scs, String scsExternalSystemKey, String scsFieldKey)
			throws AppianException {
		if (StringUtils.isNotEmpty(scsExternalSystemKey)) {
			try {
				// Get Secure Credential Store
				Map<String, String> credentials = scs.getSystemSecuredValues(scsExternalSystemKey);
				if (!credentials.containsKey(scsFieldKey)) {
					LOG.error("Field " + scsFieldKey + " does not exist in Secure Credential Store " + scsExternalSystemKey);
					throw new AppianException(ErrorCode.EXTERNAL_SYSTEM_CONFIGURATION_INVALID_ATTR_NAME, scsFieldKey);
				}
				// Return key
				return credentials.get(scsFieldKey);
			} catch (InsufficientPrivilegesException e) {
				throw new AppianException(ErrorCode.EXTERNAL_SYSTEM_NOT_FOUND_INSUFFICIENT_PRIVILEGES, e);
			}
		} else {
			throw new AppianException(ErrorCode.EXTERNAL_SYSTEM_NOT_FOUND_INSUFFICIENT_PRIVILEGES,
					scsExternalSystemKey);
		}
	}
}

For creating the "string to sign"  here is the expression rule we are using:

/* 
  Steps for creating an AWS V4 Signature:
    1. Create a Canonical Request: https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
       Canonical Request Format = 
        [HTTPMethod]\n
        [CanonicalURI]\n
        [CanonicalQueryString]\n
        [CanonicalHeaders]\n
        [SignedHeaders]\n
        [HashedPayload]
        
    2. Create a String to Sign: https://docs.aws.amazon.com/general/latest/gr/sigv4-create-string-to-sign.html
       StringToSign =
        Algorithm]\n
        [RequestDateTime]\n
        [CredentialScope]\n
        [HashedCanonicalRequest]
        
    3. Calculate the Signature: https://docs.aws.amazon.com/general/latest/gr/sigv4-calculate-signature.html
    
    4. Create Authorization Header value: https://docs.aws.amazon.com/general/latest/gr/sigv4-add-signature-to-request.html
       [algorithm] Credential=[accessKeyId]/[credentialScope], SignedHeaders=[signedHeaders], Signature=[signature]
*/
with(
  local!dateStamp: rule!TRSY_formatXAmzDateTime(dateTime: ri!dateTime, isDateOnly: true),
  local!dateTimeStamp: rule!TRSY_formatXAmzDateTime(dateTime: ri!dateTime),
  /* CredentialScope = [dateStamp]/[aws-region]/[aws-service]/aws4_request */
  local!credentialScope: joinarray(
    {
      local!dateStamp,
      cons!TRSY_SQS_AWS_REGION,
      cons!TRSY_SQS_SERVICE_NAME,
      cons!TRSY_SQS_AWS_SIGNATURE_VERSION
    },
    "/"
  ),
  /* Step 1: Canonical request */
  local!canonicalRequest: with(
    local!hashedPayload: sha256hash(ri!body),
    local!headers: joinarray({
      "content-length:"&lenb(ri!body),
      "content-type:text/plain; charset=UTF-8",
      "host:"&cons!TRSY_SQS_HOST,
      "x-amz-content-sha256:" & local!hashedPayload,
      "x-amz-date:" & local!dateTimeStamp
    }, char(10)),
    joinarray(
      {
        "POST",
        cons!TRSY_SQS_URI,
        "Action="&ri!actionParameter&"&MessageGroupId="&ri!messageGroupIdParameter,
        local!headers&char(10),
        cons!TRSY_SQS_AWS_SIGNED_HEADERS,
        local!hashedPayload
      },
      char(10)
    )
  ),
  /* Step 2: String to sign */
  local!stringToSign: joinarray(
    {
      cons!TRSY_SQS_SIGNING_ALGORITHM,
      local!dateTimeStamp,
      local!credentialScope,
      sha256hash(local!canonicalRequest)
    },
    char(10)
  ),
  /* Step 3: Signature */
  local!signature: getawsv4signature(
    scsExternalSystemKey: cons!TRSY_SQS_SCSFIELD_EXTERNALFIELD, 
    scsFieldKey: cons!TRSY_SQS_SCSFIELD_FIELDNAME_SECRETACCESSKEY, 
    dateStamp: local!dateStamp, 
    regionName: cons!TRSY_SQS_AWS_REGION, 
    serviceName: cons!TRSY_SQS_SERVICE_NAME, 
    stringToSign:local!stringToSign
  ),
  /* Step 4: Authorization header value */
  concat(
    cons!TRSY_SQS_SIGNING_ALGORITHM,
    " ",
    joinarray(
      {
        "Credential="&cons!TRSY_SQS_SECRET_ACCESS_KEY_ID&"/"&local!credentialScope,
        "SignedHeaders="&cons!TRSY_SQS_AWS_SIGNED_HEADERS,
        "Signature="&local!signature
      },
      ", "
    )
  )
)

The TRSY_formatXAmzDateTime rule is just a helper to format the date/time into the format AWS requires:

//TRSY_formatXAmzDateTime
if( 
  or(
    rule!APN_isBlank(ri!isDateOnly),
    not(ri!isDateOnly)
  ),
  text(gmt(ri!dateTime), "yyyymmddThhmmss")&"Z",
  text(gmt(ri!dateTime), "yyyymmdd")
)

Looking at the code in the link Rick provided, it looks to be more simple than the AWS V4 signature.  Here's the steps I see for the Buckaroo use case:

1. Convert the secretKey from string to byte array
2. Convert the dataToSign from string to byte array
3. Use secretKeyBytes to "sign" dataToSignBytes using hmacsha256 (output needs to be byte array).
4. Convert signatureBytes to base64 encoded string (not a hex encoded string like awsV4)

As for the AWS V4 Signature, here is the (verified working) class I came up with to produce the signature hex string.  This string then gets added to the authorization header in the actual request:

package com.commute.aws.signature;

import java.util.Map;

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;

import org.apache.commons.lang.StringUtils;
import org.apache.log4j.Logger;

import com.appiancorp.exceptions.InsufficientPrivilegesException;
import com.appiancorp.services.ServiceContext;
import com.appiancorp.suiteapi.common.exceptions.AppianException;
import com.appiancorp.suiteapi.common.exceptions.ErrorCode;
import com.appiancorp.suiteapi.expression.annotations.Category;
import com.appiancorp.suiteapi.expression.annotations.Function;
import com.appiancorp.suiteapi.expression.annotations.Parameter;
import com.appiancorp.suiteapi.security.external.SecureCredentialsStore;

@Category("awsSignatureFunctionsCategory")
public class AwsSignatureGenerator {

	private static final Logger LOG = Logger.getLogger(AwsSignatureGenerator.class);

	@Function
	public static String getAWSV4Signature(ServiceContext sc, SecureCredentialsStore scs,
			@Parameter String scsExternalSystemKey, @Parameter String scsFieldKey, @Parameter String dateStamp,
			@Parameter String regionName, @Parameter String serviceName, @Parameter String stringToSign)
			throws Exception {

		String key = getCryptoKey(scs, scsExternalSystemKey, scsFieldKey);

		/* 1. Get signature key */
		byte[] signatureKey = getSignatureKey(key, dateStamp, regionName, serviceName);
		/* 2. Calculate signature */
		byte[] signature = hmacSHA256(stringToSign, signatureKey);
		/* 3. Encode signature to hex string and return value */
		String stringHexSignature = bytesToHexString(signature);
		return stringHexSignature;
	}

	private static byte[] hmacSHA256(String data, byte[] key) throws Exception {
		String algorithm = "HmacSHA256";
		Mac mac = Mac.getInstance(algorithm);
		mac.init(new SecretKeySpec(key, algorithm));
		return mac.doFinal(data.getBytes("UTF-8"));
	}
	
	private static byte[] getSignatureKey(String key, String dateStamp, String regionName, String serviceName)
			throws Exception {
		byte[] kSecret = ("AWS4" + key).getBytes("UTF-8");
		byte[] kDate = hmacSHA256(dateStamp, kSecret);
		byte[] kRegion = hmacSHA256(regionName, kDate);
		byte[] kService = hmacSHA256(serviceName, kRegion);
		byte[] kSigning = hmacSHA256("aws4_request", kService);

		return kSigning;
	}
	
	private static String bytesToHexString(byte[] bytes) {
		StringBuilder sb = new StringBuilder();
		for (byte b : bytes) {
			sb.append(String.format("%02x", b));
		}
		return sb.toString();
	}

	private static String getCryptoKey(SecureCredentialsStore scs, String scsExternalSystemKey, String scsFieldKey)
			throws AppianException {
		if (StringUtils.isNotEmpty(scsExternalSystemKey)) {
			try {
				// Get Secure Credential Store
				Map<String, String> credentials = scs.getSystemSecuredValues(scsExternalSystemKey);
				if (!credentials.containsKey(scsFieldKey)) {
					LOG.error("Field " + scsFieldKey + " does not exist in Secure Credential Store " + scsExternalSystemKey);
					throw new AppianException(ErrorCode.EXTERNAL_SYSTEM_CONFIGURATION_INVALID_ATTR_NAME, scsFieldKey);
				}
				// Return key
				return credentials.get(scsFieldKey);
			} catch (InsufficientPrivilegesException e) {
				throw new AppianException(ErrorCode.EXTERNAL_SYSTEM_NOT_FOUND_INSUFFICIENT_PRIVILEGES, e);
			}
		} else {
			throw new AppianException(ErrorCode.EXTERNAL_SYSTEM_NOT_FOUND_INSUFFICIENT_PRIVILEGES,
					scsExternalSystemKey);
		}
	}
}

For creating the "string to sign"  here is the expression rule we are using:

/* 
  Steps for creating an AWS V4 Signature:
    1. Create a Canonical Request: https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
       Canonical Request Format = 
        [HTTPMethod]\n
        [CanonicalURI]\n
        [CanonicalQueryString]\n
        [CanonicalHeaders]\n
        [SignedHeaders]\n
        [HashedPayload]
        
    2. Create a String to Sign: https://docs.aws.amazon.com/general/latest/gr/sigv4-create-string-to-sign.html
       StringToSign =
        Algorithm]\n
        [RequestDateTime]\n
        [CredentialScope]\n
        [HashedCanonicalRequest]
        
    3. Calculate the Signature: https://docs.aws.amazon.com/general/latest/gr/sigv4-calculate-signature.html
    
    4. Create Authorization Header value: https://docs.aws.amazon.com/general/latest/gr/sigv4-add-signature-to-request.html
       [algorithm] Credential=[accessKeyId]/[credentialScope], SignedHeaders=[signedHeaders], Signature=[signature]
*/
with(
  local!dateStamp: rule!TRSY_formatXAmzDateTime(dateTime: ri!dateTime, isDateOnly: true),
  local!dateTimeStamp: rule!TRSY_formatXAmzDateTime(dateTime: ri!dateTime),
  /* CredentialScope = [dateStamp]/[aws-region]/[aws-service]/aws4_request */
  local!credentialScope: joinarray(
    {
      local!dateStamp,
      cons!TRSY_SQS_AWS_REGION,
      cons!TRSY_SQS_SERVICE_NAME,
      cons!TRSY_SQS_AWS_SIGNATURE_VERSION
    },
    "/"
  ),
  /* Step 1: Canonical request */
  local!canonicalRequest: with(
    local!hashedPayload: sha256hash(ri!body),
    local!headers: joinarray({
      "content-length:"&lenb(ri!body),
      "content-type:text/plain; charset=UTF-8",
      "host:"&cons!TRSY_SQS_HOST,
      "x-amz-content-sha256:" & local!hashedPayload,
      "x-amz-date:" & local!dateTimeStamp
    }, char(10)),
    joinarray(
      {
        "POST",
        cons!TRSY_SQS_URI,
        "Action="&ri!actionParameter&"&MessageGroupId="&ri!messageGroupIdParameter,
        local!headers&char(10),
        cons!TRSY_SQS_AWS_SIGNED_HEADERS,
        local!hashedPayload
      },
      char(10)
    )
  ),
  /* Step 2: String to sign */
  local!stringToSign: joinarray(
    {
      cons!TRSY_SQS_SIGNING_ALGORITHM,
      local!dateTimeStamp,
      local!credentialScope,
      sha256hash(local!canonicalRequest)
    },
    char(10)
  ),
  /* Step 3: Signature */
  local!signature: getawsv4signature(
    scsExternalSystemKey: cons!TRSY_SQS_SCSFIELD_EXTERNALFIELD, 
    scsFieldKey: cons!TRSY_SQS_SCSFIELD_FIELDNAME_SECRETACCESSKEY, 
    dateStamp: local!dateStamp, 
    regionName: cons!TRSY_SQS_AWS_REGION, 
    serviceName: cons!TRSY_SQS_SERVICE_NAME, 
    stringToSign:local!stringToSign
  ),
  /* Step 4: Authorization header value */
  concat(
    cons!TRSY_SQS_SIGNING_ALGORITHM,
    " ",
    joinarray(
      {
        "Credential="&cons!TRSY_SQS_SECRET_ACCESS_KEY_ID&"/"&local!credentialScope,
        "SignedHeaders="&cons!TRSY_SQS_AWS_SIGNED_HEADERS,
        "Signature="&local!signature
      },
      ", "
    )
  )
)

The TRSY_formatXAmzDateTime rule is just a helper to format the date/time into the format AWS requires:

//TRSY_formatXAmzDateTime
if( 
  or(
    rule!APN_isBlank(ri!isDateOnly),
    not(ri!isDateOnly)
  ),
  text(gmt(ri!dateTime), "yyyymmddThhmmss")&"Z",
  text(gmt(ri!dateTime), "yyyymmdd")
)