I am unable to change the Portal Start Page to the Rules tab via the URL /rules/

This is needed because the code that validates the URL uses the context of the request to form the URL meaning that when you type

/rules/GetRulesRoot.do

Appian will internally review what URL your browser has, let's say myappian.appiancorp.com/.../designer and based on that will create the following URL for validation:

myappian.appiancorp.com/.../GetRulesRoot.do

and execute something like this:

urlHomePage = new URL(url);
urlHomePage.getContent();

when Appian calls the getContent() method to verify this is a valid URL it will go through HTTPS which in your case is at the Web Server layer, if the certificate your Web Server is using is not installed per the instructions above then it won't find a valid certificate. You don't see the error in the application server log because the exception is caught and the stack trace is not printed but basically that is what is happening.

Given the URL is formed using the page context you cannot control this through custom.properties, but if you can bypass the web server temporarily you will also be able to set this page although the easiest option is to import the certificate.

This is needed because the code that validates the URL uses the context of the request to form the URL meaning that when you type

/rules/GetRulesRoot.do

Appian will internally review what URL your browser has, let's say myappian.appiancorp.com/.../designer and based on that will create the following URL for validation:

myappian.appiancorp.com/.../GetRulesRoot.do

and execute something like this:

urlHomePage = new URL(url);
urlHomePage.getContent();

when Appian calls the getContent() method to verify this is a valid URL it will go through HTTPS which in your case is at the Web Server layer, if the certificate your Web Server is using is not installed per the instructions above then it won't find a valid certificate. You don't see the error in the application server log because the exception is caught and the stack trace is not printed but basically that is what is happening.

Given the URL is formed using the page context you cannot control this through custom.properties, but if you can bypass the web server temporarily you will also be able to set this page although the easiest option is to import the certificate.