I am trying to configure the mail server on a 7.2 server. I have done the c

Did you try these settings in custom.properties:

conf.mailhandler.mail.transport.protocol=smtps
conf.mailhandler.mail.smtp.starttls.enable=true
conf.mailhandler.mail.smtp.starttls.required=true

Also, check if your mail server needs authentication to connect, then you also have to enable:
conf.mailhandler.mail.smtp.auth=true
And set username:
conf.mailhandler.mail.user=<MAIL_SERVER_AUTH_USERNAME>

Set password in password.properties, create the file if it's not already there in <APPIAN_HOME>/ear/suite.ear/conf/passwords.properties with the below entry:
conf.password.SMTP=<MAIL_SERVER_AUTH_PASSWORD>

Not many enterprises use secured mail server, in anycase make sure you have the right port to the mail server (Default is 25). IP is preferred than the host name, if the corporate DNS is set the right way you shouldn't worry much but IP rules out issues related to DNS:
conf.mailhandler.mail.smtp.host=<MAIL_SERVER_IP>:<MAIL_SERVER_PORT>

Hope this helps.

Hi

Yeah, mail server does need auth which I have configured & the password too.
I checked the documentation of 7.2 but the above mentioned configuration are not there. (although, they are present in 7.10).
smtp host is configured & I will again cross verify the port (whether I am using the right one or not).

Will give you update in a while.

I checked the port, and it was correct and it is still saying this below error.

12:36:45,313 INFO [stdout] (default-short-running-threads-threads - 20) Caused by: java.net.ConnectException: Connection refused
12:36:45,313 INFO [stdout] (default-short-running-threads-threads - 20)           at java.net.PlainSocketImpl.socketConnect(Native Method)
12:36:45,313 INFO [stdout] (default-short-running-threads-threads - 20)           at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
12:36:45,313 INFO [stdout] (default-short-running-threads-threads - 20)           at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
12:36:45,313 INFO [stdout] (default-short-running-threads-threads - 20)           at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
12:36:45,313 INFO [stdout] (default-short-running-threads-threads - 20)           at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
12:36:45,314 INFO [stdout] (default-short-running-threads-threads - 20)           at java.net.Socket.connect(Socket.java:579)
12:36:45,314 INFO


Any idea..?

Could be firewall issue.
Check you can telnet to the mail server and port from the Appian server machine to rule out firewall related issues.

Note that SMTP uses port 25 by default, but SSL/TLS encrypted SMTP uses port 465