Is there a way to get Appian to encrypt files (documents) before sending to an e

Not available out of the box as-is but you can build your own plug-in since Java allows for that.

See:

Java Cryptography Architecture (JCA) Tools forum.appian.com/.../summary

File Encryption and Decryption Simple Example www.codejava.net/.../file-encryption-and-decryption-simple-example

I am creating requirements and in our organization incoming/outgoing emails have some policies when email has an attachments. Can we encrypt an email ? Please let me know whether possible OOB or any other source of info.

Usually the encryption of emails is done in the email server side using specific third party solutions.