How to block highly restricted doc upload

How would Appian detect such a document?

I don't think that there exists a fixed format to check whether a document is highly restricted or not.
And on the other hand, the user should have knowledge about the highly restricted document present in his system before opening it.

You can restrict the user to upload any file on the basis of file extension, but just knowing that it is "highly restricted doc" will not help.

The only set of restrictions you can add is - extension-based, size-based, and count-based. Rest everything is out of Appian's control. 

To add to what others here have already suggested - the only thing slightly stronger than validating against a certain file extension is to enable File Extension Blocking in the Admin Console.  Again this only has to do with file extensions and that doesn't necessarily correlate to "highly restricted" per se.