How to create Pallete category and Pallete name in Appian for separating custom

You can modify palette-config.xml under <APPIAN_HOME>/ear/suite.ear/web.war/WEB-INF/conf/process
This only applies to the OOTB smart services however

Thanks Aleksi. I Chaged the Palette category in the eclipse wizard while creating the New Smart Service and Appian picked the categoryfrom the deployed plugin .

The best practice is to annotate your smart services. If the paletteCategory and/or palette doesn't exist, Appian will create it.

This is an example:

@PaletteInfo(paletteCategory = "Appian Smart Services", palette = "Appian Support Tools")
public class AppianSupportSmartService extends AppianSmartService {
....
}