How do you clean the transaction log? does running the checkpoint script do it?

Checkpointing appends transactions to the database and effectively clears the transaction log.
This is an interesting question: can you provide additional background?
forum.appian.com/.../Configuring_Application_Checkpointing.html

We have to turn off task notifications. I put in the settings but it did not work.
forum.appian.com/.../Configuring_Notifications.html
One of the steps needed is to clear the transaction log.

Changes to notification settings require double engine start/stop. Unfortunately, this means an outage.

So just stop and start and stop and start? Because we stopped made the change and started again.

Right, it needs to be done twice.

Remember this means using stop-suite to shutdown the engines and start-suite to bring them up again, is not only a restart of the app server; this is an engine level setting.

yes I did that but its still not working

Where did you change those settings? runtime_ear\\suite.ear\\conf\\custom.properties or ear\\suite.ear\\conf\\custom.properties? If you're running the ACM are you making sure the setting is making it to the custom.properties file after running the build?

in addition to those questions how are you testing this is working or not? Are you using an account who has never modified these settings in his alerts settings?

we made the change user runtime. We have users that have never modified settings.