migrating instances of specific application to new environment

Transporting an application to a higher version is no problem. But what do you mean with "instances"? The data and processes? The data from database can be migrated, but the process instances AFAIK can not without also migrating all other processes from other apps.

Can you please elaborate on your requirement ?

If I understand it correctly, the current state of the environment (the transaction data and logs) can be moved from 7.11 to higher version by following the Update guide available in forum.

It is always possible to move components/applications from lower environment to upper. They should work if they have all their dependencies are taken their to target environment.
It is not possible to move process instances from one environment to other. It is also not possible to run different applications on different versions of Appian in Same Environment . Hope this answers your questions.

If you follow the migration guide and ensure that the Environment is gracefully shutdown, A point to note is, that prior to doing the migration, its advisable to disable any automatically scheduled processes; reason being that when the migration is complete and the Environment "wakes back up" in the future, it will try to kick off all the timed processes missed..

The Upgrade Guide is available at:
forum.appian.com/.../Upgrade_Guide.html which applies to upgrading to Appian 7.8+ to 16.3.

As always, it is best practice to make sure you have a proper, full backup of the site in case of any issues with the migration. I also recommend doing a practice migration on a Dev or Test environment first, to confirm that there will be no unexpected surprises or your Applications break due to dependencies on deprecated features - forum.appian.com/.../Appian_Release_Notes.html