How to Backup and Synchronize Data after Rollback

Synchronize Data

If data is corrupted or lost in one or more of the application components, it is necessary to rollback these components to a previous state from a backup location. Because the components are from independent systems, with possibly different backup times, performing a rollback may require data synchronization between the components. This is a problem common to all integrated software platforms and Appian is no different in this regard.

The goals of data synchronization across systems after a rollback are:

• Ensure that all systems have a consistent and homogeneous view of the data.
• Restore a working state of the system where the status of the data in the database and other systems are consistent with the status of the Appian processes.

Once components are rolled back, you need to determine what the data discrepancies are and resolve them. This process is application design dependent and there is no automated way to make the determination. Since you may have multiple applications deployed within the platform, this needs to be addressed for all applications.

As part of the post rollback data synchronization you may need to:

• Restart processes that did not reach a key point. For example, if a “Request” process has not yet passed the “Approval” stage, the “Request” may be started anew.
• Restart previous step in the process to align with external DB. In this scenario, although the process may be ahead, a previous step needs to be re-executed to write data not yet in the database.
• Create reports to extract data/state from processes and update external databases accordingly in situations where processes are ahead of the database.

Options for Minimizing Impacts of Rollback

• Record centric design with short-lived processes
  • Since processes are short-lived, there are fewer items in progress, hence a lower chance of out-of-synch situation.
  • Since items in progress are smaller in scope they can be more easily restarted.
  • Since all data should be stored in RDBMS, there is a single source of truth, data is not duplicated and processes can be started based on the current state of data in the database.
• Synchronize backups of critical components
  • If critical components are backed up at the same time then data synchronization may not be necessary.
  • For example, Appian Cloud backs up all components at the same time (this excludes self-managed integration points accessed through VPN tunnels)
• Follow the recommended guidance and testing procedures for addressing a production application defect to prevent the introduction of data inconsistencies into the application.

Configure Backup and Recovery

There are primarily four data components to be backed up:

• Appian engines (process data)
• Appian application data (documents, archived processes, and other files)
• Web application collateral (including any custom pages/code)
• Data Stores and External data (such as external RDBMS data)

Each of these components is backed up separately, yet all backups must occur simultaneously to ensure that during recovery, the system's data snapshot is synchronized and consistent. Otherwise, you might encounter synchronization issues.

How To Backup

The tools used to backup don’t matter, so use ones that you already have available and are comfortable with. Example tools may include:

1. Snap Mirror
2. SRDF
3. Oracle Data Guard

A successful backup consists of several data components that were taken at the same time. Components backed up at different times cannot successfully be restored without synchronization issues.

1. Run the cleanup script at least 15 minutes before the scheduled backup time to reduce the risk of running out of disk space.
2. At the scheduled backup time, simultaneously:
   • (Only for non-HA sites) Run the checkpoint-suite script to prepare the Appian engines for backup. This will cause the current process state to be written to disk as a new KDB file.
   • Backup the application data and web collateral
   • Backup the primary Appian database and any business databases that will be restored in the event that a restoration of Appian is required. Additional considerations are required for external systems that are managed outside of Appian.
   • (Only for non-HA sites) Backup the KDB file and transaction logs at least 15 minutes after the checkpoint-suite script was started.
   • (Only for HA sites) Backup the KDB file and transaction logs.

Test the backup and restoration procedure as soon as possible after initial configuration. Do not consider either to be working until this is complete. Continue to test backup and restoration biannually thereafter.

See Configuring Backup and Restoration and Data Synchronization After Rollback for more details.

When To Backup

The frequency of the backup varies depending on the usage of the system and business requirements. Daily to weekly backup intervals are recommended. Production environments will generally have a lower tolerance for data loss than non-production environments. Determine your tolerance for data loss in each environment.

The more frequent a backup occurs, the more demand is put on the system resources. Backing up the Appian engines requires a checkpoint which locks activity in each engine for the duration of the checkpoint. This can take a few seconds to minutes depending on the amount of data and hardware performance.

Hardware solutions such as RAID and SAN replication can reduce the risk of data loss but are not a replacement for an actual backup.

Considerations For External Systems

External systems that Appian does not control the master data for may require additional considerations when restoring data. This is chiefly a consideration when integrating with systems (such as a database or web service) that will not have data restored in the event Appian is restored from a backup.

For example, take the following sequence of events:

1. A user starts a process that updates data in an external system
2. The external system is updated
3. A fault in server hardware hosting Appian requires a restoration of Appian. The only backup available is before step 1 occurred.
4. After restoration of Appian the process expects that it needs to update the external system again.

Depending on the application design, this may be as simple as cancelling the processes or may involve creating reports to identify processes that are out of sync and using in-flight modification to “fast-forward” the processes to the correct state.