How to Undergo a Hardening Sprint

A hardening sprint is a period of a project where the team stops delivering new features and instead focuses on fixing bugs, preparing deployment activities, performing various types of testing, and cleaning up tech debt.

What is the Value of Hardening?

Hardening mitigates risk and improves confidence to your production release by focusing the team’s time on full end-to-end testing, catching and resolving bugs, testing in flight instances, verifying system translations, and tuning performance prior to a production go-live depending on your application. 

How is Hardening performed?

When: Hardening is typically a 2 week sprint at the end of a release, but depending on the length of the release, complexity of the features delivered, and the level of confidence of the team, teams may choose to hold hardening more frequently or for different lengths.

Who: The development team and testers should perform hardening activities. 

Where:  Hardening testing typically occurs in the UAT environment or another TEST-like environment. It should not occur in DEV. Hardening should be performed with the code that is going to PROD, and perhaps there is on-going development in DEV. Using the UAT environment poses the least amount of risk as to testing what is going to be released.

The DEV environment is used to address technical debt and resolve critical issues identified during testing. As there may be ongoing development occurring in the DEV environment, it is recommended that exploratory, in-flight, and regression testing occur in the UAT or another TEST-like environment.

Activities Before Hardening Begins

Before hardening starts, review your Go-Live Readiness Deployment Preparation and use this to help guide your activities in hardening. Ensuring you are ready for go-live is the top priority during go live.

Deployment preparation also should include:

• Cleaning app containers - If you create apps per story or per sprint for deployment purposes, make sure they are removed and that all objects are in the appropriate apps.
• Preparing and verify SQL statements
• Setting the appropriate Admin console settings

Activities During Hardening

1. Health Check Review

• During Hardening, the Health Check should be run and analyzed to identify areas for improvement in app design. However, teams should not wait until Hardening to do this. Rather, the Health Check should be analyzed regularly throughout development to make small improvements whenever problems arise.

2. Bug Resolution and Regression Testing

• This Hardening period is an allocated time for fixing bugs before go-live. With bug resolution comes regression testing to make sure none of the changes you made altered any other areas of the application.
• Expression Rule test cases should be re-run in Hardening to ensure that changes made with bug fixes didn’t cause test cases to fail. All Expression Rule test cases should be run consistently with each sprint deployment in addition to Hardening.
• Teams can perform regression testing with end-to-end Automated Testing tools like FitNesse for Appian. If the team has maintained Automated Testing scripts throughout the project, running those scripts during Hardening to regression test can save time.

3. Small UI changes based on end-user feedback (e.g. label changes)

• Any larger changes should be considered an enhancement and wait until after the initial go-live to implement. During Hardening, no new feature development should take place. Only small changes (low risk) should be implemented and regression tested during this time.

4. Tech Debt Resolution

• Many development teams keep a running list of Tech debt, containing ways to enhance the app or non-blocker bugs that they would fix during Hardening. The team must weigh risk, time to test tech debt, etc before committing to completing items on the list.

5. In-Flight Testing (If Applicable)

• When deploying process model changes to a higher environment, in-flight (active) process instances continue to use the pre-deployment process model version. Developers must take precautions to ensure that both existing and new processes will work after the deployment by staging some tests during Hardening. In-flight testing is not needed for first-time releases of an application because no process instances exist in the production environment prior to deployment.

6. Exploratory Testing

• Functionality:
  • Before hardening, divide portions of the application based on areas of risk. Typically, this corresponds to features being released.
  • Document risks and the time needed to explore them. Increase time needed based on the severity of the risk.
  • Assign out areas of risk to exploratory test and document bugs found.
  • Once the bug is resolved, re-test to ensure functionality works as expected.
• Security:
  • Try accessing data from user accounts which should not have permissions to do so. For example:
    • Test accessing the link to a record that a user cannot access from the front end to ensure they don’t see record data
    • Test accessing the link to a document that the user cannot access from the front end to ensure they can’t access documents they shouldn’t see

7. Estimate App Disk and RAM Usage

• The sizing process before an app is built is usually not clear and sometimes skipped. After the app is built, and you have tables and process sizing, you can do some real math to estimate your disk and RAM needs. Talk with an architect if you need assistance in determining these numbers.

8. Security Summary Review

• Use the Security Summary feature to quickly identify and fix security settings in application objects. The Security Summary should be reviewed each sprint, but should be double-checked and confirmed during hardening, especially if any smaller UI or tech-debt changes were made.

9. Data Load Testing 

• Work with the Product Owner and Subject Matter Experts to determine how many records will exist after 1 year of usage. It is typically recommended to populate 3-5 years worth of data, so the team should estimate the number of rows in each of the supplemental tables that need to be inserted to simulate PROD usage for that time frame.
• To allow testing with the appropriate amount of data, use the Database Load Generation app to quickly generate multiple rows for each of the application’s transactional tables.

10. Web API Load Testing

• If an application uses a high volume of web APIs calls, load testing the API is useful to see how the integration will behave under stress or when pulling back/receiving data. Call the integration with an anticipated amount of data (data that is passed in or returned based on your needs) to ensure there will not be unexpected performance issues. Also, call the integration at the expected frequency to ensure the system can handle the stress.

11. Automated Performance Testing

• Writing and maintaining Performance testing scripts should be a continual process throughout the release, done as development occurs and new performance testing areas are discovered. Performance tests are typically executed during Hardening, when the application is stable and the full set of functionality has been delivered.

12. Another Health Check Review