Performance Testing Methodology

Methodology Overview

This methodology is intended to provide an entry point for projects of all sizes and types, while identifying opportunities to extend and customize the performance testing effort based on specific project goals and constraints.

Note: some activities are based on Performance Testing Guidance for Web Applications.

In order to simplify project estimation and scheduling Appian has developed a performance testing task plan. You can use this plan as a starting point and extend it to address specific performance objectives.

Validate your performance test plan with the Performace Testing Initiation Best Practices Checklist.

Activity Participants External Dependencies Artifacts/Templates Guidance
Identify stakeholders Project Manager, Stakeholders None Project Charter Include in existing kickoff activities.
Educate stakeholders and obtain commitment Project Manager, Stakeholders None Project Charter Performance Test Planning Presentation Include in existing kickoff activities.
Identify the test environment IT Infrastructure and Tool SMEs, Test Architect Physical and logical architecture diagrams Performance Testing Questionnaire Base assumptions on questionnaire responses. Don't try to address environment limitations. Document limitations and their impact in acceptance criteria for acknowledgement and sign-off.
Identify acceptance criteria Project Manager, Sponsor and/or Key Stakeholders, Test Architect None Baseline Performance Acceptance Criteria Stick to criteria defined in the Baseline template. Project delivery is not dependent on meeting this criteria.
Plan and design tests Test Architect UAT scenarios Performance Test Plan Worksheet Base performance scenarios directly on UAT scenarios. Limit implementation scope to major use cases representing most important and most common user activities. Plan for one 1 hour test run at peak user load.
Configure the test environment IT Administrators, Test Architect Environment availability None Assign responsibility to IT admins. Use Appian performance logs to capture monitoring data.
Implement the test design Test Engineer Test tool availability, Stable application deployment Using JMeter with Appian Work only with stable application features. Use JMeter unless an alternative tool is already used. Minimize the use of variable data.
Execute the tests Test Architect Stable application deployment Using JMeter with Appian Allocate time to run multiple tests to demonstrate consistency and stability, as well as support test and system tuning between iterations.
Analyze results, report, and retest Project Manager, Test Architect, IT Administrators None

JMeter Analysis Worksheet

Appian Log Analysis Worksheet

Use templates to report against performance criteria. Root cause analysis and recommendations will be time constrained. Rework is included in existing UAT schedule. Significant issues will cause schedule slip.

The Role of Stakeholders

There are multiple stakeholders responsible for performance testing. The ownership of the respective responsibilities may differ based on the composition of your team. What’s important is that all areas are owned by someone on the team. 

We’ve provided a sample presentation to leverage as a Sprint 0 artifact once you've identified the stakeholder responsibilities and the non-functional requirements. 

Business Stakeholder

  • Collaborates with the Performance Test Architect to determine the application response time goals
  • Provides the Performance Architect with the expected number of transactions for each workflow (per year, peak per day, etc.) and the expected number of users in the system
  • Sets aside a set amount of development time for performance improvements.
  • Balances the investment in more hardware versus the investment in more development time to improve system performance

Appian Performance Test Architect

  • Architects high level performance test design 
  • Models realistic usage scenarios with input from the Business Stakeholders
  • Defines metrics and collections procedures

IT Team

  • Ensures that a production like clone is available for performance testing
  • Ensures that the proper monitoring occurs during the performance test
  • Adjusts the amount of hardware available for the test as necessary

Performance Test Engineer

  • Has expertise in creating test scripts in at least one performance testing tool (Locust, JMeter, LoadRunner, LoadUI, etc.). Appian is agnostic to your choice of performance testing tool. Appian does provide Locust and JMeter tutorials for those without performance testing experience 
  • Creates test scripts based on the design from the Performance Test Architect. 
  • Executes the tests, monitors the results and provides the results to the Appian Tech Lead for interpretation
  • Introduces variability into the performance tests to simulate actual user behavior
  • Generates realistic test data based on the expected number of transactions in the system
  • Reruns the performance tests as the development team makes changes to the application

Appian Tech Lead

  • Analyzes Performance Test Results
  • Recommends application changes to improve the performance of the application
  • Coordinates with the Business Stakeholder to prioritize performance improvements over new functionality when necessary. 
  • Coordinates with the Performance Test Engineer to rerun the performance tests as recommended design changes are completed. 

Plan and Design Performance Tests

Planning and designing performance tests requires identifying and modeling usage scenarios, providing test data for those scenarios, and specifying the metrics to be collected during the test execution. This process can be extremely complex and time consuming. The following guidance provides one simple approach that can be extended based on project goals and constraints. Teams with extensive experience in performance test design may chose to develop their own approach.

User Roles

Identify unique application user roles and estimate the number of concurrent users accessing the application on average and during the hour of peak usage. For example, throughout the day there will be about 20 Sales Reps accessing the system at any given time, but between 10am-11am there will be 100. Identify how user load will grow over time in response to business changes.

Usage Scenarios

Identify potential usage scenarios and the associated user role using the acceptance criteria, existing production usage, business use case documentation, functional test designs, and ad-hoc Appian features (eg: post a message, search News). Consider scenarios that are initiated by users and scenarios that are initiated by the system (eg: scheduled processes).

Estimate how often each scenario will be executed in production:

  • User-initiated: Average instances per hour
  • System-initiated: Average instances per hour

Prioritize the scenarios using the following guidelines:

  1. Select all scenarios referenced by the acceptance criteria. Revise or eliminate acceptance criteria if this cannot be achieved.
  2. Select scenarios based on their contribution to the total application workload. An activity that accounts for 25% of all load is more important than one that accounts for 2.5%.
  3. Select scenarios that present known technical risks.

Performance tests simulate production workloads, and more accurate simulations produce more accurate results. Modeling an insufficient portion (<80%) of the total workload will significantly reduce the predictive value of performance testing. You should test with at least a few use cases that add to more than 80% of total user activity.

Document the sequence of actions involved in each scenario or reuse existing test scenario documentation (eg: UAT scripts). For scenarios with multiple paths either select the most likely or most important path or create separate scenarios for each path. Between each user-initiated action, include a delay to simulate “think time” spent reading or entering data. Incorporate a random element by specifying the average delay and a standard deviation.

Test Design

Design a set of performance tests at different loads. For example:

  • Test normal conditions using the peak and average user estimates over a 1 hour and 8 hour period
  • Test stress conditions over the same time periods using 4 times as many users
  Normal Conditions Stress Conditions (4x user load)
Peak Test 100 users for 1 hour 400 users for 1 hour
Endurance Test 20 users for 8 hours 80 users for 8 hours

Validate the test design by comparing the number of activities to be performed in the test against business expectations. Revise the scenario and user load estimates until the test design closely approximates the expected production load.

Variable Test Data

Nearly all test scenarios require user input (eg: login credentials, task form values, message text). Using the same inputs for every instance of a test scenario can cause unexpected performance problems (false positives) or cause other performance problems to go unnoticed (false negatives). Add variability to a scenario using the following procedure:

  1. Identify the inputs expected by each action in the scenario
  2. Record a sample set of valid values for each input

    1. Inputs used to control logical flow (eg: PVs used in gateways, username associated with a specific task) may have only one valid value for a given scenario
    2. Include a range of small and large values where appropriate
    3. Check the features of the available tools to help determine the best format for variable input data
  3. Related inputs (eg: username and password) should be recorded together

Example test data for creating a Case and collecting three data points, title, description and priority (notice the various edge cases and different users executing the action):

Dataset Name Username Password Case Title Case Description Case Priority
A basic.user1 tR4YasW4 Long title with extra description Short description 1
B basic.user2 jEhe9EtR Short title Longer description with additional text 3
C manager.user1 ga4aVuxU Short title <empty> 2

Prepopulated Test Data

Application performance often depends on how much data is already in the application. Unless the test scenarios are designed to add data during the test, prepopulated data will need to be loaded before each test run. Consider the following types of data that may impact the performance acceptance criteria:

  • Appian processes
  • Database records
  • Documents
  • Other files (logs, archived processes, KDB backups)

Determine how prepopulated data will be generated or loaded. For example:

  • Create database records using a SQL script
  • Create processes using a test script or plugin
  • Migrate actual production data

If possible, take a snapshot of the environment after test data has been pre-populated and use that to restore a clean state before future test runs.

Metric Collection

Metric data can come from a variety of sources (eg: client testing tools, server monitoring tools, Appian logs) and each source can often be measured and recorded in several ways. To ensure consistency across all tests it is important to document the specific definition and collection procedure for each metric. Metric documentation can include the following information:

  • Unit of measurement (eg: utilization %, milliseconds, # per hour)
  • Collection interval (eg: every second, every hour, once per test)
  • Aggregation method (eg: instantaneous, sum, average (mean), 95% confidence interval)
  • Source system (eg: Appian application server, client test workstation)
  • Collection procedure (eg: manipulate log data in Excel, automated tool output)

Verify that the documented metrics will be able to determine whether the performance acceptance criteria are being met. Additional metrics for individual system components (eg: web service, query rule) can be a useful metric when diagnosing performance problems.

Configure the Performance Test Environment

The test environment should be setup as part of the overall configuration management plan, typically performed by IT (or the Appian Cloud team). Special consideration should be given to any monitoring tools that need to be installed or configured. Make sure that the system time is correct on all servers and client machines. If Appian logs are used as a source of metric collection then the appropriate changes should be made to the log4j settings.

The application to be tested should be deployed to the environment using the same procedure to be used in production. Once the application is deployed, run several sample tests (smoke test) to confirm that the environment and application are configured properly and that the expected metric values are being collected.

Implement the Performance Test Design

Implementation of the test design is closely tied to the available tools (eg: JMeter). It typically involves recording test scripts based on actual user interaction and then modifying those scripts to add variable inputs and validation checks. Tool-specific techniques are not covered here.

Test development is an iterative build-verify cycle. Tests should be run frequently using a single user to confirm that they interact correctly with the application and that the expected metric values are being generated.

Recorded test scripts contain numerous references to system identifiers that cannot be reused (eg: task ids). Locate and replace system identifiers with variables. Extract the runtime values from earlier server responses (typically using regular expressions) and update the variables.

If the test design calls for variable data, locate the static references in the recorded requests (eg: form inputs) and replace them with variables. Populate the variables with dynamic values from a lookup table, random generator, or other data source.

Build in assertions to verify that the expected response is returned. For example, verify the title of the next task in an activity chain. If it doesn't match then the previous submission may have failed validation or followed an unexpected process path. Additional examples include:

  • Verify that a report did not return a timeout message
  • Verify that the expected task is in the task list

Additional considerations when implementing tests include:

  • “Think time” delays should not be included in response time metrics

Execute Performance Tests

Verify that the test environment, application, and monitoring tools are functioning properly by running a small set of tests (smoke test). Remove existing data (eg: processes, database records, documents, log files) to restore the environment to a clean base state. If the test design calls for prepopulated data, add it after old data has been removed.

Run the tests and monitor their execution. Watch for errors that may indicate a problem with the tests. For example:

  • Metric values not being captured as expected
  • Unexpected HTTP response codes (eg: 500 Server Error, 403 Forbidden, etc)

If the tests are not going to provide valid results it is better to halt them early.

Once halted, it is recommended to first verify the test scenario and infrastructure stability before performing application-side troubleshooting. If the test results are invalid because of a systemic problem there may be little value in further analysis.

Record and store the raw test results and observations for later reference. Make sure to associate the results with a specific version of the test implementation.

Troubleshooting Common Issues

High client error rate

  • Description:
    • Performance test client returns a high percentage of HTTP error codes (40x, 50x)
  • Potential Root Cause(s):
    • May be a sign that the test implementation/parameters are incorrect and/or test script does not align with the current application state
  • Next Steps:
    • Verify if recent application changes have been deployed that conflict with expected text scenario
      • Adjust scripts or revert application changes as needed
    • Execute smoke test with one virtual user
      • Consider running in GUI mode with enhanced logging to assist with analysis
      • If error rate continues, it is likely the script needs to be adjusted and re-validated

High client response times and low server utilization metrics

  • Description:
    • Test client shows high response times, but server resources are at low-to-normal operating levels
  • Potential Root Cause(s):
    • May be a sign that the test client cannot handle the request volume (i.e. resource saturation)
  • Next steps:
    • Distribute clients or utilize Cloud-based testing services
    • Disable client GUI mode to remove overhead
    • Disable any unnecessary logging
    • Verify any external dependencies, such as web service integrations or network latency, are not long-running

High client response times and high server utilization metrics

  • Description:
    • Server-side resources are at high-to-critical operating levels
  • Potential Root Cause(s):
    • May be a sign that the test implementation is creating more requests than expected for the given load or test is not properly isolated
    • May be a sign that the underlying application and/or architecture is not designed to scale to required load
  • Next steps:
    • Validate test scenario
      • Evaluate test script and test parameters for valid wait times
      • Validate test script behavior matches performance acceptance criteria (i.e. throughput and requests per sec are valid)
      • Verify distribution of requests is realistically distributed (i.e. not excessively uneven or front-loaded)
      • Ensure adequate ramp-up/down time is implemented
    • Validate non-test load are not occurring on the environment during the test
      • Look for other users logged in, active processes not run by test etc.
      • Look for uneven request distribution over time
    • Investigate if system has cascading errors or unexpected process flows
      • Verify randomized elements, such as random interval timers, do not have unrealistically high variance
    • Analyze performance of application
    • Analyze system resources

Low client response times and low server utilization metrics

  • Description:
    • Response times are low or otherwise meet SLAs
    • Server resource metrics are underutilized
  • Potential Root Cause(s):
    • The load created by the test scenario did not significantly impact the test system
  • Next steps:
    • Verify test results as noted in the following section, ensuring that the load from the test was what was expected (i.e. not a false positive)

Analyze Results

Compare the performance test results against the performance acceptance criteria. If there are any failures, analyze the underlying metrics to determine the root cause. For example:

  • Show Case Summary task violates the response time criteria because it calls 10 query rules
  • Update Vendor Details task violates the response time criteria because an external web service takes 15 seconds to respond
  • Appian engine server violates the memory utilization threshold criteria because the test environment does not have as much memory as the production environment

Validate that the tangible outcomes of a test align with expected values by analyzing the number of processes started or completed, service calls executed, records created, etc. For example:

  • A 100 concurrent user test is meant to initiate 100 cases within a test interval, but creates a number much lower or higher during the sampling interval.
  • A test scenario is expected to take 20 minutes to complete, but the process report shows the activity in 5 minutes.

Develop recommendations to resolve performance concerns based on this investigation. For example:

  • Redesign the form to call fewer query rules, possibly using a view to create a consolidated CDT
  • Call the external web service asynchronously rather than as part of an activity chained flow
  • Revise the acceptance criteria to acknowledge the memory constraint in the test environment

Generate a stakeholder report that includes a summary of acceptance criteria results and recommendations for addressing any failures. Represent results visually and highlight targets or thresholds. Have the details of any root cause analysis available to support your recommendations. Address any impacts on the testing process or performance results due to limitations in the test environment, tools, or testing schedule.

Revise the test design to incorporate any knowledge gained from test execution and analysis. For example, if the stress conditions did not cause the application to reach capacity try increasing the user load or add additional pre-populated data.

Retest the application after the recommendations have been implemented to confirm performance improvements. Include trend analysis in subsequent stakeholder reports.