Exploratory Testing

Exploratory Testing Overview

Exploratory Testing (ET) is a risk-focused testing method in which testers explore systems to identify areas of risk, in addition to checking expected functionality. As testing reveals risks, the tester uses what they have learned to inform the next tests that they should perform. This continues until all areas of risk in the application are explored and addressed. Exploratory Testing works well with agile projects because it improves testers' ability to interact with the system and allows them to create effective tests quickly. It also encourages testers to be creative and discover bugs that other testing methods, like using defined test scripts, cannot.

Exploratory Testing is most useful when it complements tests for expected behavior, but it should make up the majority of the manual testing on any project. If possible, it is also recommended to use ET in conjunction with other testing methods, like Automated Testing and Performance Testing, to ensure full code coverage.

Why Use Exploratory Testing

Exploratory Testing can:

• Identify bugs in portions of software missed by other forms of testing
• Increase overall confidence in the application
• Expose the need for scripted tests in specific areas based on risks that are found
• Reveal necessary changes in design by identifying limitations and unexpected weaknesses in the application
• Reduce the need for formal testing documentation and allow testers to free-form test based on experience and intuition
• Find edge cases and requirements missed during backlog grooming
• Check the work of another tester by doing a brief independent investigation

When Should You Exploratory Test

Exploratory Testing should be performed as part of agile sprints to ensure that each sprint results in fully tested software. While Exploratory Testing typically occurs after functional testing, it can occur at any point in the sprint (e.g. during development, peer review, or regression testing).

When and how to test should be decided early in the project and should be included in the team’s Definition-of-Done  documentation. All user stories should be Exploratory Tested, especially  those flagged as high-risk during sprint planning.

Examples of When to  Exploratory Test

One way to use ET is to hold a Bug Bash where the team dedicates time to Exploratory Testing. This can focus on testing an entire application or specific pieces of functionality and can be done at any point during development or as a way to prepare for Go-Live. Doing so will catch bugs in the application which may have been missed during development.

ET can also be conducted after Sprint Demonstrations  so stakeholders can also participate. The benefit from this practice is that stakeholders can simulate their real-world roles on all available devices (PCs, tablets, cell phones, laptops). Results from these sessions provide quick feedback and build trust in the system.

Who Should Test

Any available project resource can perform Exploratory Testing . Often the development team is used to perform ET, since they have intimate knowledge of the functionality and code being tested. If possible, use all available resources including stakeholders (in UAT, for example) to help conduct ET.

How to Exploratory Test with Appian

Regardless of when a team decides to Exploratory Test, following the guidelines outlined in this section will ensure that the tests are specific and well-documented.

Design Test Charters

In order to make tests specific, testers should design test charters to guide their tests. Test charters should be broad enough to allow for exploration, but specific enough to ensure that they can be tested deeply and fully. Each test charter should include three features:

1. What is going to be explored
2. How that feature is going to be explored (tools and techniques that will be used)
3. The information the tester is trying to learn

Example: Explore paging grid functionality with various paging/sorting/filtering combinations to discover problems with grid saves and data refreshes.

When testing charters, if an area of risk is found when testing one charter, create another charter to discover bugs within that area of risk. This should continue until all areas of risk have been tested and the tester is comfortable with the results.

If a team develops with design documents, test charters can be included along with the design and reviewed with the team before development begins to ensure alignment and fuller test coverage.

Timebox Testing

It is helpful to timebox your testing to reduce redundancy and maximize development time. The allotted time should be based on testing complexity and can be an estimate. Generally, testing should be finished when the tester is confident that they have tested every risk that has been identified. If the tester has not used up the time allotted for testing but feels that new tests would be redundant, they should stop. Likewise, if the tester has used up the time they originally allotted but still feels that there are areas of uncertainty in the functionality, they should continue until they feel comfortable that they have tested thoroughly.

If timeboxing testing is new to a team, it may be useful to track the estimated time required against the actual time taken to test to determine whether estimates are good and adjust as necessary.

Useful Heuristics for Exploratory Testing

The following heuristics are named and outlined by Elisabeth Hendrickson in Explore It! Reduce Risk and Increase Confidence with Exploratory Testing¹. These are techniques for testing an application in an exploratory manner. Each testing heuristic includes one or more examples illustrating how it can be used in an Appian context. While testing, think of other ideas or patterns you can try to disrupt, using these heuristics as a baseline.

• Never and Always: Determine what your system should never do and always do, and see if these can be violated in any way by the functionality you are testing.
  • Examples: 
    • If there is a component or section that should always be hidden after an action is taken (a button is clicked, etc.), test it with different contexts or user groups to verify that it will always stay hidden.
    • If you are using record-level security, test different combinations of record  data and user groups/roles to ensure the security is set correctly.
• Beginning, Middle, End: In situations where the order of elements matters, try switching the order of certain elements or performing actions on an element in a specific position.  
  • Example:
    • If you are using a paging grid and you delete the last item in the grid,  did you reset paging to avoid a SAIL error?
• CRUD (Create, Read, Update, Delete): Using the other heuristics in this list, such as “Beginning, Middle, and End” or “Zero, One, Many”, change data elements by creating, reading, updating, and deleting them to ensure that the behavior is expected.
  • Example:
    • Use CRUD with Zero, One, Many in a scenario where the user can upload and delete documents by checking data elements in the database and in the user interface after uploading a document, deleting a document so that there are none uploaded, adding several documents, then deleting several documents, etc. 
• Follow the Data: In all actions which alter data, follow the changes in the data to find unexpected results.
  • Examples:
    • If clicking a “Submit” button on a form in your application makes saves to the database, check the database before that button is clicked and again after, checking every field that could have been impacted.
    • If a process variable is passed throughout a process model, monitor the process and check the process variable each time it is modified by the user or the system
• Goldilocks: Use inputs that are too big, too small, and just right. This may lead to errors or data truncation (see also Too Few and Too Many).
  • Examples:
    • If there is a date field on a form for the user to input dates, try choosing a date 15 years in the future or in the past.
    • If the user has to enter their age, try testing with negative values.
• Reverse: When actions occur sequentially, try to perform them out of the expected order.
  • Example:
    • If you are using a breadcrumb-style approach to wizard navigation, try going backward through your breadcrumbs, or taking a path that users shouldn’t normally take.
• Too Few/Too Many: When a certain number of inputs or outputs is expected, test with fewer (or more) things than the software expects.
  • Example:
    • If the user interface contains an editable grid, double or triple the number of line items and ensure the grid functions as expected.
    • If the user can delete items from a grid, delete the last line item.
• Violate Data Format Rules: If the application expects values of a certain format, try violating those expectations to see how the system responds.
  • Examples:
    • If there is a field for a user to type in their name, see what happens when they type in an integer or special characters instead.
    • If there is an email field, try entering an invalid email address
• Zero, One, Many: When appropriate, test with inputs of zero, one, or many to find:
  • Incorrect plurals (i.e. “0 item available” or “1 items available”)
  • Incorrect counts or calculations, including divide-by-zero and off-by-one errors
  • Examples:
    • If you are testing a grid, test with no records to show, one record, and a number of records much larger than the batch size.
    • If you are adding a certain number of days to a date, try adding zero days, one day, and 5000 days and verify the results.

1. Hendrickson, Elisabeth. Explore It! Reduce Risk and Increase Confidence with Exploratory Testing.Dallas: The Programmatic Programmers, 2013.