Exploratory testing: Extend your Quality Assurance

A testing approach in which testers dynamically design and execute tests based on their knowledge, exploration of the test object and the results of previous tests. (ISTQB Glossary, 2022) Exploratory testing is an efficient way to test software by integrating design, execution and analysis of tests during the testing session. (Nauman Ghazi, Petersen, Bjarnason, & Runeson, 2018)

Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continuously increase the value of his or her own work by viewing test-related learning, test design, test execution, and interpretation of the test as mutually supportive activities that run in parallel throughout the project. (Kaner, Falk, & Nguyen, 1999)

Exploratory testing experienced an upswing with the use of the agile manifesto in software development. Software teams today need to adapt quickly to customer requirements and software must be continuously developed, tested and released, which also led to an increased use of exploratory testing.(Atlassian, 2022)

Explorative testing brings with it, due to the freedom and the experience that the tester has here, that edge cases are more likely to be used and errors that might have gone undetected to the customer are discovered.

Instead of a test plan or test concept, exploratory testing creates a test charter to keep a rough overview of the goal and scope of testing. (IEEE Xplore, 2022) A widely used template for a test charter is as follows: (Hendrickson, 2014)

• Target: Where are you exploring? It could be a feature, a requirement, or a module.
• Resources: What resources will you bring with you? Resources can be anything: a tool, a data set, a technique, a configuration, or perhaps an interdependent feature.
• Information: What kind of information are you hoping to find? Are you characterizing the security, performance, reliability, capability, usability, or some other aspect of the system? Are you looking for consistency of design or violations of a standard? (Hendrickson, 2014)

A formulated example of a test charter looks like the following: (Hendrickson, 2014)

Be careful to not get too specific or too broad with your test charter.

Being too specific will defeat the purpose of exploratory testing because you are limiting the things you could expore with this type of experience based testing technique.

While being too broad will give you too much options to explore and you will loose track of whats actually to to test within this session.

Exploratory testing is rarely used exclusively to test software; it is usually performed as a supporting method in addition to scripted tests.

Advantages and disadvantages

A disadvantage of exploratory testing is that the tests invented and performed on the fly cannot be verified in advance and thus misunderstandings regarding the requirements or errors in test cases cannot be prevented. Furthermore, it can be difficult to show exactly which tests have been performed.

The traceability of the errors found may suffer as a result. It is unlikely that exploratory tests will be executed in exactly the same way when they are performed again. This can be an advantage if it is important to find new bugs, but also a disadvantage if it is more important to repeat certain details of the previous tests. This can be controlled with specific techniques (e.g. recording the Tests).

A case study at three companies found that the ability to provide quick feedback was seen as an advantage of exploratory test, while managing test coverage was rated as a shortcoming. (Wikipedia, 2022)

Goal

The aim of the agile working method is to increase the efficiency of teams and to reduce overhead in terms of planning, designing and documenting. Teams are more able to focus on their product or feature. Especially software test departments, have a lot of work to do in creating test plans, test reports, error reports, bug reports, manual and automatic tests and bug verification.

Agility means that more targeted testing, less documentation and faster delivery of results is provided. It makes sense to add exploratory testing to the testing method portfolio since it supports the agility benefits mentioned above.

Procedure

Explorative testing is implicitly used daily when performing manual tests, automatic tests, error
verification, test adaptations and test maintenance. During all of these processes, the test object is worked with and, based on the testers’ experience, various other scenarios are run through, even apart from the current scenario to be tested.

In the agile approach, changes are constantly being implemented by developers and tested by testers using exploratory testing. Here, the change itself is explicitly checked and additional actions are performed by the tester to ensure that also features or functions close to the changed one are not affected by the change.

In projects where e.g. new features are implemented, there is a requirements catalog, which contains various requirements. Here, usually both scripted and exploratory testing is executed. Exploratory testing covers requirements, whose full impact is not foreseeable and it is therefore appropriate to use the given freedom to run through various scenarios, but also for requirements that are of low priority and therefore do not require long-term documentation. The results of the exploratory tests are only briefly and concisely documented, discussed with developers or possibly recorded in a bug tracking system.

Even when explorative testing is performed implicitly it should also be considered to plan it properly, give this technique a specific time slot, create a test charter and then just sandbox around in the given time slot and try to execute some scenarios which usually are not tested with your scripted manual or automated test cases.

Conclusion

Exploratory testing is a still underestimated but indispensable method for testing software, especially in agile teams. It gives the testers the greatest possible freedom to find errors or unexpected behavior based on their experience. Documentation as well as planning overhead is reduced by creating a simple test charter, which roughly shows the way to go to avoid testing endlessly but leaves enough room to test multiple scenarios.

Making exploratory testing an essential part of the testing process in order to be able to deliver test results faster should be considered thoroughly. Another advantage of making exploratory testing an official part of the testing process is that developers can be involved in this process, because no extensive software test knowledge is necessary.

And in general it is just more fun to try out stuff based on your experience rather than following some fixed steps of test cases that you have executed already countless times.