To identify potential bugs or defects, software testing professionals compare the software's performance against standard principles by which a problem can be recognized:
While these principles are widely accepted and key components of software testing training, there are also a number of controversies regarding the best practices of software testing. Some professionals believe that responsible software testing includes a changing set of practices that testers can choose from for each testing situation, while others purport that testers should adhere to documented standards in all instances. Following is an overview of the some of the main software testing controversies being debated today.
Agile vs. Traditional – Agile testing emphasizes testing from the user or consumer's viewpoint, and asserts that testers should be prepared to work under continuously changing conditions. Rather than using rigidly defined testing procedures, testers compare performance against newly developed code, and work with developers in a more collaborative, test-driven development model. The agile testing movement is popular in commercial circles.
On the other hand, the Capability Maturity Model (CMM) is a development model elicited from actual data. Originally funded by the U.S. Department of Defense, it allows for an effective method of resolving software development issues, and was embraced by military and government software providers.
These two software testing movements are actually different approaches to the same problem: agile testing is a way of working, while CMM is a process improvement idea. The fact remains that software testers, even when using flexible methods, must have direction or their efforts will (be) unsuccessful.
Exploratory vs. Scripted – Exploratory testing emphasizes learning, with simultaneous testing of design and execution. While commonly used, some consider it a dangerous practice. Others believe it is essential to proper testing. Scripted testing refers to learning and test design that occur prior to - and possibly again during - test execution.
Exploratory testing has two known disadvantages: a lack of opportunity to prevent defects, which are often discovered as a result of designing tests in advance, and difficulty in achieving repeatability of tests.
When testers are familiar with the software, a compromise can be introduced, in the form of structured exploratory testing. In this software testing approach, a plan is developed describing what functions need to be tested, but not how the testing is to be done.
Manual vs. Automated – Some experts believe that test automation is too expensive relative to its value - so it should be used cautiously. Others claim automation should be used in all software testing, believing that it can be successful if test planning is complete and comprehensive. Many of today's top software companies use automation for their internal software testing. While automated testing is becoming more widespread, many maintain that certain aspects of software testing should continue to be conducted manually.
Software Design vs. Software Implementation – This controversy refers to the question of whether software testing should occur throughout the design process or only at the end, prior to release. Some experts assert that software testers must test design as well as implementation. This practice leads to changes in the test cycle as well as the interaction between software testers and developers; however, it offers an important advantage in that testers are able to identify and remove bugs earlier in software development.
Who Watches the Watchmen? This classic question, first posed by the ancient Roman poet Juvenal, refers to the idea that any form of observation will affect that which is being observed. In software testing, it means that the testing process can fail not because of defects in the software, but in the testing method. In one controversial practice, developers intentionally hide bugs in code to gauge the percentage of them that are caught during software testing. However, this assumes that intentional and unintentional bugs are the same, whereas different tools and methods may be necessary to catch both varieties of bugs.