Smoke Testing

Smoke testing is a non-exhaustive software testing covering the most crucial functionalities in the system so as to ascertain that they work as expected. Here focus is to cover these identified crucial functionalities using very few sample test cases without  bothering with finer details. Smoke testing is effectively applied to build testing where in code that is changed is validated before checking in that change into product’s configuration library. In other terms, here smoke testing is restricted only to test and validate against changed code but not for entire product that is built from configuration library that includes even changed code. Like this, smoke tests are designed to confirm that changes in the code are implemented and work as  expected and also, change does not destabilize an entire build.

A common practice at Microsoft and some other shrink-wrap software companies is the "daily build and smoke test" process. Every file is compiled, linked, and combined into an executable program every day, and the program is then put through a "smoke test," a relatively simple check to see whether the product "smokes" when it runs.

BENEFITS. This simple process produces several significant benefits.

It minimizes integration risk. One of the greatest risks that a team project faces is that, when the different team members combine or "integrate" the code they have been working on separately, the resulting composite code does not work well. Depending on how late in the project the incompatibility is discovered, debugging might take longer than it would have if integration had occurred earlier, program interfaces might have to be changed, or major parts of the system might have to be redesigned and reimplemented. In extreme cases, integration errors have caused projects to be cancelled. The daily build and smoke test process keeps integration errors small and manageable, and it prevents runaway integration problems.

It reduces the risk of low quality. Related to the risk of unsuccessful or problematic integration is the risk of low quality. By minimally smoke-testing all the code daily, quality problems are prevented from taking control of the project.

It supports easier defect diagnosis. When the product is built and tested every day, it's easy to pinpoint why the product is broken on any given day. If the product worked on Day 17 and is broken on Day 18, something that happened between the two builds broke the product.

It improves morale. Seeing a product work provides an incredible boost to morale. It almost doesn't matter what the product does. Developers can be excited just to see it display a rectangle! With daily builds, a bit more of the product works every day, and that keeps morale high.

Check for broken builds. For the daily-build process to work, the software that's built has to work. If the software isn't usable, the build is considered to be broken and fixing it becomes top priority.

Each project sets its own standard for what constitutes "breaking the build." The standard needs to set a quality level that's strict enough to keep showstopper defects out but lenient enough to dis-regard trivial defects, an undue attention to which could paralyze progress.

At a minimum, a "good" build should

·         compile all files, libraries, and other components successfully;

·         link all files, libraries, and other components successfully;

·         not contain any showstopper bugs that prevent the program from being launched or that make it hazardous to operate; and

·         pass the smoke test.

Smoke test daily. The smoke test should exercise the entire system from end to end. It does not have to be exhaustive, but it should be capable of exposing major problems. The smoke test should be thorough enough that if the build passes, you can assume that it is stable enough to be tested more thoroughly.

The smoke test must evolve as the system evolves. At first, the smoke test will probably test something simple, such as whether the system can say, "Hello, World." As the system develops, the smoke test will become more thorough. The first test might take a matter of seconds to run; as the system grows, the smoke test can grow to 30 minutes, an hour, or more.

Mutation Testing

In structured testing, effective testing means better test coverage and more number of reported errors. This demands more number of test suites. However it is difficult to measure accuracy of test suites. The method of Mutation Testing was introduced to measure the accuracy of test suites.

In mutation testing, we perform following activities:

Step 1:  We consider a perfect program and its corresponding perfect test suite. This means, we select test suite that covers all possible and a program that passes this test suite.

Step 2: We change the code of the program. This activity is called mutating. The resulting program with this change is referred to as mutant.

Step 3: Execute test suite on mutant.

Step 4: Observe the result. If results of the program are affected by the change and test suite detects the change, then the mutant is called a ‘killed mutant’. If the results of the program do not change and the test suite does not detect the mutation, then the mutant is called an ‘equivalent mutant’.

Step 5: Continue creating more mutants and run tests using test suites. 

Step 6: Count Total Number of Mutants; Total Number of Killed Mutants

Step 7: Calculate Ratio of Total Number of Killed Mutants to Total Number of Mutants. The value arrived at provides information about accuracy of the test suite

Adhoc Testing

This means impromptu testing using test cases that have been developed with no support of project related documents. These techniques require creative thinking and many a times use knowledge gleaned in the past on programmers or product. Ad hoc testing is less comprehensive, often inventive requiring lesser time, effort and documentation.