Your test plan is a description of the method you will use to ensure that your product meets specifications (from your product specification) and works correctly in all cases (no bugs or crashes). A good test plan anticipates all of the actions that a user might take when using your product and checks to make sure that the product behaves correctly. A test plan should also identify methods of making sure that internal software and hardware that is not exposed to the outside world works correctly. Tests are usually described as either black box or white box tests.

Black Box Tests

Tests in which the product is "sealed" and can only be examined by its external characteristics are known as Black Box tests. In essence, the product is considered a "black box" that cannot be opened. Most tests of functional specifications (functional testing) are black box tests. For instance, a test of a digital thermometer that checks to make sure that the displayed temperature is within 0.5 degrees of the actual temperature is a black box test. This is because no knowledge of how the digital thermometer checks and displays temperature is needed - only that it does, and it does so correctly. 

Black box tests can be used for both hardware and software systems, and often may require both to work together (as in the above example). Almost every functional specification should be tested using a black box test. 

Black box tests are convenient to administer because they use the complete finished product and do not require any knowledge of its construction. Black box tests are often administered by independent testing laboratories to insure impartiality.

White Box Tests

With black box tests, one must ideally test all inputs because the product might have a data-sensitive failure mode. Of course, testing is limited in that it is impossible to test all input combinations (especially if analog input is used). If the internal design of a system is exposed, it is easier to provide closer to full coverage of all possible inputs by testing individual portions of the product one at a time. Testing in which the internal architecture of the system is exposed is known as White Box testing.

Most white box tests are unit tests - tests of the operation of an individual subsystem. For instance, a white box test might confirm the accuracy and noise limits on an internal amplifier used in the front end of a digital thermometer. A black box tester would not even know of the existence of this amplifier. This white box test might concentrate at the known limits of the amplifier circuit in order to identify any likely failure modes. White box testing is also used for software subroutines and subsystems. For instance, a white box test for a data conversion routine might be used to confirm that the individual routine works correctly, even for extreme data that is unlikely to be encountered in normal operation.

One important aspect of white box testing is that it may require modification to the hardware and software. You may have to write special dummy front-end code in order to fully test a software system (as well as software that saves and confirms the results). You may have to isolate certain hardware systems in order to control the inputs and access the outputs of the system. For this reason, white box testing should be done sooner in the design cycle than black box testing.

Individual Test Plans and Results

Each different white box or black box test should be described in the form of an individual test plan. An individual test plan should include:

Test Plan Document

Your test plan document should be organized in the following manner:

Introduction and Overview of the Test Plan

Introduce the reader to this document. Give an overview of how it is organized and how a tester should follow the plans given in this document. 

Your test plan must include at least 25 individual tests. There are minimum constraints of the number of each type of tests in each section below (these sum to 21 tests - you must have at least four additional tests taken from any category). 

White Box Tests

White box tests should test individual units of your design, both hardware and software. 

Hardware Tests

Describe all needed hardware white box tests. Each should include a complete individual test plan as outlined above.

You must have enough white box hardware tests to adequately test your system. At a minimum, three white box hardware tests are required.

Software Tests

Describe all needed software white box tests. Each should include a complete individual test plan as outlined above.

You must have enough white box software tests to adequately test your system. At a minimum, five white box software tests are required.

Black Box Tests

Black box tests should test the overall functionality of the product. No black box tests should require opening of your product's case or knowledge of how the product works. The microcontroller's monitor program should not be used other than to start the main software for any black box tests.

System Tests

Most black box tests are system-level tests: Multiple systems must work together to complete the test. For instance, for a thermometer to correctly display temperature, the analog front end, A/D, conversion routine, display temperature software, LCD driver, and LCD (not to mention the power supply) must all be working. Describe all system-level black box tests using individual test plans as outlined above. Note that some tests are trivial and should be combined together.

You must have enough black box system tests to adequately test your system. At a minimum, ten black box system tests are required.

Individual Characteristic Tests

Some black box tests cover individual characteristics (i.e. battery life or water resistance). Describe all individual characteristic black box tests using individual test plans as outlined above.

You must have enough black box individual characteristic tests to adequately test your system. At a minimum, three black box individual characteristic tests are required.

Coverage of all Specifications

One of the major purposes of the test plan is to make sure that your product meets all  specifications. List all specifications from your functional spec and identify which tests listed above test this aspect of your system. (Two good ways to present this are: 1. A table with specs in one column and tests in another and 2. Individual subsections for each spec.) All specifications should be tested by at least one test, and most specifications should be tested in multiple ways.

Combining Trivial Tests

Some of your specifications may result in trivial tests. For example, you may have specified that your product weigh less than 1 pound - a black box test that says to simply weigh the product is a trivial test.

In general, a test is trivial if it can be done by a simple inspection or by a simple measurement. If it takes less than five minutes to do a test, it is probably trivial. Here are some examples of specifications that result in trivial tests:

You must combine all of your "trivial tests" into one black box system test (even if some are white box tests).  If you are unsure of the whether a test is trivial or not, ask the instructor.

Kevin Bolding September 22, 2009