Effective Engineering e-Newsletter – 6/10/2004
This is your bi-weekly e-Newsletter from Effective Engineering Consulting
If you would like to receive Effective Engineering e-newsletters as
they are published, please send an email to firstname.lastname@example.org,
and we will add you to our distribution list.
Comments and suggestions are welcome and encouraged!
Development Methodology: Test –- to Verify
Tom Dennis – President, Effective Engineering [email@example.com]
previous e-Newsletters, I described the basics of development methodologies
(see eN-030828 –
Development Methodology Basics: Stages of Development, and eN-030911
– Development Methodology Basics: Management of Development).
Those e-Newsletters were able to cover only very broadly the basic
elements of development methodologies. The
Development e-Newsletter described Requirements, Architecture,
& Planning (see also eN-030925
– Development Methodology: Requirements, eN-031009
– Development Methodology: Architecture, and eN-031023
– Development Methodology: Failing to Plan Means You Are Planning to Fail!),
and briefly described Build & Test (see also eN-040527
– Development Methodology: Build – to Last), and Release
& Support as two other stages of development.
This and some subsequent e-Newsletters will cover in somewhat more
depth Test, Release, and Support.
The purpose of Test is to verify that products conform to their Requirements
and Architecture, and that the Build process has been
implemented with quality and care; it is not to attempt to Test
or Inspect quality into products.
Quality must be ensured by design, and the characteristics of a Quality
by Design methodology will be the topic of a subsequent e-Newsletter.
Key people involved in Test should be actively involved in discussions
during the Requirements, Architecture, Planning, and Build
stages, so that design elements that may impact or be impacted by Test
can be properly accounted for in advance.
There are many aspects and functions of test, some of which are carried out by
the developers during the Build stage, and some of which are carried by
the testers during the Test stage.
We will briefly touch on these in this discussion.
During the Build stage, it is the developers’ responsibility to Unit
tests their designs. That is, they must verify that their designs operate properly
before handing them off to others. They
must verify that their design functions as specified and that all areas of the
design have been tested (all possible logic combinations in a digital design;
all code paths, error paths, memory leaks, exit paths in a software design;
all strength, stress, heat flow aspects in a mechanical design).
Development test tools should be used extensively to help ensure all
aspects are covered, and adherence to the discipline of carrying all of this
out should be enforced. Design
reviews are also required to ensure that someone other than the designer
concurs with the design. This
helps to avoid “forest and trees” types of problems.
Only after all of this has been done can developers’ designs be “checked
in” as “ready” to be handed off to Test.
In cases where individual designs do not stand alone, but only operate
in conjunction with other designs as part of “modules”, then all of
the above should apply at the “module” level as well.
Once a product or elements of a product are released to Test, many
methods are applied to verify the integrity of the design.
We will go into some of these in more depth in later e-Newsletters, but
will briefly describe some of them here:
► Integration Testing: This is the testing of
combinations of design elements as they are turned over to Test.
More and more pieces of the design get combined to begin testing the
design more completely. Such incremental testing continues up to and including the
initial testing of a full product release, and beyond.
Integration testing is done in close cooperation between test and
development. Bugs are reported
using the formal bug tracking system, but developers work closely in the loop
with test to see and understand bugs as they are found.
► Functional Testing: This is testing of the
functionality of the design of modules up to and including the full product
release. It concentrates on
verifying that the specific functionality and features are operating per the
► System-Level Testing: This is testing as a
customer sees the product, verifying that the product works as a complete
product, with all interactions and system level functionality fully
operational. Unlike functional
testing, which may concentrate on the specifics of one functional area of the
product, system-level test looks at the overall operation of the product in
its actual intended environment.
► Negative Testing: This is testing that
deliberately introduces errors to verify that the product operates properly in
the presence of such errors.
► Outside-the-Box Testing: This is performing
testing that “colors outside the lines”; that is doing things that are not
expected, in the way customers unfamiliar with the product may try to do.
The intent is to ensure that bad things don’t happen when users do
► Regression Testing: As problems (bugs) are found
and reported, developers are assigned to fix them.
Once they are fixed, regression testing is performed to both ensure
that the fix actually corrects the problem, and to ensure that the fix
didn’t break something else that had been working before.
All reported bugs must be regression tested and verified as fixed
before a product can be released.
► Black Box Testing: Once a product has been fully
integration tested, functionally tested, system-level tested, etc., while
working in close cooperation with development, it is essential that all
changes are made and a “clean build” of the product be put together in the
same way that will be done once the product has been released to manufacture.
This “clean build” must then be tested as a “black box” that a
customer would receive to verify that all tests it is subjected to operate
properly and completely. At this
point, the testers work independently of development to ensure the product is
truly ready for release.
► Other Testing: Many other types of testing can
and are performed, including ad-hoc testing, acceptance
testing, configuration testing, performance
testing, usability testing, and much more.
Smaller companies may combine many of these tests to reflect the
limited resources they have available, but they still need to cover the needed
territory. The specific testing
methods used are a function of the product and its testing needs.
Test is a critical element
of a development methodology. Too
often it is viewed as an element to correct quality problems.
If viewed that way, the development may rely on Test to find their
quality problems. This is wrong
and far too late in the process. Developers
must be fully disabused of this notion. The
real purpose of Test is to verify that the design is correct, not to
find the problems that development missed. To be effective, quality must be designed in, and Test
will serve to verify that designed-in quality. When this is truly the case, the Test process can be a
straightforward one. Wouldn’t
that be great!?
Effective Engineering Consulting Services, All Rights Reserved