Test Assertion Guide

Abstract

The definition and provision of metadata has proved helpful in a variety of ways during the test development and test execution processes. This document defines a minimal set of metadata elements that can usefully be applied to tests that are intended for publication within a test suite.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

You may email comments on this document to www-qa@w3.org with the subject of your email "[comments] Test Assertion Guide" and one comment by email, the publicly archived list of the QA Interest Group.

@@Patent policy ??@@

This document has been produced as part of the W3C Quality Assurance Interest Group (QAIG). The goals of the QAIG are discussed in the QA Interest Group charter. The QAIG is part of the Quality Assurance activity at the W3C.

This document is a preliminary outline for a general guideline for defining and creating test assertions.  It is a draft document that does not fully represent the consensus of the group at this time. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to cite this document as anything other than a “work in progress”. Translations of this document may be available.

Table of contents

• Status
• 1. Introduction
• --Scope, Audience
• 2. Rationale and Benefits of Test Assertions
• 3. Definitions (what is a TA)
• 4. What is a TA?
• 5. Methods for building TAs
• 6. Steps to Building TAs (regardless of method)
• 7. General, guiding principles
• Acknowledgments
• References

1. Introduction

--Scope, Audience

This document is a guide to defining and creating test assertions for specifications. Its purpose is to help the reader understand what test assertions are, why create them and most importantly, how to create them. As you will discover, there are no definitive definitions or ways to create test assertions. Achieving a standard way to represent these is not as important as being able to complement a specification with a well defined set of test assertions that can be used later as a starting point for a conformance test suite and that will provide invaluable insights on the meaning of conformance for this specification. We will share with you our experiences in developing assertions, lessons learned, tricks and tools we found helpful, hazards to avoid, and other tidbits of knowledge that may be helpful in crafting test assertions.

This document is organized to appeal to both the novice and those more experienced in creating assertions. The document presents basic definitions, rationale for creating assertions, when to create them, and the steps and principles to follow in order to create them. Throughout the document, we will provide examples and samples to illustrate what is presented. We hope that after using this guide, you will believe in the benefits of test assertions and be ready and able to create your own.

2. Rationale and Benefits of Test Assertions

--Can be used as a starting point for developing conformance tests.

--The activity of defining test assertions has also proved very beneficial in helping tighten the loose ends of a specification, something that is better done as early as possible in the standardization process.

3. Definitions (what is a TA)

Many definitions, depends on your perspective.

Should we include different definitions and their sources or agree on a definition or both?

4. What is a TA?

What are the components that make up a TA - Pre condition, condition, post condition?

What constitutes a TA?

5. Methods for building TAs

Can be manual, automatically generated, any other ways?

6. Steps to Building TAs (regardless of method)

What is the thought process? How do you determine pre and post conditions? How do you break a statement into small testable chunks? What are you looking for?

7. General, guiding principles

Apply the following principles when creating TAs:

1. Do restrict a TA to an atomic, simple statement by:
   1. Addressing one feature at a time.
   2. Keeping each TA as simple as possible. Multiple single-feature TA are easier to test than a multi-part assertion. Also, identifying the source of a failure to achieve conformance for a given function is easier when assertions are not multi-part.
   3. Group assertions into logical sets. Consider ordering them in a natural progression, beginning with easiest. This makes your document easier to read and later testing and trace-back more sensible (see next item). If an implementation can't support an 'easy' assertions, then it is unlikely to support a more complex one.
   4. Ensure traceability of assertions to text (a statement, sections) in the specification.
2. Do make your TA technology-neutral.
3. Do not change the semantics of the specification.
4. Do not mix terminologies, by doing the following:
   1. Using a Glossary agreed upon by all.
   2. Keeping the balance of your text self-contained, with as few footnotes and external references as practical
   3. Avoiding sets of terms that assign different interpretations to the same words.
5. Do constrain options and allowed values, by:
   1. Describing features, values, attributes, etc. to be measured and indications of success or failure.

   2. Examples:      ... shall offer A, B, or C, and no others
      
      ... occurs one or more times
      

6. Do indicate explicit dependencies and constraints.
7. Do not state how to test.
8. Do not rely upon formatting or context to convey intentions.

Acknowledgements

The following QA Interest Group participants have contributed significantly to this document:

• Tim Boland (NIST),
• Jacques Durand (@@Fujitsu@@),
• Karl Dubost (W3C),
• Dominique Hazaël-Massieux (W3C),
• Lofton Henderson (CGM Open),
• Patrick Curran (Sun Microsystems),
• Lynne Rosenthal (NIST),
• Mark Skall (NIST).

4. References

SPEC-VARIABILITY

Variability in Specifications, D. Hazaël-Massieux, L. Rosenthal, Editors, W3C Working Group Note, 31 August 2005, http://www.w3.org/TR/2005/NOTE-spec-variability-20050831/ . Latest version available at http://www.w3.org/TR/spec-variability/ .