Testing/Assertions
From W3C Wiki
This page will contain information collected by the "Documentation Practices" task force of the Web Testing Interest Group. The goal of this task force is focused on determining the current practices for developing test assertions as part of W3C specs, broadening consensus on the optimum approach to test assertion documentation, and promoting adoption of the consensus approach.
Note: the data below is just a first pass that needs to be triaged to remove specs not being worked on, etc... The list was collected by grep'ing the spec source (spec list at http://www.w3.org/TR/tr-date-all) for the known assertion markup approaches (or markup approaches that could be useful for collecting assertions). The data will be updated to remove specs that are not being worked on, specs for which test suites are not intended, and to add other specs in progress for which the assertion annotation method (if any) is unknown. The goal is to include a reference to all specs in progress, with an indication of the approach used toward developing test assertions.
A Method for Writing Testable Conformance Requirements (identified as '1' below) is the test assertion methodology developed for use in the W3C Widgets specs. It defines a methodology for annotating specifications to enable automated test assertion generation, using "necessary information to create a testable assertion":
- Test assertions are scoped in paragraphs (<p>) tagged with a test assertion id per convention e.g. "id='ta-get-118n-string'"
- Test assertions include the processable (where possible) attributes
- Product (e.g. class='product-ua'): A product that is supposed to follow the requirement — in this case, the “user agent”. (see also the definition of “classes of product” in http://www.w3.org/TR/qaframe-spec/)
- Strictness level (class='ct'): The strictness of the applicability of the requirement to a product — in this case, “the user agent must” do something. W3C specifications use the [RFC2119] keywords (must, should, may, etc.) to indicate the level of requirement that is imposed on a product.
- Prerequisites: An explanation of the prerequisites that need to be in place in order for the requirement to apply — in this case, “if the src attribute of the content element is absent or an empty string”.
- Behavior: a clear explanation of what the product is supposed to do — in this case, “ignore this element”.
- Terms: Keywords that are relevant to understanding how to apply the desired behavior. For instance, what it actually means to “ignore” (definitively and algorithmically) needs to be specified somewhere in the specification.
Markup conventions referenced below:
- 1 class='ct' or class=ct
- 2 <em title="must" class="rfc2119">must</em>
- 3 <span class="assert must">
