Testing/Assertions

From W3C Wiki

This page will contain information collected by the "Documentation Practices" task force of the Web Testing Interest Group. The goal of this task force is focused on determining the current practices for developing test assertions as part of W3C specs, broadening consensus on the optimum approach to test assertion documentation, and promoting adoption of the consensus approach.

Note: the data below is just a first pass that needs to be triaged to remove specs not being worked on, etc... The list was collected by grep'ing the spec source (spec list at http://www.w3.org/TR/tr-date-all) for the known assertion markup approaches (or markup approaches that could be useful for collecting assertions). The data will be updated to remove specs that are not being worked on, specs for which test suites are not intended, and to add other specs in progress for which the assertion annotation method (if any) is unknown. The goal is to include a reference to all specs in progress, with an indication of the approach used toward developing test assertions.

A Method for Writing Testable Conformance Requirements (identified as '1' below) is the test assertion methodology developed for use in the W3C Widgets specs. It defines a methodology for annotating specifications to enable automated test assertion generation, using "necessary information to create a testable assertion":

  • Test assertions are scoped in paragraphs (<p>) tagged with a test assertion id per convention e.g. "id='ta-get-118n-string'"
  • Test assertions include the processable (where possible) attributes
    • Product (e.g. class='product-ua'): A product that is supposed to follow the requirement — in this case, the “user agent”. (see also the definition of “classes of product” in http://www.w3.org/TR/qaframe-spec/)
    • Strictness level (class='ct'): The strictness of the applicability of the requirement to a product — in this case, “the user agent must” do something. W3C specifications use the [RFC2119] keywords (must, should, may, etc.) to indicate the level of requirement that is imposed on a product.
    • Prerequisites: An explanation of the prerequisites that need to be in place in order for the requirement to apply — in this case, “if the src attribute of the content element is absent or an empty string”.
    • Behavior: a clear explanation of what the product is supposed to do — in this case, “ignore this element”.
    • Terms: Keywords that are relevant to understanding how to apply the desired behavior. For instance, what it actually means to “ignore” (definitively and algorithmically) needs to be specified somewhere in the specification.


Markup conventions referenced below:

  • 1 class='ct' or class=ct
  • 2 <em title="must" class="rfc2119">must</em>
  • 3 <span class="assert must">


Test Assertion Markup Practices in Specs
Group Spec Markup
Convention 		Notes
Audio Working Group Web Audio API TBD
MediaStream Processing API TBD
Audio Processing API TBD
Authoring Tool Accessibility Guidelines Working Group Authoring Tool Accessibility Guidelines 1.0 TBD
Implementing ATAG 2.0 TBD
Authoring Tool Accessibility Guidelines (ATAG) 2.0 TBD
Authoring Tool Accessibility Guidelines "Wombat" TBD
Cascading Style Sheets (CSS) Working Group Selectors Level 3 TBD
CSS Namespaces Module TBD
CSS Color Module Level 3 TBD
Cascading Style Sheets Level 2 Revision 1 (CSS 2.1) Specification TBD
Cascading Style Sheets (CSS1) Level 1 Specification TBD
Ruby Annotation TBD
CSS Text Level 3 TBD
CSS Basic User Interface Module Level 3 (CSS3 UI) TBD
CSS Image Values and Replaced Content Module Level 3 TBD
CSS 2D Transforms TBD
CSS Exclusions and Shapes Module Level 3 TBD
CSS Regions Module Level 3 TBD
CSS Template Layout Module TBD
CSS Generated Content for Paged Media Module TBD
CSS Flexible Box Layout Module TBD
CSS Fonts Module Level 3 TBD
Selectors Level 4 TBD
CSS Device Adaptation TBD
CSS Values and Units Module Level 3 TBD
CSS Writing Modes Module Level 3 TBD
CSS Conditional Rules Module Level 3 TBD
CSS Speech Module TBD
CSSOM View Module TBD
CSSOM TBD
CSS3 Ruby Module TBD
CSS Lists and Counters Module Level 3 TBD
CSS Multi-column Layout Module TBD
Grid Layout TBD
CSS Backgrounds and Borders Module Level 3 TBD
CSS Style Attributes TBD
Media Queries TBD
CSS Generated Content for Paged Media Module TBD
CSS Template Layout Module TBD
CSS Transitions Module Level 3 TBD
CSS Animations Module Level 3 TBD
CSS 3D Transforms Module Level 3 TBD
CSS Mobile Profile 2.0 TBD
CSS Marquee Module Level 3 TBD
Behavioral Extensions to CSS TBD
CSS Grid Positioning Module Level 3 TBD
CSS basic box model TBD
CSS Print Profile TBD
CSS3 Module: Paged Media TBD
CSS3 module: Cascading and inheritance TBD
The CSS 'Reader' Media Type TBD
CSS3 Hyperlink Presentation Module TBD
CSS3 module: Syntax TBD
CSS3 module: Presentation Levels TBD
CSS3 Generated and Replaced Content Module TBD
CSS TV Profile 1.0 TBD
CSS3 module: line TBD
Device APIs Working Group Requirements for Home Networking Scenarios 2
Battery Status API 2
Vibration API 2
Web Application Privacy Best Practices TBD
Contacts API 2
The Network Information API 2
Calendar API 2
The Messaging API 2
HTML Media Capture 2
Permissions for Device API Access TBD
The Media Capture API TBD
The System Information API 2
Education and Outreach Working Group
Efficient XML Interchange Working Group Efficient XML Interchange (EXI) Format 1.0 TBD
Efficient XML Interchange (EXI) Profile TBD
Efficient XML Interchange (EXI) Primer TBD
Efficient XML Interchange Evaluation TBD
Efficient XML Interchange (EXI) Impacts TBD
Efficient XML Interchange (EXI) Best Practices TBD
Efficient XML Interchange Measurements Note TBD
Evaluation and Repair Tools Working Group Pointer Methods in RDF 1.0 TBD
HTTP Vocabulary in RDF 1.0 TBD
Evaluation and Report Language (EARL) 1.0 Schema TBD
Developer Guide for Evaluation and Report Language (EARL) 1.0 TBD
Representing Content in RDF 1.0 TBD
Requirements for the Evaluation and Report Language (EARL) 1.0 TBD
Forms Working Group XForms 1.1 TBD
XForms for HTML TBD
XForms 1.0 Basic Profile TBD
XForms Requirements TBD
Geolocation Working Group DeviceOrientation Event Specification TBD
Geolocation API Specification Level 2 TBD
Geolocation API Specification TBD
HTML Working Group HTML+RDFa 1.1 TBD
HTML Microdata TBD
HTML5 TBD
HTML5 differences from HTML4 TBD
Polyglot Markup: HTML-Compatible XHTML Documents TBD
HTML: The Markup Language TBD
HTML5: Techniques for providing useful text alternatives TBD
HTML to Platform Accessibility APIs Implementation Guide TBD
HTML Canvas 2D Context TBD
HTML Design Principles TBD
Internationalization Core Working Group Character Model for the World Wide Web 1.0: Fundamentals TBD
Requirements for Japanese Text Layout TBD
Additional Requirements for Bidi in HTML TBD
Web Services Internationalization (WS-I18N) TBD
Language Tags and Locale Identifiers for the World Wide Web TBD
Character Model for the World Wide Web 1.0: Normalization TBD
Character Model for the World Wide Web 1.0: Resource Identifiers TBD
Authoring Techniques for XHTML & HTML Internationalization: Characters and Encodings 1.0 TBD
Math Working Group A MathML for CSS Profile TBD
Mathematical Markup Language (MathML) Version 3.0 TBD
XML Entity Definitions for Characters TBD
Mathematical Markup Language (MathML) Version 2.0 (Second Edition) TBD
Mathematical Markup Language (MathML) 1.01 Specification TBD
Media Annotations Working Group Ontology for Media Resources 1.0 TBD
API for Media Resources 1.0 2
Ontology for Media Resources 1.0 TBD
Use Cases and Requirements for Ontology and API for Media Resource 1.0 TBD
Media Fragments Working Group Media Fragments URI 1.0 TBD
Protocol for Media Fragments 1.0 Resolution in HTTP TBD
Use cases and requirements for Media Fragments TBD
OWL Working Group rdf:PlainLiteral: A Datatype for RDF Plain Literals TBD
OWL 2 Web Ontology Language XML Serialization TBD
OWL 2 Web Ontology Language Structural Specification and Functional-Style Syntax TBD
OWL 2 Web Ontology Language RDF-Based Semantics TBD
OWL 2 Web Ontology Language Quick Reference Guide TBD
OWL 2 Web Ontology Language Profiles TBD
OWL 2 Web Ontology Language Primer TBD
OWL 2 Web Ontology Language Document Overview TBD
OWL 2 Web Ontology Language New Features and Rationale TBD
OWL 2 Web Ontology Language Mapping to RDF Graphs TBD
OWL 2 Web Ontology Language Direct Semantics TBD
OWL 2 Web Ontology Language Conformance TBD
Points of Interest (POI) Working Group Points of Interest Core 2
Protocols and Formats Working Group WAI-ARIA 1.0 User Agent Implementation Guide TBD
Media Accessibility User Requirements TBD
Accessible Rich Internet Applications (WAI-ARIA) 1.0 TBD
Role Attribute 1.0 2
WAI-ARIA 1.0 Primer TBD
WAI-ARIA 1.0 Authoring Practices TBD
Roadmap for Accessible Rich Internet Applications (WAI-ARIA Roadmap) TBD
XML Accessibility Guidelines TBD
Provenance Working Group PROV Model Primer TBD
PROV-AQ: Provenance Access and Query TBD
The PROV Data Model and Abstract Syntax Notation 2
The PROV Ontology: Model and Formal Semantics 2
RDB2RDF Working Group A Direct Mapping of Relational Data to RDF TBD
R2RML: RDB to RDF Mapping Language TBD
Use Cases and Requirements for Mapping Relational Databases to RDF TBD
RDF Web Applications Working Group XHTML+RDFa 1.1 2
RDFa Lite 1.1 2
RDFa Core 1.1 2
RDFa 1.1 Primer TBD
HTML+RDFa 1.1 2
RDF Interfaces 1.0 2
RDFa API 2
RDF Working Group RDF 1.1 Concepts and Abstract Syntax 2
Turtle 2
Rule Interchange Format Working Group RIF RDF and OWL Compatibility TBD
RIF Production Rule Dialect TBD
RIF Framework for Logic Dialects TBD
RIF Datatypes and Built-Ins 1.0 TBD
RIF Core Dialect TBD
RIF Basic Logic Dialect TBD
rdf:PlainLiteral: A Datatype for RDF Plain Literals TBD
RIF Combination with XML data TBD
RIF Test Cases TBD
RIF Use Cases and Requirements TBD
SOAP-JMS Binding Working Group SOAP over Java Message Service 1.0 TBD
SPARQL Working Group SPARQL Query Language for RDF TBD
SPARQL Protocol for RDF TBD
SPARQL Query Results XML Format TBD
SPARQL 1.1 Update TBD
SPARQL 1.1 Service Description TBD
SPARQL 1.1 Query Language TBD
SPARQL 1.1 Protocol TBD
SPARQL 1.1 Entailment Regimes TBD
SPARQL 1.1 Overview TBD
SPARQL 1.1 Federated Query TBD
SPARQL 1.1 Query Results JSON Format TBD
SPARQL 1.1 Query Results CSV and TSV Formats 2
SPARQL 1.1 Graph Store HTTP Protocol TBD
SPARQL 1.1 Property Paths TBD
SPARQL New Features and Rationale TBD
SPARQL Protocol for RDF Using WSDL 1.1 TBD
RDF Data Access Use Cases and Requirements TBD
SVG Working Group Scalable Vector Graphics (SVG) 1.1 (Second Edition) TBD
Scalable Vector Graphics (SVG) Tiny 1.2 Specification TBD
Mobile SVG Profiles: SVG Tiny and SVG Basic TBD
SMIL Animation TBD
SVG Compositing Specification TBD
SVG Color 1.2, Part 1: Primer TBD
SVG Color 1.2, Part 2: Language TBD
SVG Parameters 1.0, Part 1: Primer TBD
SVG Parameters 1.0, Part 2: Language TBD
SVG Transforms 1.0, Part 2: Language TBD
SVG Filter Requirements TBD
SVG Filters 1.2, Part 1: Primer TBD
SVG Filters 1.2, Part 2: Language TBD
Media Access Events TBD
SVG 1.1/1.2/2.0 Requirements TBD
SYMM Working Group Synchronized Multimedia Integration Language (SMIL 3.0) TBD
SMIL Timesheets 1.0 TBD
Technical Architecture Group Architecture of the World Wide Web, Volume One TBD
Usage Patterns For Client-Side URI parameters TBD
The Disposition of Names in an XML Namespace TBD
Timed Text Working Group Timed Text Markup Language (TTML) 1.0 TBD
Tracking Protection Working Group Tracking Preference Expression (DNT) 2
Tracking Compliance and Scope 2
User Agent Accessibility Guidelines Working Group User Agent Accessibility Guidelines 1.0 TBD
User Agent Accessibility Guidelines (UAAG) 2.0 TBD
Implementing UAAG 2.0 TBD
User Agent Accessibility Guidelines 2.0 Requirements TBD
Web Application Security Working Group Content Security Policy 2
Web Applications Working Group Widget Packaging and XML Configuration 1
Element Traversal Specification TBD
XMLHttpRequest Level 2 TBD
DOM4 TBD
Widget Interface 1
Widget Access Request Policy 1
Web Storage TBD
The WebSocket API TBD
Indexed Database API 2
HTML5 Web Messaging TBD
Server-Sent Events TBD
File API TBD
Widget URI scheme 1
Web IDL TBD
Progress Events TBD
Web Workers TBD
XML Digital Signatures for Widgets 1
The 'view-mode' Media Feature 2
The From-Origin Header TBD
Document Object Model (DOM) Level 3 Events Specification 3
File API: Writer 2
File API: Directories and System 2
Widget Updates 2
Cross-Origin Resource Sharing 1
Uniform Messaging Policy, Level One 1
Selectors API Level 2 1
Selectors API Level 1 1
Web Content Accessibility Guidelines Working Group Web Content Accessibility Guidelines (WCAG) 2.0 TBD
Web Content Accessibility Guidelines 1.0 TBD
Web Events Working Group Touch Events version 1 2
Web Notification Working Group Web Notifications TBD
Web Performance Working Group Navigation Timing TBD
User Timing TBD
Performance Timeline TBD
Resource Timing TBD
Page Visibility TBD
Timing control for script-based animations TBD
Web Real-Time Communications Working Group WebRTC 1.0: Real-time Communication Between Browsers 2
Web Services Policy Working Group Web Services Policy 1.5 - Attachment TBD
Web Services Policy 1.5 - Framework TBD
Web Services Resource Access Working Group Web Services Transfer (WS-Transfer) TBD
Web Services SOAP Assertions (WS-SOAPAssertions) TBD
Web Services Metadata Exchange (WS-MetadataExchange) TBD
Web Services Fragment (WS-Fragment) TBD
Web Services Eventing (WS-Eventing) TBD
Web Services Event Descriptions (WS-EventDescriptions) TBD
Web Services Enumeration (WS-Enumeration) TBD
WebFonts Working Group WOFF File Format 1.0 TBD
XML Core Working Group Associating Style Sheets with XML documents 1.0 (Second Edition) TBD
XML Linking Language (XLink) Version 1.1 TBD
Namespaces in XML 1.0 (Third Edition) TBD
XML Base (Second Edition) TBD
Extensible Markup Language (XML) 1.0 (Fifth Edition) TBD
Canonical XML Version 1.1 TBD
XML Inclusions (XInclude) Version 1.0 (Second Edition) TBD
Extensible Markup Language (XML) 1.1 (Second Edition) TBD
Namespaces in XML 1.1 (Second Edition) TBD
xml:id Version 1.0 TBD
XML Information Set (Second Edition) TBD
Associating Style Sheets with XML documents 1.0 (Second Edition) TBD
XML Fragment Interchange TBD
XML Print and Page Layout Working Group Extensible Stylesheet Language (XSL) Version 2.0 TBD
XML Processing Model Working Group XProc: An XML Pipeline Language TBD
XML processor profiles TBD
XML Query Working Group XQuery Update Facility 1.0 TBD
XQuery and XPath Full Text 1.0 TBD
XSLT 2.0 and XQuery 1.0 Serialization (Second Edition) TBD
XML Syntax for XQuery 1.0 (XQueryX) (Second Edition) TBD
XQuery 1.0 and XPath 2.0 Formal Semantics (Second Edition) TBD
XQuery 1.0: An XML Query Language (Second Edition) TBD
XML Path Language (XPath) 2.0 (Second Edition) TBD
XQuery 1.0 and XPath 2.0 Functions and Operators (Second Edition) TBD
XQuery 1.0 and XPath 2.0 Data Model (XDM) (Second Edition) TBD
XSLT and XQuery Serialization 3.0 TBD
XQueryX 3.0 TBD
XQuery Update Facility 3.0 TBD
XQuery 3.0: An XML Query Language TBD
XPath and XQuery Functions and Operators 3.0 TBD
XQuery and XPath Full Text 3.0 TBD
XQuery and XPath Data Model 3.0 TBD
XML Path Language (XPath) 3.0 TBD
XQuery 3.0 Use Cases TBD
XQuery 3.0 Requirements TBD
XQuery Scripting Extension 1.0 TBD
XQuery Scripting Extension 1.0 Use Cases TBD
XQuery Scripting Extension 1.0 Requirements TBD
XML Schema Working Group XML Schema Part 2: Datatypes Second Edition TBD
XML Schema Part 1: Structures Second Edition TBD
XML Schema Part 0: Primer Second Edition TBD
W3C XML Schema Definition Language (XSD) 1.1 Part 2: Datatypes TBD
W3C XML Schema Definition Language (XSD) 1.1 Part 1: Structures TBD
W3C XML Schema Definition Language (XSD): Component Designators TBD
Guide to Versioning XML Languages using new XML Schema 1.1 features TBD
XML Security Working Group XML Signature Syntax and Processing (Second Edition) TBD
XML Signature Streaming Profile of XPath 1.0 TBD
XML Signature Syntax and Processing Version 2.0 2
Canonical XML Version 2.0 2
XML Security Algorithm Cross-Reference TBD
XML Encryption 1.1 CipherReference Processing using 2.0 Transforms 2
XML Encryption Syntax and Processing Version 1.1 2
Test cases for x 1.1 TBD
Test cases for Canonical XML 2.0 TBD
XML Signature Best Practices TBD
XML Security 2.0 Requirements and Design Considerations TBD
XML Security 1.1 Requirements and Design Considerations 2
XML Security Generic Hybrid Ciphers 2
XML Signature Properties 2
XML Signature Syntax and Processing Version 1.1 1
XSLT Working Group XQuery and XPath Full Text 1.0 TBD
XSLT 2.0 and XQuery 1.0 Serialization (Second Edition) TBD
XQuery 1.0 and XPath 2.0 Formal Semantics (Second Edition) TBD
XML Path Language (XPath) 2.0 (Second Edition) TBD
XQuery 1.0 and XPath 2.0 Functions and Operators (Second Edition) TBD
XQuery 1.0 and XPath 2.0 Data Model (XDM) (Second Edition) TBD
XSL Transformations (XSLT) Version 2.0 TBD
Extensible Stylesheet Language (XSL) Version 1.1 TBD
Extensible Stylesheet Language (XSL) Version 1.0 TBD
XSL Transformations (XSLT) Version 1.0 TBD
XSLT and XQuery Serialization 3.0 TBD
XPath and XQuery Functions and Operators 3.0 TBD
XQuery and XPath Full Text 3.0 TBD
XQuery and XPath Data Model 3.0 TBD
XML Path Language (XPath) 3.0 TBD
Requirements for Japanese Text Layout TBD
Requirements and Use Cases for XSLT 2.1 TBD
XSL Transformations (XSLT) Version 2.1 TBD
XSL Transformations (XSLT) Version 2.0 (Second Edition) TBD
Extensible Stylesheet Language (XSL) Requirements Version 2.0 TBD
Retrieved from "https://www.w3.org/wiki/index.php?title=Testing/Assertions&oldid=56245"