CSS Test Suite Documentation

This Version:
http://www.w3.org/Style/CSS/Test/testsuitedocumentation-20030129.html
Latest Version:
http://www.w3.org/Style/CSS/Test/testsuitedocumentation.html
Previous Versions:
http://www.w3.org/Style/CSS/Test/testsuitedocumentation-20020429.html
Editors:
Tantek Çelı̇k, Microsoft Corporation, tantekc@microsoft.com
Ian Hickson, ian@hixie.ch

Abstract

[abstract]

Table of contents

1. Introduction

This document explains how to use the current CSS test suites, provides a description of the officially adopted CSS test suite format, and instructs how to write a test suite in this format.

Sample text and hyperlinks from test suites are presented in this document with a thin grey border like this: test suite text. test suite hyperlink. Sample hyperlinks from test suites are not underlined to avoid misrepresenting them as actual hyperlinks in this document.

2. History

The CSS working group published the first version of the CSS1 Test Suite on the 11th of May 1998. This test suite introduced much of the format and structure of current CSS test suites. On 24th October 2001, the CSS working group published the CSS Mobile Profile 1.0 Candidate Recommendation and decided that the mobile profile also required a test suite.

Since the mobile profile referred to features already defined in previous CSS specifications, there was a strong desire to reuse previous tests rather than write new tests. During a CSS working group teleconference it was suggested that the tests in the test suite should be separated into test pages and navigation pages, specifically using the <OBJECT> element in order to facilitate reuse of the tests themselves.

Based on that discussion, a member of the CSS Working Group took the initiative to attempt a test suite experiment - a version of the CSS1 Test Suite with its test pages separated into navigation pages and test pages. The TNG Test Suite Experiment was demonstrated at the August 2002 CSS Working Group ftf meeting.

As noted in the minutes of the meeting, numerous implementations were able to use the new test suite format without difficulty, including a cell phone browser. The TNG format was subsequently adopted as the official primary test suite format for test suites produced by the CSS working group. The CSS1 Test Suite was updated to use the new format, and the CSS Mobile Profile 1.0 Test Suite was produced in the new format. In addition, the Selectors Test Suite now includes several TNG versions for each of the document languages tested.

3. Using the test suites

As of this writing there are at least three test suites which use the official CSS test suite format:

Most of these are directly written in the TNG format (CSS1, CSS Mobile Profile 1.0), and thus their cover pages are consistent and recognizable. The Selectors Test Suite is written in a meta-format which is then used to generate several variants, including the TNG format, for several document languages. For such test suites, you have to first click on a link or two to select the cover page for the variant of the document language that you want to test.

3.1. Cover page

The cover page of a test suite contains a table of contents, structured parallel to the specification being tested. Often a test suite contains a -- Prologue which documents among other things requirements or recommendations for a specific system configuration, or suggestions for setting your system preferences in such a way as to minimize their interference with the tests themselves.

3.2. Getting started with the tests

Get started with running the tests in a test suite by clicking on the first numbered link in the table of contents (skipping over the -- Prologue), or by clicking the get started here hyperlink that many test suites provide on their cover page.

3.3. Next and Previous

On every following page, there will be a [Next] hyperlink that will take you linearly through each section, subsection and test of the test suite, finally bringing you back to the cover page at which point you have completed the test suite. On any of these pages you can always backtrack this linear order by following the [Previous] hyperlinks.

3.4. Test content

On every page that contains a test, you should see the test content immediately below the abovementioned hyperlinks. Instead of the test itself, you see a Test hyperlink, then your browser doesn't properly support the <OBJECT> element. No matter, simply click the Test hyperlink to navigate to the test itself. After viewing the test, use the Back functionality in your web browser to return to the navigation page for that test and proceed to the next page by clicking the [Next] hyperlink.

The test content is self-describing - that is, the test content contains a textual description of what the page should look like according to the test, e.g. This sentence should be green.

4. Test Suite Principles

The current test suite format is the evolutionary culmination of the work of many contributors over several years. Along the way several principles were identified that aided in the design and advancement of the format.

4.1. Valid tests

This should go without saying, yet the existence of numerous invalid 3rd party test suites (even some hosted at W3C!) which do not pass the simplest of validators underscores the importance of emphasizing this principle. In essence, all files in the test suite, whether navigational or tests themselves must at a minimum pass the W3C Validator. In addition, if the relevant selectors, @-rules, properties and values have been added to the W3C CSS Validator, then the all test suite files should also pass the W3C CSS Validator.

4.2. Simple format

Simplicity is always a good design principle. In the case of a test suite, keeping the format simple helps minimize unintended effects from the format itself. In addition, a simple format also makes it easier to author the test and navigation pages.

4.3. Ease of authoring

There are very few easy to access and easy to run test suites at W3C. The ratio of RECs to Test Suites is large and only getting larger, when it should be shrinking until it approaches 1. Others have noted that the task of authoring test suites often demands the allocation of significant time and resources. To reduce the effective cost of producing test suites, and therefore hopefully increase the number of test suites that are produced, it is only logical to adopt of the principle of ease of authoring.

Not wanting to depend on any particular tools, or any particular platform, or device for that matter, we interpret the principle of ease of authoring to imply that the test suite format must be easily hand authorable, that is, easily authorable with only the use of a simple text editor.

4.4. Something rather than nothing

Another way of looking at the problem of a dearth of W3C test suites, is to ask which is the bigger problem - the lack of fully automated W3C test suites which are presented in numerous variants for every conceivable document language, or the lack of W3C test suites period? Is a simple test suite better than none at all? The experience of the CSS working group, with the CSS1 Test Suite, and the many compliant and interoperable implementations that followed, demonstrated fairly clearly that a simple test suite is far far better than none at all.

4.5. Sooner rather than later

Corollary: a simple test suite within a few months is more useful, and thus preferred to, a more complex, more fully featured test suite a year or more from now.

4.6. Test the technology being tested, not other technologies

When writing a test suite for a particular technology (e.g. CSS), it is often necessary to utilize other technologies (e.g. HTML, DOM etc.) in the construction of that test suite. Test suite authors must be careful while testing one technology, avoid testing other technologies. Those other technologies should be authored to be maximally compatible, in a way that is most forgiving to the various user agents which will be using the test suite.

Specifically, when writing CSS test suite should not try to test:

Consequently, CSS test suite files should:

4.7. Atomic tests

A test suite should contain at least one atomic (meaning, testing a single feature at a time) test per feature or area.

4.8. Feature coverage

A test suite should contain at least one basic test for each explicit feature of a technology. In the case of CSS, there should be at least one test for each rule, selector, property and value type. Note that some value types, such as <length>, may take a number of different units. For such value types, there should be at least one test for each unit, but that does not mean that every such unit must be tested on every property.

In addition, a number of additional composite, devious, evil, and failure (error checking) tests should be written for any issue that is likely to be mis-implemented. In writing these tests, experience with existing implementations is a great help. As implementations progress, new areas worthy of being tested will come to light, and the test suite should be updated regularly to track these developments.

4.9. Self describing results

Test cases should describe in prose, in the test cases themselves, what is expected. Each test case should at a minimum describe correct behavior. Error handling tests should at a minimum describe incorrect, or possibly unexpected behavior.

4.10. Incremental difficulty

Test suites should begin with simpler, easier test cases, and proceed to more complex, more challenging tests. This makes sense both from a test case authoring perspective, since simpler cases are easier to author, and from an technology implementer's perspective, since simpler cases are easier to implement.

4.11. Non-requirements

Finally, as with any set of principles and requirements, there are some non-requirements. These non-requirements are exactly that - they are features not required of a test suite. They may be desirable, or even included - yet they are not necessary to produce a test suite.

4.12. Additional principles

A separate document has been written to give guidelines and techniques to use when authoring CSS test cases.

The SVG Test Manual documents additional Principles Applicable to Test Suite Content . Future test suites authors should consult these principles as well.

5. Test Suite Format

5.1. Test suite pages

The test suite format consists of several different kinds of pages. At the core are the test pages themselves that exercise the features of the technology. Each test page is referenced by a navigation page that also references the previous / next navigation pages, and its section page. The test suite starts with a cover page, which links directly to each of the section pages. This document describes these pages from the inside out.

5.2. Test page format

The test page format is documented in a separate document, which also includes a template.

Each test page is contained within a navigation page. The navigation page starts with a heading documenting the test suite, followed by the numerical section of the specification being tested which is also used for the page <TITLE>, e.g. CSS1 Test Suite: 1.1 Containment in HTML. A horizontal rule separates the heading from the following links:

[Previous] [Next] [Section] [Contents] [Specification]

The Previous and Next hyperlinks link to the previous and next navigation pages respectively. Previous of the first navigation page of a section links to that section page. Next of the last navigation page of a section links to the next section. And Next of the last navigation page in the suite links to the cover page.

The Section hyperlink links to the section page for all the navigation pages for this section of the specification being tested. The Contents hyperlink links to the cover page of the test suite. The Specification hyperlink links to the specific section of the specification being tested.

The navigation page transcludes the test page with an <OBJECT> element, which references the test file using the data attribute and explicitly sets the type of the expected data to "text/html". In order to encourage a user agent to use the entire width of the document, and the remainder of the document, both the width and height attributes are set to 100%.

The <OBJECT> element was chosen because it is present in all "current" versions of HTML, from HTML 4.0 Transitional, to XHTML Basic, to XHTML 1.1. This permits user agents which support any one of these languages to easily utilize the navigational pages. In addition, the <OBJECT> element provides an excellent immediate fall back mechanism for user agents that either happen to not support <OBJECT>, or that have difficulties transcluding HTML content.

Inside the <OBJECT> element is a simple Test hyperlink which directly links to the test file. Thus user agents that do not recognize the <OBJECT> element and therefore ignore its markup, show the simple hyperlink instead, thereby providing access to the test file.

The navigation pages use valid HTML 4.0 Transitional, which is the most compatible version of HTML for present day user agents. In the future, XHTML 1.0 Transitional which follows the XHTML 1.0 Appendix C guidelines may be more compatible across more user agents. XHTML Basic, again following the XHTML 1.0 Appendix C guidelines, may also be more compatible. Since the navigation pages are separated from the test pages, this could easily change in the future without having to change the test pages. Similarly, the test pages could be updated to use XHTML Basic for example without having to update the navigational pages.

Lastly, the navigation pages are named "sec" followed by the section number of the section being tested (without periods, e.g. 1.1 becomes 11) with the extension ".htm". E.g. "sec11.htm". Note that the navigation file has no test content whatsoever. This allows the test file to be updated to fix problems in tests without having to update navigation files.

5.4. Section page format

For each chapter in the specification, there is a corresponding section page which hyperlinks to the navigation pages of all the tests for that chapter. The section page format very much resembles the navigation page format.

The section page contains a flat list of hyperlinks to the specific navigation pages, in specification order, and labeled with the respective section and subsection (and potentially subsubsection) of the specification. E.g. 1.1 Containment in HTML

The list is demarcated on both top and bottom by horizontal rules, and a link bar with the following links:

[Previous] [Next] [Contents] [Specification]

The Previous hyperlink links to the last navigation page of the previous section. The Previous of the first section (typically a -- Prologue section) simply links to the test suite cover page. The Next hyperlink links to the first navigation page of this section.

The Contents hyperlink links to the cover page, and the Specification hyperlink links to the respective section in the specification.

Similar to the navigation pages, the section pages are also titled with a heading documenting the name of the test suite, followed by the numerical section of the specification. This heading is also used for the page <TITLE>, e.g. CSS1 Test Suite: 1.0 Basic Concepts. The section pages are also written in valid HTML 4.0 Transitional, and named "sec" followed by the top level section number with a trailing "0" and the extension ".htm". E.g. "sec10.htm".

5.5. Cover page format

A test suite's cover page begins with the W3C logo and a heading stating the name of the test suite which is also used for the page <TITLE>. E.g.

W3C CSS1 Test Suite

A short paragraph of text introduces the test suite, and documents its purpose, usage, and any caveats. Another paragraph minimally explains the structure of the test suite itself. Any additional test suite specific documentation should be placed in the -- Prologue. A subheading labels the Table of Contents which immediately follows the introductory text.

The table of contents is a simple list of numbered hyperlinks ( e.g.: 1.0 Basic Concepts ) to the section pages. Following the numbered section pages for the tests, there may be one or more lettered pages for test suite appendices, such as A. Acknowledgments, B. Test Suite Errata, and C. Related Resources.

A horizontal rule separates the Table of Contents from the cover page footer. The footer contains a mailto hyperlink for test suite feedback, the proper validation stamps (HTML, CSS), and recognition of the organizations responsible for the production of the test suite, along with their respective logos.

6. Writing a Test Suite

The authoring of a test suite for a particular specification begins with the identification of testable assertions in the specification grouped by section and subsection of the specification.

Try to make use of existing test pages. Some specifications, such as profiles, do not contain new features of their own. New versions of specifications often build upon features of older specifications. In both cases, it should be possible, and it is desirable, to reuse existing test cases.

Start by creating a new directory for your test suite, and copying the necessary support files (listed below). All test suite files for your test suite will go into this one directory.

Using the templates listed below, author navigation pages for each feature and their respective testable assertions. If possible, author navigation pages which refer to tests from previous test suites. Use a relative URL in the data attribute of the <OBJECT> tag, and href of the anchor <A>, to refer to test pages from previous test suites. Replace the heading and document <TITLE> with the name of the test suite, the subsection number and title. Fix the link bar hyperlinks (Previous, Next, Section, Specification) to point to the appropriate files.

For navigation pages which require new tests, write new test pages, also starting with the template listed below. Be sure to update the document <TITLE> to be the same as the corresponding navigation page.

After completing all the navigation (and necessary test) pages for a section in the specification, write the section page (again, there is a template which can be used as a starting point). Once the section pages have been completed, construct the cover page and related support pages (Prologue, Acknowledgments, Test Suite Errata (hopefully fairly empty to begin with), and Related Resources). Templates aren't included for for the related support pages, but they can be easily written starting from the respective pages in the CSS1 Test Suite.

Congratulations, you've authored a test suite.

6.1. Support Files

6.2. Templates

These templates are derived from the latest version of the CSS1 Test Suite, with a small bit of cleanup, so there may be minor inconsequential differences.

The template for a navigation page is shown below, and is also available as a separate file: secnntemplate.htm

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<HTML><HEAD><TITLE>CSSn Test Suite: n.n Subsection Title </TITLE>
<META http-equiv="Content-Style-Type" content="text/css">
<LINK rel="stylesheet" type="text/css" media="screen" href="static.css"></HEAD>

<BODY>
<DIV class="navigation">
<H2>CSSn Test Suite: n.n Subsection Title</H2>

<HR>
[<A HREF="secnn.htm">Previous</A>] [<A HREF="secnn.htm">Next</A>] [<A HREF="secnn.htm">Section</A>] [<A HREF="index.html">Contents</A>] [<A HREF="http://www.w3.org/TR/">Specification</A>]<BR>
</DIV>
<OBJECT height="100%" width="100%" border="0" type="text/html" data="testnn.htm"><A class="navigation" href="testnn.htm" target="testwindow">Test</A></OBJECT>
</BODY></HTML>

6.2.2. Section page

The template for a section page is shown below, and is also available as a separate file: secn0template.htm

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<HTML><HEAD><TITLE>CSSn Test Suite: n.0 Section Title</TITLE>
<META http-equiv="Content-Style-Type" content="text/css">
<LINK rel="stylesheet" type="text/css" media="screen" href="static.css">
</HEAD>

<BODY class="navigation">
<H2>CSSn Test Suite: n.0 Section Title</H2>

<HR>
[<A HREF="secnn.htm">Previous</A>] [<A HREF="secnn.htm">Next</A>] [<A HREF="index.html">Contents</A>] [<A HREF="http://www.w3.org/TR/">Specification</A>]<BR>
<HR>
<P>
In this section:
</P>
<UL>
<LI>n.1 <TT><A HREF="secn1.htm">Subsection 1</A></TT></LI>
<LI>n.2 <TT><A HREF="secn2.htm">Subsection 2</A></TT></LI>
<LI>n.3 <TT><A HREF="secn3.htm">Subsection 3</A></TT></LI>
<LI>n.4 <TT><A HREF="secn4.htm">Subsection 4</A></TT></LI>
<LI>n.5 <TT><A HREF="secn5.htm">Subsection 5</A></TT></LI>
<LI>n.6 <TT><A HREF="secn6.htm">Subsection 6</A></TT></LI>
<LI>n.7 <TT><A HREF="secn7.htm">Subsection 7</A></TT></LI>
</UL>
<HR>
[<A HREF="secnn.htm">Previous</A>] [<A HREF="secnn.htm">Next</A>] [<A HREF="index.html">Contents</A>] [<A HREF="http://www.w3.org/TR/">Specification</A>]<BR>
</BODY></HTML>

6.2.3. Cover page

The template for the cover page is shown below, and is also available as a separate file: indextemplate.htm

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<HTML><HEAD><TITLE>CSSn Test Suite</TITLE>
<META http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<META http-equiv="Content-Style-Type" content="text/css">
<LINK REL="stylesheet" TYPE="text/css" MEDIA="screen" HREF="static.css">
<LINK rel="alternate stylesheet" type="text/css" href="W3C-rsrc-test.css" media="screen" title="Experimental W3C Styles">
<STYLE TYPE="text/css">
H1 {border: solid 1px black; border-left: 3px solid black; background-color: #FFFFFF;}
H4 {border-bottom: 1px solid black; margin-left: 2em; margin-right: 5em; font-variant: small-caps; margin-bottom: 0.5em;}
UL {list-style-type: none;}
 .validated { text-align:left; font-size: smaller }
</STYLE>
</HEAD>

<BODY CLASS="navigation">
<H1><A HREF="http://www.w3.org/"><IMG BORDER="0" SRC="http://www.w3.org/Icons/WWW/w3c_home" ALT="W3C" WIDTH="72" HEIGHT="48" align="middle"></A> 
CSSn Test Suite</H1>

<P>
The CSSn Test Suite is provided as a way for vendors and page authors to test 
their browser's conformance to the CSSn specification.
</P>
<P>
For each feature, there is at least one page which tests the feature
in various ways, using HTML 4.0 markup.  The test pages are broken out into 
a number of sections which reflect the structure of the CSSn specification itself, 
in addition to a Prologue and a few appendicies.
</P>

<H4>Table of Contents</H4>

<UL>
<LI style="font-weight: bold;">-- <A HREF="sec00.htm">Prologue</A></LI>
</UL>
<UL>
<LI>1.0 <A HREF="sec10.htm">Section 1</A></LI>
<LI>2.0 <A HREF="sec20.htm">Section 2</A></LI>
<LI>3.0 <A HREF="sec30.htm">Section 3</A></LI>
<LI>4.0 <A HREF="sec40.htm">Section 4</A></LI>
<LI>5.0 <A HREF="sec50.htm">Section 5</A></LI>
<LI>6.0 <A HREF="sec60.htm">Section 6</A></LI>
<LI>7.0 <A HREF="sec70.htm">Section 7</A></LI>
</UL>
<UL style="margin-top: 0;">
<LI> A. <A HREF="tsack.html">Acknowledgments</A></LI>
<LI> B. <A HREF="tserr.html">Test Suite Errata</A></LI>
<LI> C. <A HREF="tsref.html">Related Resources</A></LI>
</UL>

<HR>

<P>
Feedback regarding the CSSn Test Suite should be sent to 
<TT><A HREF="mailto:public-css-testsuite@w3.org">public-css-testsuite@w3.org</A></TT>.
</P>

<P CLASS="validated">
<A HREF="http://validator.w3.org/check/referer"><IMG SRC="http://www.w3.org/Icons/valid-html40" ALT="Valid HTML 4.0!"></A>
<A HREF="http://jigsaw.w3.org/css-validator/check/referer"><IMG SRC="http://jigsaw.w3.org/css-validator/images/vcss" ALT="Valid CSS!"></A>
</P>

<TABLE cellpadding="5" cellspacing="0" border="0" style="border: solid 1px black;">
<TR bgcolor="white" valign="middle">
<TD style="width:50%; border: none 0px;">
<SMALL>The CSS1 Test Suite is a joint effort of the <A HREF="http://www.w3.org/">World Wide Web Consortium</A>,
and the <A HREF="http://www.example.com/">(your company/organization name here)</A>.</SMALL>
</TD>
<TD align="center" style="width:50%; border: none 0px; white-space: nowrap;">
<A HREF="http://www.w3.org/"><IMG SRC="W3C.gif" border="0" alt=" W3C " height="32" width="57"></A>
<A HREF="http://www.example.com/"><IMG SRC="logo.gif" border="0" alt="(your logo here)" ></A>
</TD>
</TR>
</TABLE></BODY></HTML>