The HTML4 Test Suite is based upon the W3C HTML 4.01 recommendation. This test suite is applicable to user agents (particularly visual user agents). The W3C Validator should be used for validation of document authoring. The test suite consists of the following:
This document explains how to use and create the tests and test harnesses for the HTML Test Suite. The HTML Test Suite is in the TNG format used by the CSS Test Suites. This document is derived from the CSS Test Suite Documentation. Concepts from the SVG Conformance Test Manual and the XML/SOAP Test Suite are also incorporated.
Sample text and hyperlinks from test suites are presented in this document with a thin grey border like this: test suite text. test suite hyperlink. Sample hyperlinks from test suites are not underlined to avoid misrepresenting them as actual hyperlinks in this document.
The cover page of a test suite contains a table of contents, structured parallel to the specification being tested. Often a test suite contains a Prologue which documents among other things requirements or recommendations for a specific system configuration, or suggestions for setting your system preferences in such a way as to minimize their interference with the tests themselves.
Get started with running the tests in a test suite by clicking on the first numbered link in the table of contents (skipping over the Prologue), or by clicking the get started here hyperlink that many test suites provide on their cover page.
On every following page, there will be a [Next] hyperlink that will take you linearly through each section, subsection and test of the test suite, finally bringing you back to the cover page at which point you have completed the test suite. On any of these pages you can always backtrack this linear order by following the [Previous] hyperlinks.
On every page that contains a test, you should see the test content immediately below
the abovementioned hyperlinks. Instead of the test itself, if you see a
Test hyperlink, then your browser
doesn't properly support the <OBJECT> element. No matter,
simply click the Test hyperlink
to navigate to the test itself. After viewing the test, use the Back functionality
in your web browser to return to the navigation page for that test and proceed to the next page
by clicking the [Next] hyperlink.
The test content is self-describing - that is, the test content contains a textual description of what the page should look like according to the test, e.g. 1. This first item of an ordered list should be numbered
The test suite format consists of several different kinds of pages. At the core are the test pages themselves that exercise the features of the technology. Each test page is referenced by a navigation page that also references the previous / next navigation pages, and its section page. The test suite starts with a cover page, which links directly to each of the section pages. Here is a visualization of the relatively flat conceptual hierarchical structure of the test suite:
The individual types of pages in the test suite format are described from the inside out.
The HTML test pages typically contain the test purpose and assertions tested stated in a <div class="TestPurpose"> and <div class="AssertionsTested"> block, e.g.
Verify that ordered and unordered lists are rendered. Tests for assertions 10.2-8, 10.2-9
Followed by a series of tests, each of which tests one item clearly. There may also be compound tests that follow the atomic tests.
Each test contains prose which describes the correct behavior.
The test pages use valid HTML 4.0 Strict or Transitional, which is the most compatible version of HTML for present day user agents. In the future, XHTML 1.0 Transitional which follows the XHTML 1.0 Appendix C guidelines may be more compatible across more user agents. XHTML Basic, again following the XHTML 1.0 Appendix C guidelines, may also be more compatible.
Finally, the test is named by the section.subsection number of the section being tested (with periods replace by underscores e.g. 1_1), a hyphen, a test type, a hyphen, a two digit number. E.g. 1_1-BF-01. This differs somewhat from the CSS and SVG Test file naming conventions. The underscore in the section name allows better tracking to the relevant section of the specification. The test type gives an indication of the type of test. Test types include:
The two digit number allows otherwise identical names to be used.
Test file names should be the same as the test title with the proper suffix (almost always .html) appended. E.g. 1_1-BF-01.html Supporting files for the test should be named with an alphabet character after the test name and before the .html suffix. E.g. 1_1-BF-01a.html Generic supporting files such as image files may be named differently. Note that the test file has no navigational content whatsoever. This allows the test file to be referenced from multiple different navigational files, and allows alteration of the navigation without having to update test files which would otherwise require implementers to retest test files every time test suite navigation was changed.
Each test page is contained within a navigation page. The navigation page is titled with the name of the test suite followed by the test name
<title>,e.g.
HTML Test Suite: 10_2-BF-01. The navigation page
starts with a heading documenting the test suite, followed by the test name and the section heading.
e.g. HTML Test Suite: Test 10_2-BF-01 Unordered lists.
A horizontal rule separates the heading from the following links:
[Previous] [Next] [Section] [Contents] [Specification]
The Previous and Next hyperlinks link to the previous and next navigation pages respectively. Previous of the first navigation page of a section links to that section page. Next of the last navigation page of a section links to the next section. And Next of the last navigation page in the suite links to the cover page.
The Section hyperlink links to the section page for all the navigation pages for this section of the specification being tested. The Contents hyperlink links to the cover page of the test suite. The Specification hyperlink links to the specific section of the specification being tested.
The navigation page transcludes the test page with an
<OBJECT> element, which references the test file using the
data attribute and explicitly sets the type
of the expected data to "text/html".
In order to encourage a user agent to use the entire width of the document,
and the remainder of the document, both the width and height
attributes are set to 100%.
The <OBJECT> element was
chosen because it is present in all "current" versions of HTML, from HTML 4.0 Transitional,
to XHTML Basic, to XHTML 1.1. This permits user agents which support any one of these
languages to easily utilize the navigational pages. In addition, the
<OBJECT> element provides an excellent immediate fall back mechanism
for user agents that either happen to not support <OBJECT>, or
that have difficulties transcluding HTML content.
Inside the <OBJECT> element is a simple
Test hyperlink which
directly links to the test file. Thus user agents that do not recognize the
<OBJECT> element and therefore ignore its markup,
show the simple hyperlink instead, thereby providing access to the test file.
The navigation pages also use valid HTML 4.0 Transitional, for the same reason as the test pages. Since the navigation pages are separated from the test pages, this could easily change in the future without having to change the test pages. Similarly, the test pages could be updated to use XHTML Basic for example without having to update the navigational pages.
Lastly, the navigation pages are named "sec" followed by the name of the test file being tested with the extension ".html". E.g. "sec1_1-BF-01.html". Note that the navigation file has no test content whatsoever. This allows the test file to be updated to fix problems in tests without having to update navigation files.
Note that some HTML test will not work within the navigation pages. In those instances a link to the actual test file will be displayed.
For each chapter in the specification, there is a corresponding section page which hyperlinks to the navigation pages of all the tests for that chapter. The section page format very much resembles the navigation page format.
The section page contains a flat list of hyperlinks to the specific navigation pages, in specification order. E.g. Test 6_15-BF-01.html
The list is demarcated on both top and bottom by horizontal rules, and a link bar with the following links:
[Previous] [Next] [Contents] [Specification]
The Previous hyperlink links to the last navigation page of the previous section. The Previous of the first section (typically a Prologue section) simply links to the test suite cover page. The Next hyperlink links to the first navigation page of this section.
The Contents hyperlink links to the cover page, and the Specification hyperlink links to the respective section in the specification.
Similar to the navigation pages, the section pages are also titled with a
heading documenting the name of the test suite, followed by the numerical
section of the specification. This heading is also used for the page
<title>,e.g.
HTML Test Suite: 10 Lists.
The section pages are also written in valid HTML 4.0 Transitional,
and named "section" followed by the top level section number and
the extension ".html". E.g. "section10.html".
A test suite's cover page begins with the W3C logo and a heading stating the
name of the test suite which is also used for the page
<title>.E.g.
HTML4 Test Suite
A short paragraph of text introduces the test suite, and documents its purpose, usage, and any caveats. Another paragraph minimally explains the structure of the test suite itself. Any additional test suite specific documentation should be placed in the Prologue. A subheading labels the Table of Contents which immediately follows the introductory text.
The table of contents is a simple list of numbered hyperlinks ( e.g.: 10 Lists ) to the section pages. Following the numbered section pages for the tests, there may be one or more lettered pages for test suite appendices, such as A. Acknowledgments, B. Test Suite Errata, and C. Related Resources.
A horizontal rule separates the Table of Contents from the cover page footer. The footer contains a mailto hyperlink for test suite feedback, the proper validation stamps (HTML, CSS), and recognition of the organizations responsible for the production of the test suite, along with their respective logos.
The authoring of a test suite for a particular specification begins with the identification of testable assertions in the specification grouped by section and subsection of the specification.
Try to make use of existing test pages, by reference if possible rather than duplicating. Some specifications, such as profiles, do not contain new features of their own. New versions of specifications often build upon features of older specifications. In both cases, it should be possible, and it is desirable, to reuse existing test cases.
Start by creating a new directory for your test suite, and copying the necessary support files (listed below). All test suite files for your test suite will go into this one directory.
Using the templates listed below, author navigation pages for each feature and their
respective testable assertions. If possible, author navigation pages which refer to tests from
previous test suites. Use a relative URL in the data attribute of the
<OBJECT> tag, and href of the anchor <A>,
to refer to test pages from previous test suites.
Replace the heading and document <title> with the name of the test suite,
the subsection number and title. Fix the link bar hyperlinks (Previous, Next, Section, Specification)
to point to the appropriate files.
For navigation pages which require new tests, write new test pages, also starting with the template
listed below. Be sure to update the document <title> to be the same as the
corresponding navigation page.
After completing all the navigation (and necessary test) pages for a section in the specification, write the section page (again, there is a template which can be used as a starting point). Once the section pages have been completed, construct the cover page and related support pages (Prologue, Acknowledgments, Test Suite Errata (hopefully fairly empty to begin with), and Related Resources). Templates aren't included for the related support pages, but they can be easily written starting from the respective pages in the CSS1 Test Suite.
Congratulations, you've authored a test suite.
These templates are derived from the latest version of the CSS1 Test Suite, with a small bit of cleanup, so there may be minor inconsequential differences. Test suite authors MUST use these templates to write new CSS Test Suites so that alternative versions (such as XML or XHTML versions) may be mechanically generated by a tool.
The template for a test page is shown below:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html401/strict.dtd"> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <title>test name e.g. 10_2-BF-01</title> </head> <body> <div class="TestPurpose">Verify some test purpose<div> <div class="AssertionsTested">Tests for assertions assertion_number<div> /* enter self-documenting test here */ </body> </html>
The template for a navigation page is shown below:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd"> <html><head><title>HTML Test Suite: n.n Subsection Title </title> <meta http-equiv="Content-Style-Type" content="text/css"> <link rel="stylesheet" type="text/css" media="screen" href="static.css"></head> <body> <div class="navigation"> <h2>HTML4 Test Suite: n.n Subsection Title</h2> <hr> [<a href="secnn.html">Previous</a>] [<a href="secnn.html">Next</a>] [<a href="sectionnn.html">Section</a>] [<a href="index.html">Contents</a>] [<a href="http://www.w3.org/TR/">Specification</a>]<br> </div> <object height="100%" width="100%" border="0" type="text/html" data="testnntemplate.html"><a class="navigation" href="testnntemplate.html" target="testwindow">Test</a></object> </body></html>
The template for a section page is shown below:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd"> <html><head><title>HTML Test Suite: n.0 Section Title</title> <meta http-equiv="Content-Style-Type" content="text/css"> <link rel="stylesheet" type="text/css" media="screen" href="static.css"> </head> <body class="navigation"> <h2>HTML4 Test Suite: n.0 Section Title</h2> <hr> [<a href="secnn.html">Previous</a>] [<a href="secnn.html">Next</a>] [<a href="index.html">Contents</a>] [<a href="http://www.w3.org/TR/">Specification</a>]<br> <hr> <p> In this section: </p> <ul> <li><a href="secTestName1.html">Test secTestName1.html</a></li> <li><a href="secTestName2.html">Test secTestName2.html</a></li> <li><a href="secTestName3.html">Test secTestName3.html</a></li> <li><a href="secTestName4.html">Test secTestName4.html</a></li> <li><a href="secTestName5.html">Test secTestName5.html</a></li> <li><a href="secTestName6.html">Test secTestName6.html</a></li> <li> <a href="secTestName7.html">Test secTestName7.html</a></li> </ul> <hr> [<a href="secnn.html">Previous</a>] [<a href="secnn.html">Next</a>] [<a href="index.html">Contents</a>] [<a href="http://www.w3.org/TR/">Specification</a>]<br> </body></html>
The template for the cover page is shown below:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><head><title>HTML Test Suite</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta http-equiv="Content-Style-Type" content="text/css">
<link rel="stylesheet" type="text/css" media="screen" href="static.css">
<link rel="alternate stylesheet" type="text/css" href="W3C-rsrc-test.css" media="screen" title="Experimental W3C Styles">
<style type="text/css">
h1 {border: solid 1px black; border-left: 3px solid black; background-color: #FFFFFF;}
h4 {border-bottom: 1px solid black; margin-left: 2em; margin-right: 5em; font-variant: small-caps; margin-bottom: 0.5em;}
ul {list-style-type: none;}
.validated { text-align:left; font-size: smaller }
</style>
</head>
<body class="navigation">
<h1><a href="http://www.w3.org/"><img border="0" SRC="http://www.w3.org/Icons/WWW/w3c_home" alt="W3C" width="72" height="48" align="middle"></a>
HTML Test Suite</h1>
<p>
The HTML4 Test Suite is provided as a way for vendors and page authors to test
their browser's conformance to the HTML4 specification.
</p>
<p>
For each feature, there is at least one page which tests the feature
in various ways, using HTML 4.0 markup. The test pages are broken out into
a number of sections which reflect the structure of the HTML4 specification itself,
in addition to a Prologue and a few appendicies.
</p>
<h4>Table of Contents</h4>
<ul>
<li style="font-weight: bold;">-- <a href="prologue.html">Prologue</a></li>
</ul>
<ul>
<li>1.0 <a href="sec10.html">Section 1</a></li>
<li>2.0 <a href="sec20.html">Section 2</a></li>
<li>3.0 <a href="sec30.html">Section 3</a></li>
<li>4.0 <a href="sec40.html">Section 4</a></li>
<li>5.0 <a href="sec50.html">Section 5</a></li>
<li>6.0 <a href="sec60.html">Section 6</a></li>
<li>7.0 <a href="sec70.html">Section 7</a></li>
</ul>
<ul style="margin-top: 0;">
<li> A. <a href="tsack.html">Acknowledgments</a></li>
<li> B. <a href="tserr.html">Test Suite Errata</a></li>
<li> C. <a href="tsref.html">Related Resources</a></li>
</ul>
<hr>
<p CLASS="validated">
<a href="http://validator.w3.org/check/referer"><img src="http://www.w3.org/Icons/valid-html40" ALT="Valid HTML 4.0!"></a>
<a href="http://jigsaw.w3.org/css-validator/check/referer"><img src="http://jigsaw.w3.org/css-validator/images/vcss" ALT="Valid CSS!"></a>
</p>
<table cellpadding="5" cellspacing="0" border="0" style="border: solid 1px black;">
<tr bgcolor="white" valign="middle">
<td style="width:50%; border: none 0px;">
<small>The HTML Test Suite is a joint effort of the <a href="http://www.w3.org/">World Wide Web Consortium</a>,
and the <a href="http://www.example.com/">(your company/organization name here)</a>.</small>
</td>
<td align="center" style="width:50%; border: none 0px; white-space: nowrap;">
<a href="http://www.w3.org/"><img src="W3C.gif" border="0" alt=" W3C " height="32" width="57"></a>
<a href="http://www.example.com/"><img src="logo.gif" border="0" alt="(your logo here)" ></a>
</td>
</tr>
</table></body></html>
Some basic principles have been learned during the construction of previous test suites, applicable to both graphics suites and others:
All tests must test what they purport to test and all files in the test suite, whether harness or tests themselves must at a minimum pass the W3C Validator .
The tests should be self-documenting. Test cases should describe in prose, in the test cases themselves, what is expected. Each test case should at a minimum describe correct behavior. Error handling tests should at a minimum describe incorrect, or possibly unexpected behavior.
Simplicity is always a good design principle. Failure of a test should be caused by the assertion being tested and not because of numerous variables introduced by a complex test. A simple format also makes it easier to author and analyze the test.
A test suite should contain atomic (meaning, testing a single feature at a time) tests. Some features have little or no effect by themselves, and therefore may require simple compound tests at a minimum.
Tests must be based upon, and traceable to, testable statements in the specification being tested.The HTML test suite will be based on the testable assertions document.
Without sacrificing the principle of atomic testing, the number of test instances (files) can be reduced by having a single instance combine multiple related test purposes.
Ease of authoring to implies that the test suite format must be easily hand authorable, that is, easily authorable with only the use of a simple text editor without a dependency upon any particular tools or any particular platform or device.
Test suites should begin with simpler, easier test cases, and proceed to more complex, more challenging tests. This makes sense both from a test case authoring perspective, since simpler cases are easier to author, and from a technology implementer's perspective, since simpler cases are easier to implement.
The detailed tests should try to methodically vary and test all values, plus boundary conditions and extreme conditions, of each parameter, attribute, or property.
The creation of a test suite is an iterative process. The creation of a conformance test suite for HTML is daunting. Start with something, then make it better. A simple incomplete test suite is far far better than none at all.
Corollary: a simple test suite within a few months is more useful, and thus preferred to, a more complex, more fully featured test suite a year or more from now.
The first version of the test suite will be imperfect and incomplete. Iterate to achieve perfection and completeness.
When writing a test suite for a particular technology (e.g. HTML), it is often necessary to utilize other technologies (e.g. CSS, ECMAScript, DOM etc.) in the construction of that test suite. Test suite authors must be careful while testing one technology, to avoid testing other technologies. Those other technologies should be authored to be maximally compatible, in a way that is most forgiving to the various user agents which will be using the test suite. Consequently, HTML test suite files should:
Test suites should attempt to reuse test cases from previous test suites.
Finally, as with any set of principles and requirements, there are some non-requirements. These non-requirements are exactly that - they are features not required of a test suite. They may be desirable, or even included - yet they are not necessary to produce a test suite.
The SVG Test Manual documents additional Principles Applicable to Test Suite Content . Future test suite authors should consult these principles as well.
This HTML Test Suite Documentation is mostly an edited version of the CSS Test Suite Documentation and this document would have been very difficult to create without it. The HTML Test Suite is derived from the work of many people at the W3C and other organizations and companies.
Feedback regarding the HTML4 Test Suite should be sent to www-html-testsuite@w3.org.
| The HTML4 Test Suite is an effort of the World Wide Web Consortium. |
|