SVG2/Testing Requirements
Overview
SVG requires a testing suite that can be used by renderers that may or may not have support for:
- JavaScript/EcmaScript
- SMIL
- CSS (and CSS3)
The tests should be able to link to the relevant places in the specifications (SVG 2, CSS3, etc.). The specifications should include links to the relevant tests.
There does exist a very old page on testing [[1]].
The CSS group has put together a very nice set of Wiki pages on testing including:
Current Situation
SVG test suites are available at [[2]].
The SVG 1.1 test suite must be manually run, visually looking at each test SVG. This includes tests of the DOM that merely set the fill of a rectangle to green or red to indicate that a test passes or fails.
By convention, a test that shows red indicates a failed test.
Tests include data in the private namespace 'd'. The data includes a test description, instructions for running (if interaction is required), and the pass criteria.
There is only limited testing of internal CSS style sheets.
Types of Tests
Reference tests
SVG vs SVG Ref
Test SVG can be compared pixel by pixel with a reference SVG. These types of tests could be automated.
- Tests of CSS styling vs. presentation attributes.
- Tests of illegal syntax.
SVG vs PNG Ref
Test SVG cannot be compared exactly with a reference SVG.
This includes tests where renderers can produce different results due to (allowed) rounding errors, etc. (recall the SVG specification allows a one pixel error).
ED: Note that another complication might be that errors can accumulate so that the one pixel error margin may not enough by itself.
- Tests with filters.
- Tests with gradients.
Techniques
Inkscape at one point had a regular testing program.[[3]] The program relied on comparing a newly generated image against known good and bad images. By default only a binary diff was done. There was an option to use an external program to do the comparison. perceptialdiff was the suggested program.[[4]]
Non-static Tests
Tests of Animation
- Tests of SMIL
- Tests of CSS Animations and Transitions.
Question: Can the results of animation via SMIL be compared to CSS Animation via a ref test or via JavaScript (querying the final positions)?
Tests of DOM
- Tests of JavaScript/EcmaScript
Techniques
The CSS Test Frame
Links
Links provided by Peter Linss during the March 1st, 2012 SVG WG Conference Call.
- Testing Harness Wiki (Test Suite Build System).
- Mercurial Repository for CSS Testing.
- Tools Directory in Repository.
- Test Library.
- Test Format.
- Shepherd (Web interface for managing tests).
- Test Framework.
Adapting SVG Test Template for CSS Test Frame
SVG 1.1 Structure
<svg id="svg-root" width="100%" height="100%" viewBox="0 0 480 360" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"> <!-- Copyright ... -> <d:SVGTestCase xmlns:d="http://www.w3.org/2000/02/svg/testsuite/description/" template-version="1.4" reviewer="DAS" author="CM" status="accepted" version="$Revision: 1.11 $" testname="$RCSfile: animate-dom-01-f.svg,v $"> <d:testDescription xmlns="http://www.w3.org/1999/xhtml" href="http://www.w3.org/TR/SVG11/animate.html#InterfaceSVGAnimationElement"> <p>Description ...</p> </d:testDescription> <d:operatorScript xmlns="http://www.w3.org/1999/xhtml"> <p>Operator instructions...</p> </d:operatorScript> <d:passCriteria xmlns="http://www.w3.org/1999/xhtml"> <p>Pass criteria...</p> </d:passCriteria> </d:SVGTestCase> <title id="test-title">$RCSfile: animate-dom-01-f.svg,v $</title> <defs>...</defs> <g id="test-body-content" ...>Content</g> <g>Revision ...</g> <rect id="test-frame"... /> <g id="draft-watermark...>...</g> </svg>
Suggested new structure (see CSS structure)
<svg id="svg-root" width="100%" height="100%" viewBox="0 0 480 360" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"> <!-- Copyright ... -> <d:SVGTestCase xmlns:d="New URL" xmlns="http://www.w3.org/1999/xhtml"> <head> <title>SVG Test: DESCRIPTION OF TEST</title> <link rel="author" title="NAME_OF_AUTHOR" href="mailto:EMAIL OR http://CONTACT_PAGE"/> <link rel="reviewer" title="NAME_OF_REVIEWER" href="mailto:EMAIL OR http://CONTACT_PAGE" /> <!-- YYYY-MM-DD --> <link rel="help" href="RELEVANT_SPEC_SECTION"/> <link rel="match" href="RELATIVE_PATH_TO_REFERENCE_FILE" /> <link rel="mismatch" href="RELATIVE_PATH_TO_REFERENCE_FILE" /> <meta name="flags" content="TOKENS" /> <meta name="assert" content="TEST ASSERTION"/> <!-- CSS format inserts the CSS that is being tested here. --> </head> </d:SVGTestCase> <style id="test-style"> Style that is being tested (if needed). </style> <defs>...</defs> <g id="test-body-content" ...>Content</g> <g>Revision ...</g> <rect id="test-frame"... /> </svg>
Notes:
- The suggested format has been designed to minimize the differences between SVG and CSS testing to facilitate adapting the CSS test framework to our needs.
- Multiple links with the same "rel" are allowed (e.g. one can have multiple authors).
- The CSS Test Format page gives clear instructions on file names. Maybe we should adopt the same naming conventions.
- The CSS framework provides a set of shared resources in the support directory.
Issues:
- We will need a new NS URL for SVGTestCase.
- Not needed with a template adopted below.
- Do we need to keep the copyright section? The CSS tests do not include a copyright.
- Include via link.
- The CSS that the CSS test is testing is included in the <head> section where it can easily be found. This idea would be nice for SVG tests that test CSS so the CSS can be displayed along with the test (example). However we cannot put our CSS in the same place. We need a style section outside the SVGTestCase block with an appropriate 'id'.
- Do we need to keep the revision displayed in the SVG or can we rely on Mecurial?
- We'll rely on Mecurial. Revision wasn't reliably updated anyway.
- Do we need the "Not Approved" warning box? The adopted test directory structure keeps the approved tests separate from unapproved tests.
- No, no longer needed with separate directories.
- CSS has a meta section for "flags" to indicate what features are needed by the renderer for the test. Two flags relevant to SVG are: "svg" and "namespaces". Since SVG is SVG, do we need to include these flags or can we just assume that any SVG renderer will support SVG and namespaces?
- The flags allow SVG tests to be used by test suites for other specs. Also not all SVG test, for example tests of SVG in HTML, can be run by all SVG renderers. We should include them.
- Do we need a frame?
- No, one can use a border to offset the SVG when including the SVG in an HTML test harness.
Structure after 8 March 2012 telcon
Most significant change is to remove <html> wrapper as this will cause problems for SVG inside HTML. Note <meta> causes HTML5 parser to auotclose the SVG, reverting to HTML, while <link> does not.
<svg id="svg-root" width="100%" height="100%" viewBox="0 0 480 360" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"> <title>SVG Test: DESCRIPTION OF TEST</title> <link rel="copyright" href="@@link to copyright page to be added@@"/> <link rel="license" href="http://www.w3.org/Consortium/Legal/2008/03-bsd-license.html"/> <link rel="author" title="NAME_OF_AUTHOR" href="mailto:EMAIL OR http://CONTACT_PAGE"/> <link rel="reviewer" title="NAME_OF_REVIEWER" href="mailto:EMAIL OR http://CONTACT_PAGE" /> <!-- YYYY-MM-DD --> <link rel="help" href="RELEVANT_SPEC_SECTION"/> <link rel="match" href="RELATIVE_PATH_TO_REFERENCE_FILE" /> <link rel="mismatch" href="RELATIVE_PATH_TO_REFERENCE_FILE" /> <metadata class="flags">TOKENS" </metadata> <desc class="assert">TEST ASSERTION</desc> <!-- CSS format inserts the CSS that is being tested here. --> <style id="test-style"> Style that is being tested (if needed). </style> <defs>...</defs> <g id="test-body-content" ...>Content</g> </svg>
Issues:
- Peter would Linss like a wrapper around all the metadata.
- Chris suggests wrapping in <g id="metadata">. See: email thread.
Revised structure after comment by Peter Linss
<svg id="svg-root" width="100%" height="100%" viewBox="0 0 480 360" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:html="http://www.w3.org/1999/xhtml"> <g id="testmeta"> <title>SVG Test: DESCRIPTION OF TEST</title> <html:link rel="copyright" href="http://www.w3.org/Graphics/SVG/Test/Copyright"/> <html:link rel="license" href="http://www.w3.org/Consortium/Legal/2008/03-bsd-license.html"/> <html:link rel="author" title="NAME_OF_AUTHOR" href="mailto:EMAIL OR http://CONTACT_PAGE"/> <html:link rel="reviewer" title="NAME_OF_REVIEWER" href="mailto:EMAIL OR http://CONTACT_PAGE" /> <!-- YYYY-MM-DD --> <html:link rel="help" href="RELEVANT_SPEC_SECTION"/> <html:link rel="match" href="RELATIVE_PATH_TO_REFERENCE_FILE" /> <html:link rel="mismatch" href="RELATIVE_PATH_TO_REFERENCE_FILE" /> <metadata class="flags"><text>SPACE-SEPARATED LIST OF TOKENS</text></metadata> <desc class="assert">TEST ASSERTION</desc> </g> <style id="test-style"> Style that is being tested (if needed). </style> <defs>...</defs> <g id="test-body-content" ...>Content</g> </svg>
Notes:
- xmlns:xlink="http://www.w3.org/1999/xlink" to be removed after SVG2 adopted.
- Several small formatting changes. Removed unneeded comment about CSS format.
Revised structure after 14 June 2012 teleconference
We decided to have two templates: one for automated tests and one for visual tests.
For automated tests:
Nominally, a test should turn a single centered red square to green.
<svg id="svg-root" width="100%" height="100%" viewBox="0 0 480 360" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:html="http://www.w3.org/1999/xhtml"> <g id="testmeta"> <title>SVG Test: DESCRIPTION OF TEST</title> <html:link rel="copyright" href="http://www.w3.org/Graphics/SVG/Test/Copyright"/> <html:link rel="license" href="http://www.w3.org/Consortium/Legal/2008/03-bsd-license.html"/> <html:link rel="author" title="NAME_OF_AUTHOR" href="mailto:EMAIL OR http://CONTACT_PAGE"/> <html:link rel="reviewer" title="NAME_OF_REVIEWER" href="mailto:EMAIL OR http://CONTACT_PAGE" /> <!-- YYYY-MM-DD --> <html:link rel="help" href="RELEVANT_SPEC_SECTION"/> <html:link rel="match" href="RELATIVE_PATH_TO_REFERENCE_FILE" /> <html:link rel="mismatch" href="RELATIVE_PATH_TO_REFERENCE_FILE" /> <metadata class="flags">TOKENS</metadata> <desc class="assert">TEST ASSERTION</desc> </g> <style id="test-style" type="text/css"> /* Style that is being tested (if needed). */ rect { fill: green; } </style> <defs></defs> <g id="test-body-content"> <!-- Your test should turn this rectangle green. --> <rect x="190" y="130" width="100" height="100" fill="red"/> </g> </svg>
For visual tests:
For tests that cannot be automated or are designed for visual scanning. A default font is specified so that labels (if used) are of known size.
<svg id="svg-root" width="100%" height="100%" viewBox="0 0 480 360" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:html="http://www.w3.org/1999/xhtml"> <g id="testmeta"> <title>SVG Test: DESCRIPTION OF TEST</title> <html:link rel="copyright" href="http://www.w3.org/Graphics/SVG/Test/Copyright"/> <html:link rel="license" href="http://www.w3.org/Consortium/Legal/2008/03-bsd-license.html"/> <html:link rel="author" title="NAME_OF_AUTHOR" href="mailto:EMAIL OR http://CONTACT_PAGE"/> <html:link rel="reviewer" title="NAME_OF_REVIEWER" href="mailto:EMAIL OR http://CONTACT_PAGE" /> <!-- YYYY-MM-DD --> <html:link rel="help" href="RELEVANT_SPEC_SECTION"/> <html:link rel="match" href="RELATIVE_PATH_TO_REFERENCE_FILE" /> <html:link rel="mismatch" href="RELATIVE_PATH_TO_REFERENCE_FILE" /> <metadata class="flags">TOKENS</metadata> <desc class="assert">TEST ASSERTION</desc> </g> <style id="test-font" type="text/css"> /* Standard Font (if needed). */ @font-face { font-family: FreeSans; src: url("fonts/FreeSans.woff") format("woff"); } text { font-family: FreeSans, sans-serif } </style> <style id="test-style" type="text/css"> /* Style that is being tested (if needed). */ text { fill: green; } </style> <defs></defs> <text id="title" x="240" y="50" style="fill:black; font-size:24px; text-anchor:middle;">SVG Test Template</text> <g id="test-body-content" font-size="16"> <text x="50" y="100" fill="red">This text should be green.</text> </g> </svg>
Added font-size="16" to test-body-content per Chris's suggestion 16 Jun 2012.