Revision: 3.00
Date: October 1, 2001
By: Lofton Henderson
|
Rev |
Date |
Description of Change |
|---|---|---|
|
1.00 |
2000-02-08 |
Standing project document, 1st WG release. Based on Cupertino docs, later conformance decisions, additional rsch, and core info from path doc. |
| 1.01 | 2000-04-27 | Releasable HTML version. |
| 2.00 | 2000-05-18 | Incorporate cumulative experience from BE suite construction. |
| 2.01 | 2000-08-20 | Improvements to "Test Review Guidelines"; also to details on writing tests. |
| 2.02 | 2000-09-27 | Comment about color mgmt artifacts; XML navigation links and harnesses. |
| 2.03 | 2000-11-14 | CVS online; templates & harnesses; 2nd generation stuff; serial numbers. |
| 2.04 | 2001-01-27 | Further edits to reflect completed BE suite, 2nd public release. |
| 3.00 | 2001-10-01 | Editorial changes to align with final REC SVG test suite public release. |
This document is intended as a permanent reference document and user manual for designers and contributers to the SVG Conformance Test Suite. It is the repository for currently agreed methods, templates, procedures, and techniques. This is the third public release of the document, and correlates with the public release of the completed BE test suite, subsequent to REC SVG publication. The first release (mid-2000) corresponded to an earlier public release of a subset (slightly over half) of the full BE suite. The second public release (early 2001) was a full BE suite, and corresponded to public CR draft of SVG.
Some parts of this document remain incomplete. In particular, the synopses of technical content of other conformance projects, in the chapter "Related Conformance Work", have not yet been detailed. In addition, some procedures and methods documented in the first public release have now been superseded or deprecated.
The ultimate goal is a comprehensive and detailed conformance test suite for SVG 1.0.
Amongst the multiple purposes which such a suite can serve, we identify the most important as: a publicly available suite to help implementation builders achieve interoperability.
There are at least three areas in which conformance testing is applicable:
This project's scope is limited to the third -- a conformance test suite for interpreters and viewers.
At the Cupertino SVG-WG meeting (11/99), the question of what sort of suite we are building was discussed and resolved. The options included:
The SVG WG decided at Cupertino: we are building #3, a publicly available suite for such uses as informal conformance analysis and developer self-testing.
Presently, there are no plans for a SVG "certification" service, therefore there is no need for #4 -- W3C doesn't currently carry out certification testing, nor is any currently proposed by other entities.
What is the difference? In what ways would the suite differ depending on its purpose?
Some potential differences include:
While the formality and rigor of a certification suite might not be needed, the SVG conformance suite will (eventually) embody "traceability" (see below) -- what specification in the standard justifies a given test?
The SVG WG agreed to two milestones:
At the time of release of this document, the BE suite is complete and the SVG 1.0 REC has been published. Some initial work has been done on DT tests, but it has not yet been integrated into the test suite structure, nor in this current (REC) test suite public release.
For those interested in a quick user guide for test construction, you can skip directly to "How to Write Tests". The rest of this document provides background, detailed explanation, and motivation for the methods used.
The material in Section 2, especially the brief synopsis of the nature and content of each existing suite, remains incomplete.
Section 3 is substantially complete.
The material in section 4 is complete for Static Rendering, but Dynamic has not been addressed. A couple of topics like overall test-suite linking structure are still incomplete (e.g., when substantial DT tests are generated and added) .
Section 5 -- How to Write Tests -- is substantially complete for Static Rendering "how-to", including incorporation of experience from a couple years' work on the BE test suite.
Section 6, Test Review Guidelines, is complete.
Section 7, Glossary, contains a useful subset of key terms used in this document.
There is now a substantial body of test suite experience and material, for several different standards:
These suites and the experiences of building them are useful to the SVG conformance effort, and to contributors of test materials, in a number of ways:
The "level of effort" data should be particularly interesting to the SVG group.
Note. Since the initial release of this document (4/2000) to this release (10/2001), there has also been substantial test suite work on XSLT, XSL-FO, DOM Java binding, etc. These could be researched and include in a future version of this document section.
See [5] and [3].
The applicability of CGM test suite experience to at least the static rendering subset of SVG is obvious.
CGM and SVG differ in other ways:
Note. As of this date (10/2001), work is substantially complete and a public release is pending for a test suite for REC WebCGM.
See [9].
[Analysis of potential applicability VRML suite and methods will be written for future document release.]
See [7].
For application of visual properties (graphical attributes) SVG borrows heavily on CSS2. The syntax for all such properties (plus some others) is patterned on CSS, and a number of CSS2 properties (esp. font properties) are adopted directly by SVG. The full font selection and matching machinery of CSS2 is required in conforming SVG processors (interpreters and viewers).
W3C has made a test suite for CSS1. The methods of the CSS1 test suite should clearly be applicable to some aspects of the SVG suite. Some actual CSS test materials might be (almost) directly usable.
See [6].
We'll have to test the SVG DOM. NIST's XML DOM suite for Javascript binding is released, for both XML and HTML. The Java binding is in progress. Methods and techniques should be applicable, and maybe some materials can be borrowed with minimal modification.
See [8].
A conforming SVG interpreter (hence also a conforming SVG viewer) "must be able to parse and process any XML constructs defined in [XML10] and [XML-NS]." A conforming SVG viewer therefore incorporates XML-suite conformance, by reference.
Size: about 200 simple, atomic tests.
Level of effort: difficult to determine, but probably about 1 - 1.5 FTE, external contractor plus NIST staff.
Size: 270 new tests, (70 new plus extensive redesign and revision of existing 200+ V1 tests).
Level of effort: difficult to determine, but probably about 1 - 2 FTE, external contractor plus NIST staff.
Size: Estimated about 1,000 tests.
Level of effort: 3 people full time for about 2 years at NIST. There was a steep learning curve. NIST released the first tests, after 3 months. They released tests as soon as a node was completed.
Size of 1st release: 1,000 XML tests -- DTD+4000 lines of XML code; 400 lines of XSL.
Level of effort: 1.5 FTE -- 2 people for approximately 9 months. One person designed the test harness and some of the tests and the other designed some tests and spent lots of time validating and filling in the holes from what other people contributed.
Note. Current release (10/2001) is about 2,000 tests. Level of effort has surely increased significantly, by an amount tbd.
Size of 1st release (Ecmascript with XML) -- 800 tests, 30,000 lines of code (this is only the Fundamental and Extended tests).
Level of Effort: 1.5 FTE -- One person half time for 9 months, who did the test harness (following much of what was done for VRML and XML); plus another full time for about 9 months; plus a third person about 4 months full time.
Unknown. To be researched for inclusion in future version of this document.
The CGM suite ([5]) consists of 269 test cases, each of which has three components:
There is no interactive harness or driver, and hence there are no navigation buttons to assist navigation through the suite. The operator has to invoke the viewer, access the Reference Picture, and access the Operator Script.
[Synopsis of content and structure of the VRML suite will be written for future document release.]
[Synopsis of content and structure of the XML suite will be written for future document release.]
[Synopsis of content and structure of the DOM suite will be written for future document release.]
[Javascript, XML and HTML finished. Java version being built now. NIST did the Javascript XML, then Javascript HTML reusing some the data file (which is a big HTML or XML document on which the DOM tests work). And the re-used the Test Assertions and XML document for the Java DOM-XML tests.]
[Synopsis of content and structure of the CSS1 suite will be written for future document release.]
The following basic process is applied for construction of most of the test suites referenced above -- CGM, VRML, XML, DOM, at least. In overview:
In practice, these steps need not be overly formal. In the case of a certification suite, formality is important. For a conformance suite, it is less so. In any case traceability (see below) is required.
Therefore, explicitly or implicitly, these steps are carried out -- the document is read exhaustively and decisions are made about what to test about each functionality, and how to realize these decisions in a set of test cases.
Section 4.2 of reference [9] contains an interesting discussion of TRs (which it calls SRs) and TCs -- the step of generating TPs is implicit in this reference, not explicitly treated as a formal step.
Some basic principles have been learned during the construction of previous test suites, applicable to both graphics suites and others:
The SVG specification divides fairly cleanly into semi-independent functional modules. Test materials will be developed and released progressively, subject to the constraint that we have agreed to make an entire breadth-first, basic effectivity (BE) test suite release first (now finished), and a drill-down (DT) release subsequent to that.
A major natural division in the specification is:
Static rendering got first priority for development and release of BE tests, although work on dynamic proceeded in parallel, and followed closely on the static tests. Prioritization (if any) for future DT work is tbd.
Functionality will be ordered in the suite, for purposes of execution and navigation through the suite, from most basic to most complex -- implementations should encounter the simplest and most basic tests first, before being subjected to progressively more complex and advanced functionality. This mirrors the organization of the REC SVG document.
Given our intent to make progressive releases of test suite modules, it made sense to generally follow this ordering for the building of the materials, at least for the completion of the DT and ER tests.
The SVG WG agreed at Cupertino (11/1999) to divide up the functionality by chapter. For static rendering, the following chapters are addressed in BE testing:
Building and executing tests in chapter order does not appear to always lead to a basic-to-complex ordering.
From most basic (or fundamental -- basic does not necessarily mean simple), to most advanced, a rough functional ordering might be:
The issue of test suite ordering and organization was resolved for the completed BE suite -- chapter order. The issue is still open for future DT work.
The decision was implicitly made for the dynamic parts of the BE suite -- chapter order, first priority assigned equally to the tests in a breadth first set touching on each of the dynamic functionalities.
Each Test Case (TC) in the static rendering module contains three principle components:
The operator script comprises a few sentences describing what is being tested, what the results should be, verdict criteria for pass/fail, allowable deviations from the reference image, etc.
#1, #2, and #3 are file instances, one for each Test Case. #3 includes navigation information, and is the source for generating the (HTML) navigation page.
Note. In the earliest test suites, for CGM, the Operator Script was a rigid and rote checklist for use by (non-expert) certification testing technicians, to score each test. It has evolved in more recent conformance work, to being more informative about the test's purpose and what to look for. It also can (and should) function to improve the accessibility of the test suite.
Details and examples of writing an Operator Script are given in the next chapter.
Other supporting material are generated for each test case:
Note. The traceability links were postponed until the SVG spec stabilized. Now that REC has been published, this should be added to the test suite..
See below, section [4.4.2] and [4.4.4], for futher details about the test harnesses.
Most of the SR materials are applicable to most dynamic tests. However, there may be cases (e.g., some DOM) which do not have graphical output, and there will be some which could (but need not necessarily) have animated graphical "reference images".
For the BE suite, the dynamic materials are the same as for static tests. This material may be further refined as more of the dynamic functionalities' tests are developed (looking forward to DT development).
Four generic test categories have been decided. These are equally applicable to static rendering and dynamic test modules:
Note. As of the date of this document (10/2001), only BE and DT tests have gotten any implementation attention.
For BE tests, an attentive reading of the applicable spec sections is required, but an exhaustiveTR enumeration is not. The generic BE test purpose is: correct basic implementation, including major variations within the functional area.
For DT and ER tests, an exhaustive TR extraction from the SVG spec is necessarily a part of the process, and test purposes are be derived from the TRs (see next chapter).
Following is a list of generic test purposes for DT tests (ER also?):
The Generic Test Purposes provide a high-level checklist for the sorts of test cases which should result from the analysis and test design of a functional area. If any major categories are not represented, it may indicate that some implicit or explicit requirements have been missed.
These requirements are agreed, at least for the static rendering module:
The VRML suite, as well as the CSS, XML, and DOM suites, employ an interactive test harness (HTML page), which:
Unlike the VRML suite, which presents the test rendering and the reference image side-by-side in one window, the SVG WG initially decided on a two-window approach, for standard release harnesses:
So in any case, a browser would have to be available for convenient viewing of all of the materials, but it is not necessary that a viewer-under-test be a browser plug-in.
Since the initial determination, additional harnesses have been added to the public release:
A strong naming convention for the materials is useful, both with the management of the test suite repository, as well as with requirement #2 above.
Test names aare brief but informative. The name design is: chapter-focus-type-num. 'Type-num' is a concatenation of the test type -- BE, DT, ER, or DM -- and its ordinal in the sequence of such tests -- 01, 02, ...
Examples: path-lines-BE-01, shapes-rect-DT-04, styling-fontProp-DM-02.
The primary test harness (static rendering, at least), is an HTML page which identifies the test, invokes the PNG reference image, and presents the operator script. The plug-in variant also presents the SVG beside the PNG.
Navigation buttons are provided to go back to a table of contents (and maybe an index), to navigate laterally through BE tests, and to drill down to "child" tests (from the BE level to the DT, ER, and DM tests), to go back up to "parent" tests from the lower levels.
Per an early WG decision (11/1999) and subsequent discussions, the principal HTML harness presented the operator script and the PNG reference image, but did not assume a browser-invokable SVG viewer -- the test administrator had to get the SVG image into another window, or onto a printer, or whatever was appropriate. There was a companion SVG-only harness to assist this (with exactly parallel navigation capabilities to the PNG-plus-Operator Script harness.)
With the current method for producing harness(es) -- XSLT stylesheet applied to instances of a simple XML grammar which describes each test case -- it is not difficult to produce multiple harness versions, including variants such as PNG plus rendered SVG plus operator script, for browser-plugin SVG viewers. These have now been included in the public release.
The first generation design of simple XML grammar for describing tests, and the XSLT stylesheet for producing the HTML page, have been released. A "manual" SVG template has been released as well -- see next chapter for details.
Work was completed on a "second generation" of harness and template tools:
This evolved into a test suite editor's tool, for performing major upgrades that involve modifying the standard template in a uniform manner. Because so many exceptions arose, which had to get manual or special processing, this line of tools has been largely abandoned in favor of simpler generic methods -- 'sed' scripts and the like.
Originally planned for the second generation tools:
After some working group experience with the BE suite and processes, it was decided that this sort of "make" overhead was undesirable.
Each test , at least for the static rendering module, is put into a standard template. As just discussed, this was originally a manual process -- the test writer puts the test body content into the template, and sets the routine information items in the template. The second generation tools mentioned in the previous section relieved this requirement somewhat, as long as the content was written in the appropriate coordinate range, etc.
Features of the standard template include:
#1, #3, #5, and #7 are the most critical. The template-generating tools do carry these parts forward, and generate the other parts anew at each upgrade run. An initial TC submission which has these parts can be input to the tools, to generate a fully conformant SVG test case instance (the latter is a management process for the "test suite editor", whoever that is).
The serial number is a method for ensuring that the PNG reference image, the SVG instance, and the SVG rendering are all current, all agree. To be useful, it must unfailingly increment whenever any change is made to the SVG file, in fact whenever the SVG file is saved. Originally this was a manual process. It has been replaced with the Revision keyword of CVS (the suite is now maintained in a CVS repository).
The overall linking structure is: TOC, BE layer throughout the suite (next/previous), DT drill down from BE (child), BE pop up from DT (parent), etc. Some details are to be worked out. Does each BE point down to a different DT "stack"? Or do all BEs in a chapter or chapter-focus point down to the first DT in that area? The WG tentatively decided the latter (the structure of the suite is not likely to be regular enough to make the former widely practical).
An index of all files would be a useful addition, in some future release of the test suite.
Processes and procedures still need to be designed, and a test suite editor assigned, for:
Previously, the test suite editor was the repository, and all of these items were ensured by the editor. The repository is now in CVS. An introduction to it can be found in the document "CVS Repository for SVG Test Suite" (presently only accessible to the WG). All WG members now interact directly with the CVS repository. Experience has shown that there are varying degrees of completeness, correctness, and integrity in the results with all-WG direct access.
Once a test case was submitted, it is "owned" by the repository. All WG and public releases are from the repository, and all maintenance changes to test cases must be applied to the latest repository version.
If the second generation tools (for automatic SVG template generation) had been made operational, adherence of test cases to the exact formatting (as opposed to functional) details of the template conventions could have been somewhat automated. However test contributors would have still had to adher to certain minimum requirements in structuring the initial contribution (see previous section, plus later sections about using the templates). And, as indicated earlier, there were too many exception cases in which the automated procedures couldn't be applied.
The examples which already exist in the SVG Specification have proved to be an excellent basis for material for some of the "Basic Effectivity" tests -- simple tests which minimally illustrate an SVG functionality.
Several of the SVG WG members have in-house development efforts. Materials ranging from basic path and shape graphics, to filter effects tests, to DOM and animation functionality are known to exist. Though some adaptation and integration into the SVG Test Suite framework is required, these existing QA materials have already proved to be a valuable resource for BE suite completion.
To be done: inventory what is available within the WG (for further DT development -- BE is done).
Note. The OASIS XSLT/Xpath conformance committee has pioneered and is now (10/2001) ready for public release of tools and procedures for accepting and integrating collections of externally authored test cases. These should be investigated for any future DT test suite construction.
For graphical output functionality, there is substantial commonality between CGM and SVG (see comparison table in [12]). The CGM Test Suite (for ATA, release 3.0, see [5]) has 269 tests, conforming to the test suite principles articulated above.
There are two interesting possibilities to leverage this CGM design and implementation work:
Note that NIST-certified CGM viewers exist, as well as certified printer drivers and certified rasterizers.
Note. This potential source has never been exploited, but could be potentially useful for generating some DT tests.
Contributions from outside of the WG may some day be solicited. Miniminal processing for contributions should include:
There is only one way to assure comprehensive coverage of Test Requirements: build new tests, carefully designed and targeted at specific TR(s).
There are two ways to approach this:
Experience with the first has been: the output drivers usually don't have precise enough control of the individual pattern of elements, and manual touchup is almost always required.
From CGM experience, an interesting source of Test Purposes, and possibly even test materials, are instances which have:
These indicate trouble areas, where implementers are likely to misinterpret or incorrectly implement the specification.
Beyond static rendering module, it should be possible to leverage methodology, or tests, or both, from such resources as the DOM test suite, [6]. CSS, [7], should be applicable as well (for future DT test generation).
This is meant to be a cookbook for writing the test cases for a functional module. Functional modules have generally corresponded to chapters in the SVG spec. These techniques were prototyped with the Path chapter, and have been applied in the generation of BE-level tests for the whole spec.
Note. When we did the Path chapter, an exhaustive TR extraction was one of the first actions. This is not necessary prior to BE test case specification, and the TR analysis is postponed until after the BE test case generation in what follows.
Implicitly or explicitly, formally or informally, in this order or another which you prefer, you will go through these steps for writing BE tests:
DT test generation involves more rigor and more systematic methodology than BE test generation. The basic steps for DT tests are similar, with some differences at the beginning:
The rest follows as for BE tests. The difference between the DT outline and the BE outline is in the rigor and thoroughness of the first steps, which are deciding "what to test".
The naming convention for SVG tests is: chapter-focus-{BE|DT|ER|DM}-NN (NN=01, 02, ...).
Decide how to subdivide your functional area ("chapter") into subsections -- "focus" sections.
Chapter is self explanatory -- a one-word, though possibly complex, name for your document chapter or functional area. Examples: path, coordSystem, clipMaskComposite.
Focus is simple to specify in some cases: shapes-rect-...; path-lines-...; filters-feColorMatrix-...
In some cases, focus might not seem obvious. However, the "focus" component of the name is always required.
"{BE|DT|ER|DM}" indicates that exactly one of the two-letter test type designators is to be used, BE or DT or ER or DM. The numbering runs consecutively throughout the chapter, it does not restart with each focus subsection.
Note. Use "camel case" for compound words for chapter and focus. The first letter is lower case, and the first letter of subsequent words is upper case. Examples: gradPatt, radialGradient, textAnchor.
Remember, at the BE level, we are only trying to verify that the interpreter or viewer has implemented the given functional area. Therefore, we focus on the major functional pieces of the chapter.
There are no firm rules as to what comprises a BE test and what is DT -- sometimes it's a judgement call. However, here are some helpful guidelines:
Example. The 'path' element has a "d" attribute, which can contain a number of commands: Mm, Ll, Zz, Hh, Cc, ... It also has the "nominalLength" attribute, which is unique to Path. The BE tests for path give a basic exercise of these attributes and commands, including verification that the implementation understands the concept of subpath (holes and islands).
Note that there are also attributes like "style", "class", "transform", which are functionalites widely applicable to other parts of SVG. It is a judgement call, but in the path BE tests, we avoid these details -- they will in fact be attacked in their own modules, such as Styling, Transform, etc. (In other words, we don't deal with them extensively in the Path tests, not even in the DT tests.)
We adopted a principle for the Path tests: just enough styling -- basic colors, etc -- to make the tests visually less grim (than b/w, one-pixel wide lines, no fill). This principle is applicable for all tests -- do not unnecessarily clutter the test with functionalities which are unrelated to the functionality being tested.
When starting to design the BE tests, look at the existing very simple examples in the SVG spec for starters.
The guidelines in this section apply to DT, ER, and DM tests, as well as BE tests.
The test description is only for you, the test designer, unless you're dividing the labor into specification versus production -- different people doing each -- which we actually did on the Path prototype. Nevertheless, experience shows that the challenge of writing the description force one to actually design the test in sufficient detail that it is easy to then write SVG content to implement it.
It is the Conceptual Description of the test (see below) which is useful at this early stage (and it might be done after you have sketched/drawn the Test Case).
In the Path prototype, the following format proved useful in describing Test Cases:
See later section for writing the Operator Script (#4).
The Associated Test Requirements (#5) and Document References (#6) are the crux of traceability. You will have generated this information (implicitly, at least) by the time you have designed and written your test case. If you are writing test cases, generate and preserve this information (see, for example, next section and [11]).
See the next chapter, "Test Review Guidelines", for a concise summary of other details which you should pay attention to, when you design and write a test case.
Note.Traceability data currently are not integrated into the completed (REC SVG) BE suite. This will have to be done (as links into the SVG spec) in a future test suite release.
A comprehensive list of Test Requirements is the critical first step in writing a comprehensive set of DT test cases for your functional area.
There is no magic rule. See [9], section 4.2, for an interesting discussion of this. See [11] for a fairly thorough (as far as it went -- it's incomplete) example for the Path chapter.
It starts with an intensive reading of each sentence of your chapter. Weigh the question, phrase by phrase: is there a testable assertion here? If so, highlight it and add it to your list (however you want to manage it. You also will need to read some other chapters, at least:
and maybe others like Accessibility. Depending on your chapter, you might be led off into other sections of the document for requirements, or into other standards (e.g., CSS2).
Build up a list of TRs -- testable assertions -- which might be applicable to a viewer or interpreter. In fact, don't descriminate by "applicable to" at this stage. The SVG spec is written to describe a file format, and the semantic requirements associated with data elements are not always explicitly stated. For example from the Path chapter:
"The command letter can be eliminated on subsequent commands if the same command is used multiple times in a row."
This is a statement about allowable data configurations within 'path' elements. But combined with the statement form Appendix G, "All SVG static rendering features ... must be supported and rendered ...", a testable assertion about viewers results (and a test purpose can be derived).
You'll be lucky to find any (or many) statements which jump out with a "shall" or "must" -- it doesn't occur once in my (incomplete) list of 59 TRs for Path, in [11].
So for a first pass, pick up anything and everything which looks like it might lead to a Test Requirement on an SVG viewer.
For the Path chapter, I assembled a list of Test Requirements after the first intensive reading, during which I did markup on paper. You can follow this sort of process, or do whatever is most agreeable for you.
Each entry in the list contained a document reference, and the text of the TR -- I did cut-and-paste against the HTML version of the document for the latter (note: there is some danger of volatility with this, at this stage of the document).
Simple example:
Reference: 10.3.2.Mmtable
Statement: (x,y)+ -- Mm must be followed by one or more x,y pairs.
Lengthier example:
Reference: 10.4.p1.b2
Statement: nominalLength= The distance measurement (A) for the given 'path' element computed at authoring time. The SVG user agent should compute its own distance measurement (B). The SVG user agent should then scale all distance-along-a-curve computations by A divided by B.
I used the following ad-hoc (pseudo-Xpointer) notation for referencing Test Requirements: pN.bN.sN, where pN = paragraph N, bN = bullet N, sN = sentence N. Plus unambiguous constructions like "Mmtable" to point at tables and table entries.
See [11] for the example of the complete listing of those Test Requirements (TR) which pertain to geometry and syntax of Path (extracted from the 19990914 SVG spec.)
We want to turn the TR list into a number of Test Cases which exhaustively covers the requirements in the TR list.
The first step, implicitly or explicitly, is derivation of a set of Test Purposes associated with the TR list. Example:
TR: "Mm must be followed by one or more x,y pairs"
TP: "Verify that interpreter correctly handles Mm with one (x,y) pair, or several pairs, or many pairs."
Note that, because of the "Error Processing" requirement about Path, that this suggests another TP: "Verify that interpreters respond correctly to invalid Mm data combination." (Such as "M x L x y", or "M L x y", or "M x y x y x Z".
This leads to another point. The general conformance requirements of Appendix G imply a list of "Generic Test Purposes" (see [4.3]):
Keep this list at hand while you are looking at your TR list and deciding what to test, i.e., deciding Test Purposes. (This list might be extended -- suggestions welcome.)
I have been a bit informal with my "TP list", but I still keep track of what I have covered on the TR list and the generic TP list, so that I know when I'm done. See, for example, [11], the section, "Detailed Drill-down Tests (DT) for Line Commands."
The final principle here, once you have an idea of what you're going to test and how, is to put a reasonable number of "atomic" tests together to make a Test Case. This will reduce the number of individual test cases and increase their content density. The guiding principles should be:
The first release of a simple XML grammar for describing tests, and the XSLT stylesheet for producing the HTML page, featured:
These were the extent of the "first generation" production tools. If you process the XML instances with CreateHTMLHarness.xslt (#2), you will get HTML pages which pull together and presents the PNG reference images, the operator scripts, and navigation buttons for the suite. Various batch commands have been supplied as well, to assist in making one or more of the harness types for one or a list of test cases.
Subsequent to the initial release of these tools, #2 - #4 have been rewritten, factored to improve maintainability, and augmented with another harness -- all-SVG which presents SVG and PNG side-by-side (but with no OS).
These tools are all in the CVS repository, in the tools subdirectory.
If you process the XML instances with CreateSVGHarness.xslt (#3), you will get a parallel set of SVG pages with SVG elements for navigation buttons, and inclusion by reference of the test case SVG instances themselves.
If you process the XML instances three times respectively through the three XSLT stylesheets of #4, you will get a set of three HTML files which generate frame-based pages. PNG is displayed side-by-side with SVG (the latter via a plugin), and the Operator Script is displayed below, in the bottom frame.
Initially I used only the XT tool of James Clark (get it from his Web site). You can use whatever tool you prefer, but a caveat -- different XSLT processor may give inconsistent results. Since the rewriting and revision of the stylesheets, and addition of the 4th harness (all-SVG), I use Apache/xalan for the two all-SVG harnesses (to avoid an anomoly involving appearance of unneeded namespace declarations).
A set of DOS batch command files exists, to facilitate generation of the harnesses. These are documented elsewhere (TBD).
A "manual" SVG template has been released as well.
The scheme has now been developed to the point that automatic generation of the SVG skeleton file, with some of the details filled in (see next section), is now possible. A set of advanced stylesheet tools was designed to do this from a slightly expanded XML grammar (featuring things like 'desc' strings, elements for author and creation date, etc). The final second-generation tools, however, stayed with the simple initial grammar and don't automate all of the required information. Therefore the static-output-template.svg is still the place to start if you are writing a new test case.
Starting with the static-output-template.svg and:
Note about #5: this is critical. All critical content should be inside of the "test-body-content" group. The second generation tools and maintenance processes rely on it --most of the stuff outside of this group is automatically generated, and it wouldd be lost if you put anything important there (the 'desc', and comment preamble will be preserved). Though these (2nd gen) tools are not in production, it would be wise to keep the option open.
Note about #2. That the test should be self-documenting implies that the picture should have a <text> element equivalent to the <desc> (across the very top is a good location, in most cases).
Use good and thorough comments in the SVG content itself to describe what everything is doing.
There may be some test purposes (e.g., the structure-emptySVG-BE-01.svg test) which require no graphical content, in which (only) cases the Legend may be omitted.
See below about the Serial Number (now superseded with CVS's Revision keyword).
See the next chapter, "Test Review Guidelines", for a concise summary of other details which you should pay attention to, when you design and write a test case.
The key feature of the XML instance is the Operator Script. Note when you fill in your XML instance, you will also define the name (per convention documented herein), and navigation links. The latter define the next and previous TCs in the sequence for link navigation through the suite.
Be sure that your TC's XML instance links correctly to its TC neighbors, and vice-versa! This is probably the most common mistake amongst contributors -- failure to adjust the links of neighbors when a new test case is added (or a test case is removed). (This is sufficiently annoying that an automated verification or even correction tool is on "the queue", for some future development.)
The Operator Script comprises a few sentences and is written as one or more paragraphs of the XML instance (see earlier section) for the test case.
Once again, there are no firm rules. However, the Operator Script can address any or all of:
#2 could conceivably be: "picture should look like the PNG". However, some specifics could be pointed out, such as (for an accuracy test): "All lines should pass through the cross-hairs", or "Vertexes should be at the locations of the markers".
#3 and #4 go together. If there are allowable deviations of the rendered SVG from the PNG, it should be stated (e.g., maybe a style falls back to the default style sheet, which can vary).
About #6, optionality. If a test is exploring an optional or recommended feature, that should be clearly indicated right at the beginning of the operator script.
#7 refers to a brief description of SVG functionalities, other than the one under test, which are used in the test file instance.
In addition to the other purposes of the Operator Script, a well-detailed Operator Script can be useful as an aid to accessibility (#8).
Note. The following describes first-generation methods for getting the PNG reference images. New tools are available from several sources, e.g., Adobe, CSIRO, and the Apache/batik project, which allowed direct generation of correct PNG files for all test cases in the second public release, i.e., the complete BE suite.
When you develop a test case, you submit to the repository: the PNG reference image; and, a description of how you generated the PNG.
By far, the most common method of generating the PNG reference image is:
The screen capture method relies on having an SVG viewer which can correctly display the picture. Sometimes, especially in the early days of viewer implementation development, this is not possible -- no SVG viewer handles the test instance correctly.
However, it is often the case that there is another SVG file which is exactly equivalent (pictorially). These are called "patch" files, and are named, for example: structure-nestedSVG-BE-02-patch.svg (actual example).
Example: viewer doesn't correctly establish the origin of the user space for a simple test of nested SVG elements. Then compensate for the viewer error by changing the coordinates of the innermost graphical elements so that the viewer positions them (graphically) correctly.
Example: viewer defaults something wrong. Then compensate by explicitly setting that value (assuming that the viewer does this correctly).
Example: multi-stop gradients don't work, but two-stop are correct. Make a "-patch" file where the correct picture of a multi-stop gradient is built by stringing together multiple two-stop gradients.
Test contributors should submit any "-patch" files, along with the PNG files and "how to" description.
This technique was used by some contributors, before the development of SVG viewers was very advanced:
With this method, you should be on guard against accuracy issues, as the SVG and the PNG result from independent and disjoint drawing pipelines.
A variant of this is to use a graphics program to draw just the incorrect piece of the SVG rendering, and then cut-paste with a raster editor to get a complete and correct reference PNG.
This method has been postulated, but (to my knowledge) no yet used by anyone. It would be equally applicable to formats other than CGM, when transcoders are available.
Whether or not this will work for your test case depends on whether the result of step #3 is close enough to the SVG configuration you need for the test (and correct!), and more importantly, whether the hand-editing of #4 preserves the graphical accuracy of the rendered picture.
A variant, for simple test cases, would be to hand-code the desired SVG, hand-code (graphically) equivalent clear text CGM, and not use the transcoder at all -- just use hand-coded CGM as a route to a correct picture.
This would only be useful, in place of the previous methods, if none of the existing SVG viewers could get a correct rendering of the desired SVG test case, and it was too difficult to reproduce the desired drawing in a graphics program.
Three aspects of the PNG file generation should have your attention:
The only exceptions for #1 are test cases which specifically deviate from the canonical 450x450 coordinate space, in order to test viewer handling of different SVG address spaces. In this case, match the SVG test instance.
8-bit PNG should suffice for most tests -- 256 colors are possible. 24-bit PNG is likely to be required for tests such as:
While it might be possible to compute the number of colors required in some of these cases, and optimize with PNG-8, nevertheless it is strongly recommended to be conservative and use PNG-24, if there is any doubt.
For these same cases which require PNG-24, it has been discovered that attention must be given to the color mode of the monitor, if screen capture is being used. On PC Windows systems, for example, noticeable color banding has occurred on some tests when using "High Color" (16-bit) mode, and it disappears if "True Color" (24-bit) mode is used.
Avoidance of color artifacts is not yet completely understood. However, be aware that color management systems on your computer may lead to incorrect colors in the PNG reference image. One commonly seen example is a faint pink tinge to areas that should be white. These color management artifacts can sometimes be detected by using tools in raster editors (such as Adobe ImageReady 2.0, Corel Photopaint, Macromedia Fireworks.)
During CGM test suite development, a major annoyance and quality impact arose from not being able to keep synchronization between the reference image (the PNG files, for this SVG test suite), and the test case (SVG for us). Changes made to the latter often weren't reflected by updating the former. Worst of all, there was no way to detect the problem when it occurred.
A "serial number" in the SVG, which is encoded in graphical text, is the solution for this -- it is quick and easy to determine if the PNG corresponds to the SVG file and a given rendering of the SVG file (e.g., printout or screen image) ... assuming that the PNG was generated from the SVG!
The serial number is part of the Legend of the SVG file. Previously, the serial number was manually maintained and updated, which proved to be something of an annoyance. Nevertheless, the version control benefits warranted the inconvenience. Originally, the serial number was identical to a version number -- 1, 2, 3, ... Its maintenance was solely the responsibility of the test suite editor, which somewhat alleviated the error-prone manual aspects.
Now, the serial number text string (in the Legend of test cases and templates) has been replaced with the Revision keyword of the CVS system. Every 'commit' of a changed SVG file, no matter how trivial the change, automatically updates the serial number. It is identical to the revision number in CVS.
Note: a very few .SVG files need to be treated as binary instead of text (e.g., ones with wide range UTF-16 character codes). Presently (10/2001), there are only two such test cases (still true?). Keyword substitution is suppressed for these. Therefore the serial number must be manually typed into the Legend after the rest of the test case production, and should match the current CVS revision number.
There are two reasons that these guidelines are provided:
Considering the chapter and its set of test cases as a whole, assess:
Looking at the SVG test cases instances individually, evaluate:
Note. About "Self-documenting (i.e., in rendered content)," the style of in-picture animation should be to describe what is being tested, but should not describe visual effect (the latter may be done in Operator Script). So for example,
Specifically, this refers to the Operator Script, which is to be evaluated for:
In addition, evaluate the correctness of the navigation links:
Evaluate at least these criteria for the PNG reference image:
A test which lightly exercises one of the basic functionalities of the SVG specification. Collectively, the BE tests of an SVG functional area (chapter) give a complete but lightweight examination of the major functional aspects of the chapter, without dwelling on fine detail or probing exhaustively. BE tests are intended to simply establish that a viewer has implemented the functional capability.
A test which is intended to show the capabilities of the SVG specification, or a functional area thereof, but is otherwise not necessarily tied to any particular set of testable assertions from the SVG specification.
Also called drill-down tests. DT tests probe for exact, complete, and correct conformance to the most detailed specifications and requirements of SVG. Collectively, the set of DT tests is equivalent to the set of testable assertions about the SVG specification.
See Detailed Test.
An Error Test probes the error response of viewers, especially for those cases where the SVG specification describes particular error conditions and prescribes viewer error behavior.
See Test Requirement.
See Test Requirement.
A testable assertion which is extracted from a standard specification. Also called Semantic Requirement (SR) or Test Assertion (TA) in some literature. Example. "Non-positive radius is an error condition."
A reformulation of a Test Requirement (or, one or more TRs) as a testing directive. Example. "Verify that radius is positive" would be a Test Purpose for validating SVG file instances, and "Verify that interpreter treats non-positive radius as an error condition" would be a TP for interpreter or viewer testing.
As used in this project, an executable unit of the material in the test suite which implements one or more Test Purposes (hence verifies one or more Test Requirements). Example. An SVG test file which contains an elliptical arc element with a negative radius. In practice (and abstractly), the relationship of TRs to TCs is many-to-many.
The ability, in a test suite, to trace a Test Case back to the applicable Test Requirement(s) in the standard specification.