Introduction

• Two graphical test suites
• SVG (at Candidate Recommendation, 4/2001):
  • a basic but comprehensive set of viewer tests
  • built from scratch within SVG Working Group
  • started at "stable Working Draft" (1/2000)
• WebCGM (Recommendation, 1/1999):
  • basic plus some detailed viewer tests
  • built from adapted (NIST-ATA) plus new materials
  • started 2 years after Recommendation

Components of the Suites

Both test suites (SVG/WebCGM) consist of:

• XML test case description database
• set of test case instances (.svg/.cgm files)
• matching set of raster reference images (expected results)
• set of "operator scripts" -- instructions, pass/fail, test descriptions
• test harness to organize, coalesce, and present suite & results
• Miscellaneous production and managment tools

(Refer to test suites for examples of each. To be presented with slides.)

Goals of the Two Suites

• Self-assess viewer adherence to specs; promote viewer interoperability

  (Not certification, not goodness, not example/demo, ...)

• Progressive testing, in both development and application of suites:
  1. Basic Effectivity first (BE) -- "breadth-first", light weight
  2. Detailed second (DT) -- comprehensive and detailed.
• Apply a uniform rigorous methodology:
  1. analyze document for testable assertions (Test Requirements)
  2. synthesize a set of Test Purposes
  3. embody the set of TPs in a set of test case instances

Top Problems and Issues

• Both projects short-cut traceability.
• Specs (Recommendations) not always conducive to testability.
  • normative language; optional features; organization; complexity.
• Visual comparison (w/ reference image) is imprecise and error prone.
• Uniformity and accountability difficult working within WG.
• Overall, these projects (in several aspects) are very labor intensive.

See the workshop paper for discussion of additional problems and issues.

Main Conclusions

• Improve specifications (Recs) themselves -- guidelines and standards needed.
• Early conformance work confers big benefit to quality of specifications (Recs)
• Start some conformance work (e.g., TR/TP, some BE, etc) in WG early in spec life.
• Realistic plan (acknowledging resource needs) for finishing work.
• Address in WG charter: what, how, when, by whom conformance will be done.
• Attention to more automation methods and tools, especially:
  • TR/TP extraction and synthesis; traceability implementation.
  • comparison of actual and expected results

See the workshop paper for additional issues, lessons, observations.

Note about Accessibility

• Harnesses
  • the dynamic JavaScript harness (WebCGM) is more powerful and flexible
  • the static-linked page set of SVG is more accessible
  • both are driven off of similar XML databases, so both can be provided

• Operator Scripts
  • terse pass/fail criteria of existing (NIST-ATA) WebCGM tests -- designed for certification operators -- are not very accessible
  • the style of the SVG BE conformance suite provides, essentially, detailed alternative text to the TC pictures and reference images.