Submitting, accepting and refuting tests

Dimitris Dimitriadis

W3C DOM WG

Contact for DOM TS Issues

W3C DOM TS - General

• Publically developed (to involve both developers and compaines
• www-dom-ts@w3.o rg (public mailing list for the development of the DOM TS
• There will be a TS for each level of the DOM
• Jointly developed by W3C and NIST (level 1 being finlaized, future levels will be based on experience made in this effort)

W3C DOM TS - Process

• Procdural issues
• We need a stable technical architecture (for version handling, savind, documentation and so forth)
• More importantly, we need a stable mechanism for submitting, resolving errors in the tests and interpretation of the specifications
• Process document
• http://www.w3.org/DOM/DOMTS -Process
• Deals with the whole process from submitting tests to conformance claims

W3C DOM TS

• A test, or a series of tests, get submitted to the DOM TS. Receipt is acknowledged and a mail is sent to the DOM TS mailing list (a sanity check before submitting is required).
• The moderator checks the test(s) against the relevant DOM specifications.
• The test(s) receive the "received" status

W3C DOM TS - Review

• Reviewing tests
• Relevance of the submitted test.
• Part of the specification being testes (also indicated in the XML test document)
• Quality of code
• Scenario of test(s) (Semantic requirements)
• If the test meets all criteria, it receives the "reviewed and stable" status. It becomes part of the DOM TS.
• If the test does not meet all criteria (except in some cases the scenario criterion), it receives the "inappropriate" status.

W3C DOM TS - Reevaluation

• Reevaluating tests
• Received and stable tests that are found to be erroneous:
• The DOM WG representative checks the test(s) again. The final authority is the DOM WG in case this is needed.
• Moving between states
• Tests that have been received the "received and stable" status and are at a later stage found to be erroneous are lifted out of hte TS and evaluated by the TS moderator.
• In difficult cases, where the test seems OK but won' work, or in cases where the specification is being questioned, the WG studies the test.
• If, after consideration, the test is considered to be OK, it again forms part of the DOM TS.
• If, after consideration, the test is considered to fail, it receives the "inappropriate" status.
• In any case, notification to public mailing list.

W3C DOM TS

• DOM TS Constituents
• Submitted, reviewed and stable tests.
• There wil be different versions of the DOM TS, since there are different levels of the DOM
• Up-to-date list of existing and wanted tests
• www-dom-ts@w3.o rg
• Deliverables
• DOM TS (process, architecture, scope, design)
• TS design guidelines
• Documentation of tests, expected results, scenarios, etc.

W3C DOM TS - QA

• DOM TS QA
• Will monitor guidelines developed in the QA activity (if pursued)
• Process presupposes sanity check before submitting tests
• Generic XML format for tests includes information on part of specification being tested (if atomic), expected reluts, scenario (if scenario-driven test)

W3C DOM TS

• Open questions
• Testing the test suite (done by moderator/author at this point)
• Ensured if granular enough test format is developed (vis. expected results)
• Language-neutral definition of tests (to generate other than the two current language bindings, using eg. XSLT to generate the specific language binding)
• We're looking into automatic generation of tests based on the specification
• Moving between states, efficiancy of current model.

W3C DOM TS

• Questions