WCAG 2.0 Test Samples Development Task Force (TSD TF) Review Process

Page Contents

Introduction
Review Steps
Status Labels
Structure Review
Content Review

Introduction

The WCAG 2.0 Test Samples Development Task Force (TSD TF) uses the following process to review test samples before they are published on the WCAG 2.0 Test Samples Repository. This review process is based on Conformance Test Process For WCAG 2.0. The status of a given test sample with respect to this review process is indicated by the status element in the metadata, see WCAG 2.0 Test Samples Metadata for more information.

Review Steps

Step 1: Submission

Input label: none
Output label: unconfirmed

Description: a new test sample is submitted to the task force for review. Currently the test samples need to be committed to the CVS repository manually but in the future a Web interface will be provided. The status flag of the test sample must be set to unconfirmed to initiate the review process.

Note: in some cases the test samples are resubmitted after substantial improvements and rewriting. However, they are still treated like new test samples and must undergo the whole process.

Step 2: Structure Review

Input label: unconfirmed
Output label: new, reject

Description: a task force participant receives the assignment from the task force facilitators to review the completeness and syntax of the test sample files. The participant uses the Checklist for Structure Reviews which specifies the criteria for this review, and records the outcome of this review using Wiki Template for Structure Reviews. If the structure of the test sample is correct and complete then its status flag is set to new and the test samples continues to the next stage in the review process. Otherwise the status flag is set to reject and the participant notifies the submitter of this outcome.

Note: currently the structure review is carried out manually using the Checklist for Structure Reviews. In the future a script will help automate many of these checks.

See also: FAQ on Structure Reviews

Step 3: Content Review

Input label: new
Output label: assigned

Description: a task force participant receives the assignment from the task force facilitators to review the soundness and implication of the test sample contents. The participant uses the Checklist for Content Reviews which specifies the criteria for this review, and records the outcome of this review using the Wiki Template for Content Reviews. The status flag of the test sample is set to assigned to indicate that the test sample is ready for review by the rest of the task force participants.

Step 4: Online Strawpoll

Input label: assigned
Output label: pending

Description: all task force participants receive the assigment from the task force facilitators to vote on status of the test sample. The reviews carried out in Step 2 and Step 3 of this review process are used by the task force participants as a basis to make a decision. The participants use the W3C Web Based Survey (WBS) ballot which will be provided by the task force facilitators. After the voting period for a test sample ends, its status flag is set to pending to indicate that the voting is complete, and the test sample is ready for a task force decision.

Step 5: Task Force Decision

Input label: pending
Output label: holding, assigned, rejected

Description: the test sample including its review history is brought to a teleconference agenda for a formal decision by the task force participants. If no objections or issues were raised through the Online Strawpoll, or if these were settled during the teleconference, then the status flag of the test sample is set to holding to indicate that the test sample has been accepted by the task force. The teleconference discussion may conclude an issue with the WCAG 2.0 Techniques rather than with the test sample itself. In this case the status flag is also set to holding to indicate that the test sample needs further review by the WCAG Working Group. The issue is recorded using the Wiki Template for WCAG 2.0 Issues

The task force may also decide that minor fixes are needed to better improve the test sample. In this case the test sample is (re-)assigned to a task force participant who notifies the author of the the review result and maintains a dialog to help the author fix the relevant issues. The status flag of the test sample is set to assigned until the author has carried out the necessary changes. It is then set to new to initiate another Content Review (as described in step 3 of this review process). In some cases insignificant editorial improvements need to be made but overall the test sample has been accepted by the task force. Also here the test sample is (re-)assigned to a task force participant who notifies the author of the review results and coordinates the improvements. The status flag is set to assigned until the changes are made, in which case it is then switched to holding.

Finally, the task force may decide that the test sample is not acceptable and needs substantial fixes before it can be reconsidered. In this case the status flag is set to rejected, and a task force participant receives the assignment to notify the author of the review result. Note: it is left to the discretion of the task force facilitators to decide if the issues raised through the Online Strawpoll or during the teleconference consitute sufficient changes for Editorial Fixes, Minor Fixes, or Substantial Fixes.

Step 6: WCAG Working Group Decision

Input label: holding
Output label: currently undefined

Description: the objective of this task force is to pre-process test samples for the WCAG Working Group. However, since the WCAG Working Group has ownership of all WCAG 2.0 documents and materials, it is responsible for a final review and decision. This step in the review process is not further specified at the current time, it is an interface to the WCAG Working Group who will have their own processes for making formal decisions on the queued test samples.

Summary of Status Labels

unconfirmed: test sample received and under review for completeness
new: test sample is complete and under initial evaluation
assigned: test sample is currently being reviewed by the task force
pending: review complete, waiting for decision on teleconference
holding: waiting for a decision by the WCAG Working Group
rejected: task force or WCAG Working Group decision
accepted: test sample has been accepted by the WCAG Working Group
deprecated: test sample has been become obsolete (for example due to changes in the WCAG documents or other reasons)

Checklist for Structure Review

The following is a list of (objective) criteria for carrying out a review of the structure of test samples (see also the FAQ on Structure Reviews):

Contact Information (available in Wiki)

Name and e-mail address of the submitter are available (required);
Organization on whose behalf the test sample was submitted (optional);

General Checks (applicable to files in both metadata folder and testfiles folder)

All the submitted files follow the naming convention and directory structure;
All the files include correct links unless otherwise required by the test;
All the files include correct spelling unless otherwise required by the test;

Checks for Test Files (applicable to files in testfiles folder)

All the files that are necessary to execute the test procedure have been submitted;
All the files include valid markup unless otherwise required by the test;

Checks for Metadata (applicable to files in metadata folder)

The metadata files are valid TCDL 2.0;
The metadata files use only elements and attributes specified in the Metadata Vocabulary;
All the dates and other integer or literal values have the correct format (as specified in the Metadata Vocabulary);
All static values (especially copyright notices) are included and accurate;
All titles, descriptions, and other required fields are included and accurate;
All identifiers (especially ID for techniques and rules) are used correctly;
All structures such as rules, techniques, or pointers are used correctly;

Checklist for Content Review

The following is a list of (subjective) criteria for carrying out a review of the content of test samples:

Clear Scope - the test sample must focus on a single success criterion and avoid combining several success criteria in one test. Its scope and metadata must only reflect the success criterion that it addresses. In other words, although been focused on a single success criterion, a test sample may be affected by other success criteria but, as they will not be addressed, they must not been described in the metadata;
Neutral to Interpretation - the test sample must not be used to reinterpret the test procedure(s), for example by using the expertGuidance field to rephrase the test procedure(s) rather than to describe the specifics of executing the given test sample;
Minimal & Complete - the test sample should be minimal yet complete and realistic. For example, it should not include additional content that is not necessary to carry out the test procedure(s) but at the same time the content should be sufficient to highlight a realistic scenario;
Effective & Efficient - the test sample should facilitate the execution of the test procedure(s). For example, can the test sample be restructured or simplified to improve its effectiveness or efficiency?
Consistent - the content of the test sample should be consistent with its description in the metadata and if applicable with the relevant technique(s) and test procedure(s). Especially the outcome of executing the test sample must be consistent with its description and intention.

Site Navigation