W3C logoWeb Accessibility Initiative (WAI) logo

WAI: Strategies, guidelines, resources to make the Web accessible to people with disabilities

WCAG 2.0 Test Samples Development Task Force (TSD TF) Review Process

Page Contents

Introduction

The WCAG 2.0 Test Samples Development Task Force (TSD TF) uses the following process to review test samples before they are published on the WCAG 2.0 Test Samples Repository. This review process is based on Conformance Test Process For WCAG 2.0. The status of a given test sample with respect to this review process is indicated by the status element in the metadata, see WCAG 2.0 Test Samples Metadata for more information.

Review Steps

Flow diagram of the review steps as described in the adjacent text

Step 1: Submission

Description: a new test sample is submitted to the task force for review. Currently the test samples need to be committed to the CVS repository manually but in the future a Web interface will be provided. The status flag of the test sample must be set to unconfirmed to initiate the review process.

Note: in some cases the test samples are resubmitted after substantial improvements and rewriting. However, they are still treated like new test samples and must undergo the entire process.

Step 2: Structure Review

Description: a task force participant receives the assignment from the task force facilitators to review the completeness and syntax of the test sample files. The participant uses the test samples management tool (restricted access) to carry out the review and record the outcome. The Checklist for Structure Reviews specifies the criteria for structure reviews. If the structure of the test sample passes the review criteria, the status flag is set to new and the test samples continues to the next stage in the review process. Otherwise the status flag is set to reject and the participant notifies the submitter of this outcome.

Note: currently the structure review is carried out manually using the Checklist for Structure Reviews. In the future a script will help automate many of these checks.

See also: FAQ on Structure Reviews

Step 3: Content Review

Description: a task force participant receives the assignment from the task force facilitators to review the soundness and implication of the test sample contents. The participant uses the test samples management tool (restricted access) to carry out the review and record the outcome. The Checklist for Content Reviews specifies the criteria for content reviews. Once a content review is carried out, the status flag of the test sample is set to ballot to indicate that the test sample is ready for ballot review by the rest of the task force participants.

Step 4: Ballot Review

Description: all task force participants receive the assigment from the task force facilitators to vote on status of the test sample. The reviews carried out in Step 2 and Step 3 of this review process are used by the task force participants as a basis to cast a vote. The participants use test samples management tool (restricted access) to carry out the review and record the vote. After the voting period for a test sample ends, the balloting is closed by setting the status flag to pending. This indicates that the voting is complete, and the test sample is ready for a task force decision.

Step 5: Task Force Decision

Description: the test sample including its review history is brought to a teleconference agenda for a formal decision by the task force participants. If no objections or issues were raised through the ballot reviews, or if these were settled during the teleconference, then the status flag of the test sample is set to holding to indicate that the test sample has been accepted by the task force. The teleconference discussion may conclude an issue with the WCAG 2.0 Techniques rather than with the test sample itself. In this case the status flag is also set to holding, and a comment is supplied to indicate that the test sample needs further review by the WCAG Working Group. The issue is recorded using the test samples management tool (restricted access).

The task force may also decide that minor fixes are needed to better improve the test sample. In this case the test sample is (re-)assigned to a task force participant who notifies the author of the the review result and maintains a dialog to help the author fix the relevant issues. The status flag of the test sample is set to fixes until the author has carried out the necessary changes. It is then set to new to initiate another Content Review (as described in step 3 of this review process). In some cases insignificant editorial improvements need to be made but overall the test sample has been accepted by the task force. Also here the test sample is (re-)assigned to a task force participant who notifies the author of the review results and coordinates the improvements. The status flag is set to edits until the changes are made, in which case it is then switched to holding.

Finally, the task force may decide that the test sample is not acceptable and needs substantial fixes before it can be reconsidered. In this case the status flag is set to rejected, and a task force participant receives the assignment to notify the author of the review result. Note: it is left to the discretion of the task force facilitators to decide if the issues raised through the ballot reviews or during the teleconference consitute sufficient changes for Minor Edits, Minor Fixes, or Substantial Fixes.

Step 6: WCAG Working Group Decision

Description: the objective of this task force is to pre-process test samples for the WCAG Working Group. However, since the WCAG Working Group has ownership of all WCAG 2.0 documents and materials, it is responsible for a final review and decision. This step in the review process is not further specified at the current time, it is an interface to the WCAG Working Group who will have their own processes for making formal decisions on the queued test samples.

Summary of Status Labels

unconfirmed
test sample has been received and is not yet reviewed
new
test sample successfully passed the structure review
ballot
test sample underwent a content review and is available for group balloting
pending
test sample underwent balloting and is available for a formal TF decision
fixes
test sample needs minor fixes and a renewed content review and balloting
edits
test sample needs minor edits before it can be queued for final decision
holding
test sample is queued for final decision by the WCAG Working Group
rejected
test sample has been rejected by the Task Force or the WCAG Working Group
accepted
test sample has been accepted by the WCAG Working Group
deprecated
test sample has been deprecated by the Task Force or the WCAG Working Group

Checklist for Structure Review

The following is a list of (objective) criteria for carrying out a review of the structure of test samples (see also the FAQ on Structure Reviews):

Contact Information (available in Wiki)
  • Name and e-mail address of the submitter are available (required);
  • Organization on whose behalf the test sample was submitted (optional);
General Checks (applicable to files in both metadata folder and testfiles folder)
  • All the submitted files follow the naming convention and directory structure;
  • All the files include correct links unless otherwise required by the test;
  • All the files include correct spelling unless otherwise required by the test;
Checks for Test Files (applicable to files in testfiles folder)
  • All the files that are necessary to execute the test procedure have been submitted;
  • All the files include valid markup unless otherwise required by the test;
Checks for Metadata (applicable to files in metadata folder)
  • The metadata files are valid TCDL 2.0;
  • The metadata files use only elements and attributes specified in the Metadata Vocabulary;
  • All the dates and other integer or literal values have the correct format (as specified in the Metadata Vocabulary);
  • All static values (especially copyright notices) are included and accurate;
  • All titles, descriptions, and other required fields are included and accurate;
  • All identifiers (especially ID for techniques and rules) are used correctly;
  • All structures such as rules, techniques, or pointers are used correctly;

Checklist for Content Review

The following is a list of (subjective) criteria for carrying out a review of the content of test samples: