Skip to content 
Translations
About RSS
SearchWCAG 2.0 Test Samples Development Task Force (TSD TF) Review Process
- Overview
- All Tests
- Search Tests
- [Submit Tests]
- Metadata Format
- Review Process
Page Contents
Introduction
The WCAG 2.0 Test Samples Development Task Force (TSD TF) uses the following process to review test samples before they are published on the WCAG 2.0 Test Samples Repository. This review process is based on Conformance Test Process For WCAG 2.0. The status of a given test sample with respect to this review process is indicated by the status element in the metadata, see WCAG 2.0 Test Samples Metadata for more information.
Review Steps
Step 1: Submission
- Input label: none
- Output label:
unconfirmed
Description: a new test sample is submitted to the task force for review. Currently the test samples need to be committed to the CVS repository manually but in the future a Web interface will be provided. The status flag of the test sample must be set to unconfirmed to initiate the review process.
Note: in some cases the test samples are resubmitted after substantial improvements and rewriting. However, they are still treated like new test samples and must undergo the entire process.
Step 2: Structure Review
- Input label:
unconfirmed - Output label:
new,rejected
Description: a task force participant receives the assignment from the task force facilitators to review the completeness and syntax of the test sample files. The participant uses the test samples management tool (restricted access) to carry out the review and record the outcome. The Checklist for Structure Reviews specifies the criteria for structure reviews. If the structure of the test sample passes the review criteria, the status flag is set to new and the test samples continues to the next stage in the review process. Otherwise the status flag is set to reject and the participant notifies the submitter of this outcome.
Note: currently the structure review is carried out manually using the Checklist for Structure Reviews. In the future a script will help automate many of these checks.
See also: FAQ on Structure Reviews
Step 3: Content Review
- Input label:
new - Output label:
ballot
Description: a task force participant receives the assignment from the task force facilitators to review the soundness and implication of the test sample contents. The participant uses the test samples management tool (restricted access) to carry out the review and record the outcome. The Checklist for Content Reviews specifies the criteria for content reviews. Once a content review is carried out, the status flag of the test sample is set to ballot to indicate that the test sample is ready for ballot review by the rest of the task force participants.
Step 4: Ballot Review
- Input label:
ballot - Output label:
pending
Description: all task force participants receive the assigment from the task force facilitators to vote on status of the test sample. The reviews carried out in Step 2 and Step 3 of this review process are used by the task force participants as a basis to cast a vote. The participants use test samples management tool (restricted access) to carry out the review and record the vote. After the voting period for a test sample ends, the balloting is closed by setting the status flag to pending. This indicates that the voting is complete, and the test sample is ready for a task force decision.
Step 5: Task Force Decision
- Input label:
pending - Output label:
holding,edits,fixes,rejected
Description: the test sample including its review history is brought to a teleconference agenda for a formal decision by the task force participants. If no objections or issues were raised through the ballot reviews, or if these were settled during the teleconference, then the status flag of the test sample is set to holding to indicate that the test sample has been accepted by the task force. The teleconference discussion may conclude an issue with the WCAG 2.0 Techniques rather than with the test sample itself. In this case the status flag is also set to holding, and a comment is supplied to indicate that the test sample needs further review by the WCAG Working Group. The issue is recorded using the test samples management tool (restricted access).
The task force may also decide that minor fixes are needed to better improve the test sample. In this case the test sample is (re-)assigned to a task force participant who notifies the author of the the review result and maintains a dialog to help the author fix the relevant issues. The status flag of the test sample is set to fixes until the author has carried out the necessary changes. It is then set to new to initiate another Content Review (as described in step 3 of this review process). In some cases insignificant editorial improvements need to be made but overall the test sample has been accepted by the task force. Also here the test sample is (re-)assigned to a task force participant who notifies the author of the review results and coordinates the improvements. The status flag is set to edits until the changes are made, in which case it is then switched to holding.
Finally, the task force may decide that the test sample is not acceptable and needs substantial fixes before it can be reconsidered. In this case the status flag is set to rejected, and a task force participant receives the assignment to notify the author of the review result. Note: it is left to the discretion of the task force facilitators to decide if the issues raised through the ballot reviews or during the teleconference consitute sufficient changes for Minor Edits, Minor Fixes, or Substantial Fixes.
Step 6: WCAG Working Group Decision
- Input label:
holding - Output label: currently undefined
Description: the objective of this task force is to pre-process test samples for the WCAG Working Group. However, since the WCAG Working Group has ownership of all WCAG 2.0 documents and materials, it is responsible for a final review and decision. This step in the review process is not further specified at the current time, it is an interface to the WCAG Working Group who will have their own processes for making formal decisions on the queued test samples.
Summary of Status Labels
- unconfirmed
- test sample has been received and is not yet reviewed
- new
- test sample successfully passed the structure review
- ballot
- test sample underwent a content review and is available for group balloting
- pending
- test sample underwent balloting and is available for a formal TF decision
- fixes
- test sample needs minor fixes and a renewed content review and balloting
- edits
- test sample needs minor edits before it can be queued for final decision
- holding
- test sample is queued for final decision by the WCAG Working Group
- rejected
- test sample has been rejected by the Task Force or the WCAG Working Group
- accepted
- test sample has been accepted by the WCAG Working Group
- deprecated
- test sample has been deprecated by the Task Force or the WCAG Working Group
Checklist for Structure Review
The following is a list of (objective) criteria for carrying out a review of the structure of test samples (see also the FAQ on Structure Reviews):
- Contact Information (available in Wiki)
- Name and e-mail address of the submitter are available (required);
- Organization on whose behalf the test sample was submitted (optional);
- General Checks (applicable to files in both metadata folder and testfiles folder)
- All the submitted files follow the naming convention and directory structure;
- All the files include correct links unless otherwise required by the test;
- All the files include correct spelling unless otherwise required by the test;
- Checks for Test Files (applicable to files in testfiles folder)
- All the files that are necessary to execute the test procedure have been submitted;
- All the files include valid markup unless otherwise required by the test;
- Checks for Metadata (applicable to files in metadata folder)
- The metadata files are valid TCDL 2.0;
- The metadata files use only elements and attributes specified in the Metadata Vocabulary;
- All the dates and other integer or literal values have the correct format (as specified in the Metadata Vocabulary);
- All static values (especially copyright notices) are included and accurate;
- All titles, descriptions, and other required fields are included and accurate;
- All identifiers (especially ID for techniques and rules) are used correctly;
- All structures such as rules, techniques, or pointers are used correctly;
Checklist for Content Review
The following is a list of (subjective) criteria for carrying out a review of the content of test samples:
- Clear Scope - the test sample must focus on a single technique and avoid combining several techniques in one test. Its scope and metadata must only reflect the technique that it addresses. In other words, although been focused on a single technique, a test sample may be affected by other techniques but, as they will not be addressed, they must not been described in the metadata;
- Neutral to Interpretation - the test sample must not be used to reinterpret the test procedure(s), for example by using the expertGuidance field to rephrase the test procedure(s) rather than to describe the specifics of executing the given test sample;
- Minimal & Complete - the test sample should be minimal yet complete and realistic. For example, it should not include additional content that is not necessary to carry out the test procedure(s) but at the same time the content should be sufficient to highlight a realistic scenario;
- Effective & Efficient - the test sample should facilitate the execution of the test procedure(s). For example, can the test sample be restructured or simplified to improve its effectiveness or efficiency?
- Consistent - the content of the test sample should be consistent with its description in the metadata and if applicable with the relevant technique(s) and test procedure(s). Especially the outcome of executing the test sample must be consistent with its description and intention.