Disposition of Comments from WCAG WG Review

Resolution: Editor notes not intended for the public have been removed in the 27 August 2012 Editor Draft

Rationale: Some editor notes are intended for the public while others are intended for internal discussion; the latter will be removed

Resolution: Most removed in the 27 August 2012 Editor Draft. Rest will be removed before publication.

Rationale: These Editor Notes are intended for internal discussion and will be removed before publication

Resolution: Editorial changes made in the 4 September 2012 Editor Draft.

Rationale: Editorial refinements to provide better clarity

Note: See also Comment #31

It keeps pointing out to that document for more detail [...] if the external document is valuable because it's easier to organize and maintain.

Comment repeated after 30 August Editor draft.

Resolution: No change

Rationale: The W3C/WAI Evaluation Resource Suite provides complementary background and guidance for other types of evaluation (especially "Preliminary Reviews"). All references to that suite are in optional sections of the document or made in other ways that do not cause direct dependencies on these resources. W3C/WAI guarantees persistency of these resources though the detailed content of these resources may be updated with time to better reflect current paractices.

Resolution: Changes made by adding text conveying their information right by them and adding alt-attribute referring to text that is directly next to the image.

Rationale: Very relevant comment

Resolution: No change

Rationale: This hierarchy of requirements helps emphasize the required and optional steps of the evaluation procedure; Eval TF will consider editorial improvements to achieve this goal in future drafts

[...] use of "requirements" is a bit strong for a non-normative document [...] If we were to keep them as requirements, I would push back on a number of them as going too far for a resource that will be used in so many different scenarios

I find the use of the work "requirement" odd. It makes it feel like the focus is on whether you are applying this methodology correctly, rather than whether the methodology is effective for evaluating web sites.

It is an informative document but there is normative language "must, Requirements #" etc...

This is being released as a NOTE but it is written like a "STANDARD PROCEDURE" with requirements etc. it should be either changed to a standard and processed as such -- or rewritten to be more notelike (informative but not requiring anything)

Resolution: Changed to Methodology Requirement and made substantial changes to the Abstract and Introduction sections in the 4 September 2012 Editor Draft to better clarify that these are not normative, nor additional WCAG 2.0 requirements (as suggested during the WCAG WG Telco of 30 August 2012)

Rationale: Re-emphasizes that these requirements are only for the methodology and not additional WCAG 2.0 requirements

Note: See also Comment #12

Resolution: Thanks for the positive feedback

Rationale: N/A

Resolution: No change

Rationale: A provisional diagram is provided in intro of section 3; we will consider improvements to this diagram in future drafts; we welcome specific comments on refining it for future publications. Help is proposed by WCAG WG

Resolution: No change

Rationale: This guidance goes beyond the scope of this document, as it is applicable to many situations rather than post-development conformance evaluation alone; we recommend this guidance be developed as additional WCAG 2.0 guidance, ideally as part of Understanding WCAG 2.0 or similar supporting materials so that we can reference it from this document

Note: The editor note that "this section has been completed" is purely for internal review, to help Eval TF participants quickly identify the changes made between drafts; it will not remain in the document and is not to be interpreted as an absolute statement (hence "completed" rather than "complete")

Resolution: No change

Rationale: The corresponding sections always start with "applicable as per 3.1.3 Step 1.c. Define the Conformance Target"; repeating this at the end of the same paragraph adds unnecessary redundancy and complexity. A note on the test result "not applicable" is provided in section 3.4.1

Need to find some other term than "conformance"

I agree with Michael's discomfort with the use of the term conformance. If nothing else, there seems to be potential confusion between evaluating whether a web site conforms to WCAG via this methodology, vs whether the methodology used to evaluate the web site conforms to this methodology. If the focus is on the latter, the introductory material need to be very clear that this is one approach to evaluating conformance, rather than general guidance about conformance evaluation methodology. The current document title is ambiguous.

Resolution: Changed to "Application of this Methodology"

Rationale: Replace the word conformance.

Note: See also Comment #7

Resolution: We used definition from ATAG

Rationale: Important to seek alignment with other WAI resources

I can't review this whole document now, but I went looking for this and liked this section - we need more clear statements like this

Resolution: Thanks for the positive feedback

Rationale: N/A

Note: This section in 27 August 2012 Editor Draft has been further refined

Nit: why is Methodology capitalized in "People who want to use the Methodology for ..."

Resolution: Changed to lower case in the 27 August 2012 Editor Draft

Rationale: Typo

Resolution: No change

Rationale: As per section 1.1 Scope of this Document, this methodology "focuses on only one aspect of evaluation within the development process, which is the accessibility evaluation of an already existing website"

Note: Some editorial updates have been made to the Abstract and Introduction sections in the 27 August 2012 Editor Draft

Resolution: Changed to "Note: WCAG 2.0 defines "Statement of Partial Conformance" for individual web pages that are known not to conform with WCAG 2.0 due to third-party content and/or languages lacking accessibility support. Such web pages may not be excluded from the scope of evaluation according to this methodology. In some cases this means that the website as a whole does not conform with WCAG 2.0 due to only partially conforming web pages. Section 3.5 Step 5: Report the Evaluation Findings provides more guidance on reporting evaluation results and making accessibility statements for entire websites." in the 27 August 2012 Editor Draft and changed to only due to in 3 september 2012 Editor Draft

Rationale: Agree that this section needs further clarification. We mean that one cannot claim conformance over a website by out-scoping part of a website or a webpage, especially parts that are known not to conform. Many people misunderstand the concept of "statement of partial conformance" in this context hence we felt it needs to be explained in this section

3.3.2 Requirement 3b - what if there are not two distinct web pages for some feature?

Resolution: No change

Rationale: The text says "where applicable and available"

The note in 3.4.1 is confusing. "do not apply" is an ambiguous term.

Resolution: Changed to ""Note: According to WCAG 2.0, Success Criteria to which there is no matching content are deemed to have been satisfied. An outcome such as 'Not Applicable' may be used to denote the particular situation where Success Criteria were satisfied because no relevant content was applicable", and added link to the definition of "satisfy" in WCAG2.0 in the 27 August 2012 Editor Draft

Rationale: "do not apply" has been changed to "no matching content", and further refinements have been made in response to Eval TF Comment #22

3.4.1

Typos: "parts a website" -> "parts of a website" , "on on" -> "on"

Resolution: Changed as suggested in the 27 August 2012 Editor Draft

Rationale: Typos

3.4.2 Requirement 4.b seems like it is addressed to the developer rather than the evaluator. What are you trying to say here?

Resolution: Changes to this section in 27 August 2012 Editor Draft have been made have been in response to Eval TF Comment #29

Rationale: This section tries to explain the use of WCAG 2.0 techniques by evaluators. Making improvements to the guidance in Understanding WCAG 2.0 and How to Meet WCAG 2.0, in particular with regard to "common failures", would benefit readers of this document as well as many more situations.

3.4.3, Step 4.c: Accessibility Support applies to techniques, not technologies. Please edit this section title so it doesn't reinforce the idea that entire technologies should be rejected on accessibility support grounds.

Resolution: Changed to "accessibility features"; This section in 27 August 2012 Editor Draft has been further refined accordingly

Rationale: WCAG 2.0 refers to "Web content technology (or feature of a technology)" in the definition for "accessibility supported" and elsewhere.

3.4.4 Requirement 4.d seems both onerous and not well motivated. Where must such copies live? for how long must they be saved? etc. Particularly when a web site is updated to address problems found, do copies need to be made every time there is a change anywhere?

Resolution: No change

Rationale: This section is optional and provides guidance on good practice. Where the copies live and for how long is irrelevant to the methodology; it help an evaluator in case of disputes. Copies do not need to be updated because they only need to backup statements made by the evaluator at the time of evaluation. We may consider editorial improvements to this section in future drafts.

it isn't clear to me how useful this sort of score is. A few very minor errors can produce a low score by this metric.

Resolution: No change

Rationale: We are particularly looking for public comments and input on this section. This optional section is included due to numerous requests and as per scope of the Eval TF Work Statement

Requirement 5f: If the methodology requires machine-readable reports, it seems like there should be something in the methodology that uses these reports.

Resolution: No change

Rationale: This section is optional and provides guidance on good practice. Reports are the output of an evaluation methodology rather than input. We could decide to add more information for this in a later draft.

For the purposes of the methodology, what is the distinction between preliminary reviews and other reviews? It feels like there is some tensions between using the methodology as a way to assess the status of a web site and a way to report the status of a web site.

Resolution: No change

Rationale: This methodology is a way to assess the conformance of entire websites and report their status. Preliminary evaluations are coarse assessments to get an initial impression of the status. It specifically states "A preliminary review does not check every accessibility issue and will not catch all of the problems on a site. Thus the method described in this page is not sufficient to determine if a Web site conforms to Web accessibility guidelines."

Formatting issue: I'm finding many of the lists of steps very hard to read, since the names of the steps themselves contain complex numbers and colons. Perhaps such lists should be enumerated instead of inline.

Resolution: No change for this publication

Rationale: We plan to work with EOWG to refine the formatting for the next publication

Resolution: Changes to this section in 27 August 2012 Editor Draft have been made in response to Eval TF Comment #29

Rationale: This section tries to explain the use of WCAG 2.0 techniques by evaluators. Making improvements to the guidance in Understanding WCAG 2.0 and How to Meet WCAG 2.0, in particular with regard to "common failures", would benefit readers of this document as well as many more situations.

1. WCAG conformance is page by page -- which is specifically forbidden by this document/method.

2. Addition in Survey 30 August 2012: "Can you include something that states clearly that WCAG does *not require* conformance of whole websites -- but conformance can be for independent pages of independent processes?"

Resolution: Significant changes have been made to the Abstract and Introduction section in the 4 September 2012 Editor Draft (as suggested during the WCAG WG Telco of 30 August 2012)

Rationale: This methodology is intended to provide guidance on evaluating the conformance of entire websites rather than individual web pages (which is already covered by WCAG 2.0), and making WCAG 2.0 Conformance Claims on such (typcially large) collections of web pages (as defined by WCAG 2.0); clarification of this seem necessary to avoid confusion

sounds like very dangerous Concepts - and are often used to limit what needs to be made accessible [...] predetermination of what is important is dangerous.

Added comment to this one in the WCAG WG survey of the 27 August Editor Draft.

Resolution:

1) Changed "Key Functionality" to "Common Functionality" in Changes to 27 August 2012 Editor Draft

2) Open issue to further discuss the definition or use of "Common Functionality" for future publications

Rationale: It was not the intent to limit the usability of a website for people with disabilities but in many situations sampling is necessary to manage the evaluation of many web pages and theor states. Unfortunately usage-weighting alone seems insufficient to select representative sample. This issue needs more discussion to improve the concept in future drafts. Due to comments from WCAG survey of 27 August Editor Draft, we will discuss other better terminology after this draft.

Note: See also Comment #34

Resolution: Editorial changes made in the 4 September 2012 Editor Draft

Rationale: Editorial refinements to provide better clarity

Note: See also Comment #3

Resolution: No change

Rationale: It could several web pages or a web application in a single webpage (with a single URI). All parts of this web application are seen to be a single entity, defined as a website for the purpose of this methodology

Website in Multiple Versions
... Such website versions can be considered ... [CAN or SHOULD or ???]

Resolution: No change

Rationale: Can be. It is a choice. This could mean that a commissioner will ask the evaluator to look at only one version. That would then be possible

should be RETITLED as 3.2.1 Step 2.a: Identify COMMON Web Pages of the Website

Resolution: Changed in 27 August 2012 Editor Draft

Rationale: This is more inline with the intended meaning

Note: See also Comment #30

Resolution: No change; See also Comment #34

Rationale: For many websites, especially complex web applications, it is not manageable (nor necessary) to identify every function and corresponding code path; The intent of this section is to ensure adequately representative coverage of the functionality. This needs more discussion as per Comment #34.

Resolution: Thanks for the positive feedback

Rationale: N/A

Resolution:Add to review note in section 3.3 Step 3 about sampling that we are planning to add a random sampling approach to the next version

Rationale: We plan to discuss sampling in much more detail for the next version of the document.