This is a disposition of comments received on the Website Accessibility Conformance Evaluation Methodology (WCAG-EM), W3C First Public Working Draft 27 March 2012. This page is intended for internal discussion by the WCAG 2.0 Evaluation Methodology Task Force (Eval TF).
| ID | Commenter | Location | Status | Current Text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 5a | François Junique | 1.1, 2.1 and 3.5.c | Closed | Scores better defined | Size-independence can be tricky for a very large portal, modularity/truncatibility should be better supported, and aggregation of split scores should be explained (or to present it differently, scores - in 5c - should be defined in such way to be aggregatable according to a defined computation recipe) | Resolution: No change to this section (see also Comment #5b and Comment #5c on other sections of the document) Rationale: This comment is addressed in other sections of the document (see Comment #5c) Related discussion: Eval TF survey #4 (ID 5) and teleconference on 26 July Remark: section 1.1 Scope of this Document has been updated |
|
| 19 | Peter Korn | Abstract and/or Introduction | Closed | It would be helpful to note the 3rd bullet objective from http://www.w3.org/WAI/ER/2011/eval/eval-ws - that of "Aggregating individual results into an overall conformance statement; this includes defining approaches for assessing the relative impact on failures, potentially through incorporating tolerance metrics." | Resolution: No change; open an issue to further discuss "tolerance metrics" (see also Comment #15) Rationale: The Abstract and Introduction sections do not seem like the right place to address the aspect of aggregation and tolerance metrics. Aggregation has been addressed through sections 3.5.3 Step 5.c: Provide a Performance Score (optional) and 4.2 Evaluating a Large Website with Separate Parts; "tolerance metrics" needs more discussion Related discussion: Eval TF survey #4 (ID 19) and teleconference on 9 August (item 4) |
||
| 20 | Peter Korn | 1.2 | Closed | It may be useful to recognize some of the needs of these disparate audience members. For example, the four bullets under "Other audiences..." includes policy makers and project managers - who will also need a way to evaluate not only the extent to which a website / web application is meeting WCAG 2.0, but the extent to which support for WCAG 2.0 is improving from one evaluation to another - to assess the "progress toward done" that a website / web application is making. | It may be useful to recognize some of the needs of these disparate audience members. | Resolution: No change Rationale: Reporting and scoring have been addressed in 3.5 Step 5: Report the Evaluation Findings, and other aspects have been addressed in 4. Considerations for Particular Situations Related discussion: Eval TF survey #2 (ID 20) and teleconference on 24 May |
|
| 21 | Peter Korn | 1.4 | Closed | Add definiton of "Ancillary Functionality" | Terms and Definitions defines the term "Key Functionality". Might it be useful to also define "Ancillary Functionality"? | Resolution: No change Rationale: The term "Ancillary Functionality" is not used in the document Related discussion: teleconference on 24 May |
| ID | Commenter | Location | Status | Current Text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 27 | Aurélien Levy | 2.1 | Closed | Add Partial Conformance | The definition of the scope exclude the ability to remove part of a website. Why aren't you considering the same exception as in the WCAG 2.0 for partial conformance? | Resolution: Added the note: [[Note: According to WCAG 2.0 it is possible to make a "Statement of Partial Conformance" for third-party content and languages lacking accessibility support. However, these parts must still be included in the scope of evaluation. More information on making conformance claims is provided in section 3.5.2 Step 5.b: Provide an Accessibility Statement (Optional).]] Rationale: Partial conformance relates more the reporting than to the scope of the website / evaluation Related discussion: teleconference on 31 May Remark: Section 3.5.2 Step 5.b: Provide an Accessibility Statement (Optional) has been updated and provides further clarification on "partial conformance" |
|
| 5b | François Junique | 1.1, 2.1 and 3.5.c | Closed | Scores better defined | Size-independence can be tricky for a very large portal, modularity/truncatibility should be better supported, and aggregation of split scores should be explained (or to present it differently, scores - in 5c - should be defined in such way to be aggregatable according to a defined computation recipe) | Resolution: No change to this section (see also Comment #5b and Comment #5c on other sections of the document) Rationale: This comment is addressed in other sections of the document (see Comment #5c) Related discussion: Eval TF survey #4 (ID 5) and teleconference on 26 July Remark: section 2.1 Scope of Applicability has been updated |
| ID | Commenter | Location | Status | Current Text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 1 | Greg Gay | 3.1b | Closed | Add a “common elements” review as a goal of evaluation. This review will focus on issues associated with the accessibility of templates. | Given sites now-a-days are typically template driven, in addition to detailed and indepth reviews, I would suggest a “common elements” review be included as another type or goal of evaluation. Because the issues associated with template accessibility tend to differ from issues typically identified for content, I believe a review of templates should be separate from a review of content, and be handled apart from the “Representative Sample.” The vast majority of sites implement only 2 or 3 templates, so a review is typically inexpensive and attractive to organizations that want to address accessibility as best they can, but don’t have a big budget. | Resolution: No change Rationale: The focus of this methodology is on (full) website evaluation. The method proposed here would be a more limited evaluation like in the case of the preliminary evaluation inside the evaluation suite Related discussion: Eval TF survey #1 (ID 1) | |
| 2 | Greg Gay | 3.1b | Closed | "Basic Conformance" | "Basic Review" | Replace "Basic Conformance" and instead use "Basic Review" since conformance cannot be determined from a sampling of pages. (read more in comments from Greg Gay) | Resolution: Eval TF selected the term "Basic Report" (see also Comment #4, Comment #7, and Comment #38) Rationale: "Basic Review" sounds like it is not a full conformance evaluation Related discussion: Eval TF survey #1 (ID 2), Eval TF survey #2 (ID 4), Eval TF survey #3 (ID 7), mailing list thread on "Basic Conformance only after detailed evaluation?", teleconference on 28 June (item 3), teleconference on 5 July (item 5), and Eval TF survey #4 (ID 38) |
| 3 | Greg Gay | 3.1c | Closed | “Reviewer should also evaluate Level AAA….” | “In many situations it is useful…” | Though we do suggest a conformance target that realistically reflects the effort that might be required to reach a level of conformance (Typically Level AA), or we make target level recommendations based on legal requirements, we always provide a review of Level AAA items and encourage developers to address as many of them as is feasible, some of which are relatively easy to implement. Step 1.c reads as if review of AAA items is optional. I might suggest a little stronger language here. | Resolution: No change (see also Comment #30) Rationale: It is up to the evaluator and commissioner to decide the conformance level Related discussion: Eval TF survey #3 (ID 3), Eval TF survey #4 (ID 30), and teleconference on 5 July (item 3) Remark: The note in this section was made into a separate section 4.1 Initial Conformance Assessment of a Website to better clarify the context |
| 30 | Aurélien Levy | 3.1c | Closed | Add something here (or in the reporting section) about the conformance to each WCAG level | Making an AAA evaluation can be useful but the reporting must be adapted to give conformance for each WCAG Level. Furthermore using a AAA target will tend to decrease the conformity score and discourage the commissioner (or his development team) | Resolution: No change (see also Comment #3) Rationale: It is up to the evaluator and commissioner to decide the Level. Related discussion: Eval TF survey #3 (ID 3), Eval TF survey #4 (ID 30), and teleconference on 5 July (item 3) Remark: The note in this section was made into a separate section 4.1 Initial Conformance Assessment of a Website to better clarify the context | |
| 28 | Aurélien Levy | 3.1 | Closed | Sometimes defining the scope with the evaluation commissioner can be not as good as this section suggest it because most of the time he doesn't know anything about accessibility and about what is important for user to be evaluated. Furthermore, he can also include in the scope some specific pages where the development team has concentrated his effort to improve accessibility | Resolution: No change (see also Comment #33) Rationale: The involvement of the evaluation commissioner is not required. Also, the selection of pages is carried out in section 3.3 (step 3) without the involvement of the website owner Related discussion: Eval TF survey #2 (ID 28), Eval TF sruvey #4 (ID 33), and teleconference on 21 June Remark: Note in sections 3.1, 3.2, and 3.3 were edited to better clarify the role of the evaluator (eg. "the evaluator is responsible for an objective and thorough assessment") |
||
| 32 | Aurélien Levy | 3.2a | Closed | Maybe this step needs to be optional | Not every evaluation commissioner has access to or control over templates | Resolution: No change Rationale: Text currently says templates when available to the evaluator. Related discussion: Eval TF survey #4 (ID 32) |
|
| 33 | Aurélien Levy | 3.2b | Closed | Add the help of the commissioner | In this section I think we need to do it with the help of the commissioner | Resolution: Added a note: [[Involvement of the website owner and/or website developer can be helpful to help identify key web pages, functionality, technologies, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough assessment.]] (see also Comment #28) Rationale: External evaluators may not always be able to determine the key functionality of a website on their own. There is also related concern that evaluators need to ensure that there is no bias (eg. pre-selection of what to evaluate by the developer). Related discussion: Eval TF survey #2 (ID 28), Eval TF sruvey #4 (ID 33), and teleconference on 21 June Remark: Note in sections 3.1, 3.2, and 3.3 were edited to better clarify the role of the evaluator (eg. "the evaluator is responsible for an objective and thorough assessment") |
|
| 40 | Jonathan Avila | 3.2c | Closed | include states of pages | Under the section "identify the variety of page types" it is worthwhile to strength or include different states of pages such as error states for forms, dynamically added content, simulated dialogs or pop-ups, or page renderings based on different user settings and permissions including whether an accessibility enhancement settings are enabled. | Resolution: Added a description for web applications and expanded the listing of content types in Step 2.c (see also Comment #34 and Comment #A3) Rationale: Important aspect and should be included both in steps 2 and 3 (identify these types of content and then also include them in the samples Related discussion: Eval TF survey #2 (ID 40) and teleconference on 21 June |
|
| 22 | Peter Korn | 3.2d | Closed | Add the suggestion to note UI component sets & versions, if any, used (e.g. "JQuery UI version 1.9") | Particularly for web applications, much of the accessibility support is built into the UI component sets. | Resolution: Added the note: [[Where possible, it is often also useful to identify the libraries and components used to create the website, such as Dojo, JQuery, and others. Particularly for web applications, much of the accessibility support is built into these libraries and components, and evaluation can become more effective and efficient when these are identified.]] Rationale: Particularly for web applications, much of the accessibility support is built into the UI component sets. Because the evaluator may not be able in all cases to determine whether or which UI component sets & versions were used, this directive should be qualified Related discussion: Eval TF survey #4 (ID 22) |
|
| 35 | Aurélien Levy | 3.3b | Closed | give a minimum number of pages to include in the requirement 3 text | as this section is defining a minimum number of pages per features we can give a minimum number of pages to include in the requirement 3 text. Furthermore, I'm not really sure that taking two pages with the same kind of content (pages with data table for example) is useful if those pages are generated with the same template | Resolution: No change (see also Comment #35) Rationale: Much depends upon the website - size, complexity etc. Also concern that retesting other pages in the same template may be largely redundant except for a few SC (1.3.1 and 1.1.1, for example) Related discussion: Eval TF survey #4 (ID 35) |
|
| 34 | Aurélien Levy | 3.3 | Closed | consider website composed by an unique webpage (where part of the page is updated with ajax somewhere) | Resolution: Added description for web applications (see also Comment #40 and Comment #A3) Rationale: Important clarification Related discussion: Eval TF survey #4 (ID 34) and several other Eval TF discussions |
||
| 44 | Gian Wild | 3.3 | Closed | Add an option for testing all pages. | There are some automated testing tools available now that will test an entire web site. In addition to this, there are some companies (including our own) that complete manual and automated testing on *all* pages, and this should be represented in the evaluation methodology. It is true that template issues will be found by selecting a representative sample, however content issues - such as use of headings, coded field labels etc, can only be found if all pages are tested. Although this might be outside of the capability of most testers, it is something that should be covered in this evaluation methodology. | Resolution: Added description for small websites Rationale: Further clarify that evaluating all web pages does not contradict this methodology (see also Action-2 Related discussion: Eval TF survey #2 (ID 44), teleconference discussion on 28 June (item 10), and e-mail thread. |
|
| 41 | Jonathan Avila | 3.3 | Closed | Choose reused portions of pages | In "Step 3: Select a Representative Sample", it may be useful to choose portions of pages that are reused such as header sections, navigation structures, etc. While these portions are not full web pages and full web page conformance must general be met, capturing sections of the page can be useful in targeting violations, limiting repetitive evaluation of the same structure on each page, etc. An XPath or other identifier could be used to identify these sections within a given page. For all the pages in which this section or widget appears violations could then be applied as pattern violations thus saving time during the evaluation process. | Resolution: No Change Rationale: (1) we have a note in section 3 step 3 relating to that point; (2) a similar note was added in step 4; (3) updates have been made to step 5.a that further clarify when parts do not need to be evaluated again; (4) using an approach as proposed by the commenter to make the evaluation more efficient does not conflict with the methodology Related discussion: Eval TF survey #2 (ID 41) and teleconference discussion on 28 June (item 9) |
|
| 42 | Jonathan Avila | 3.4d | Closed | Step 4d calls for the archival of the pages and capturing of screenshots. This step is very important and is something that SSB does during the sample collection of the site. In addition, it may be important to record the steps (path) used to reach a page including any data entered, user rights, etc. | Resolution: Added [[Description of a path to locate the web page (especially if it is part of a process);]] in Step 4.d Rationale: It is not always possible to "archive" a page otherwise Related discussion: Eval TF survey #4 (ID 42) |
||
| 16 | François Junique | 3.5c | Closed | A special score should be available when only automatic measurements have been possible | Resolution: No change Rationale: WCAG-EM is only for full evaluation, not for tool measurement only. This is not covered in the current requirements and therefore not in the document. Related discussion: Eval TF survey #4 (ID 16) and teleconference on 9 August (item 7) |
||
| 43 | Jonathan Avila | 3.5c | Closed | add a performance metric and a unified process for its calculation | Regarding providing conformance scores Step 5c, the current wording indicates that a score of non-conformant is likely to occur and that score is important. This is indeed the case as many sites support WCAG but are not fully conformant to a given level. WCAG requires conformance on a per level basis with only a few exceptions. In reality the accessibility of the site may be experience differently to different user groups. For example, a site may be accessible to users with hearing impairments via captions, visual equivalents for sound, etc. but may not be accessible or fully conformant to WCAG success criteria related to access by persons who are blind. Knowing the relative score may assist users in making decisions to use a particular site based on their needs. While a goal of full conformance to all criteria is key -- conformance may be an evolving process within an organization and may change with updates to a site. Thus, a performance metric and a unified process for its calculation is important to this field. | Resolution: Updated section Step 5.c addresses this comment Rationale: This was planned work Related discussion: Eval TF survey #4 (ID 43) |
|
| 45 | Sharon Litchfield | 3 | Closed | I would like to see more discussion/specifics around the quantity of the sample pages selected to evaluate. | I think some categories could be applied to this such as: 1) small sample of website - less than 25% of pages; 2) medium sample of website - between 25 - 50% of pages; 3) large sample of website - between 50 - 75% of pages; 4) extensive sample of website - greater than 75% of pages; 5) full evaluation - 100% of pages. I think having some categories like this would allow greater transparency of how much (quantity of pages) of a website is actually being assessed and evaluated. Selecting a sample size has different implications for websites of different sizes, and the category selected may vary depending accordingly. | Resolution: No change (see also Comment #35) Rationale: Currently the outcome of the discussion was that the sample depends on the website Related discussion: Eval TF survey #1 (ID 45) |
|
| 5c | François Junique | 1.1, 2.1 and 3.5.c | Closed | Scores better defined | Size-independence can be tricky for a very large portal, modularity/truncatibility should be better supported, and aggregation of split scores should be explained (or to present it differently, scores - in 5c - should be defined in such way to be aggregatable according to a defined computation recipe) | Resolution: Updated section 3.5.3 Step 5.c: Provide a Performance Score (optional) and added new section 4.2 Evaluating a Large Website with Separate Parts Rationale: Aggregation has been addressed in this section (see also Comment #5a and Comment #5b) Related discussion: Eval TF survey #4 (ID 5) and teleconference on 26 July. Remark: sections 1.1 Scope of this Document and 2.1 Scope of Applicability have also been updated. |
| ID | Commenter | Location | Status | Current Text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 17 | François Junique | 4 | Closed | what is this? If we were to have one day a EU legislation of web-accessibility based on the usual EU "new approach" on regulation, we would have some mandatory essential requirements and a presumption of conformity would be achieved by conforming with the M376-created standard using this WCAG-EM (if part of it...). Already the reliance for testing this conforming on agilely evolving techniques docs is tricky but if in addition the WCAG-EM is only one option for checking the conforming to the std (not talking about conforming to the essential requirements), this would be even more opening fuzziness | Resolution: No change Rationale: This section had a misleading editor note; this section should be clearer now Related discussion: Eval TF survey #2 (ID 17) Remark: This section has been updated |
||
| 38 | Jonathan Avila | 4 | Closed | The goal of "Basic conformance" as stated in the EM appears to be offer an option to essentially self certify without having to perform real evaluation of the site. This level of conformance is likely to be incorrectly applied and would likely mean that the site would in fact not be conformant. In cases where some national legislation references WCAG this could allow someone to argue conformance without defensible claims. making an assumption on conformance, while often well intentioned, is not safe for sites that have not previously been evaluated. | The working group should consider allowing this level of conformance only for conformance claims that are made after updates are made to an already conforming site. | Resolution: No change Rationale: The term "Basic Conformance" was changed to "Basic Report" (see also Comment #2, Comment #4, and Comment #7) Related discussion: Eval TF survey #1 (ID 2), Eval TF survey #2 (ID 4), Eval TF survey #3 (ID 7), mailing list thread on "Basic Conformance only after detailed evaluation?", teleconference on 28 June (item 3), teleconference on 5 July (item 5), and Eval TF survey #4 (ID 38) |
| ID | Commenter | Location | Status | Current Text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 4 | Greg Gay | 5 | Closed | Change language on the requirements for a conformance statement | Only detailed reviews can potentially include a conformance statement, I believe there should be strong language around the issuance of conformance statements, and when they are appropriate to make. Inevitably suits will arise, and evaluators are putting their necks on the line if they start using the word “conformance.” | Resolution: No change Rationale: The term "Basic Conformance" was changed to "Basic Report" (see also Comment #2, Comment #7, and Comment #38) Related discussion: Eval TF survey #1 (ID 2), Eval TF survey #2 (ID 4), Eval TF survey #3 (ID 7), mailing list thread on "Basic Conformance only after detailed evaluation?", teleconference on 28 June (item 3), teleconference on 5 July (item 5), and Eval TF survey #4 (ID 38) |
|
| 18 | François Junique | 5 | Closed | seems to contradict some of the options in 3 step 1.b | Resolution: No change Rationale: This section had a misleading editor note; this section has been removed Related discussion: Eval TF survey #2 (ID 18) and teleconference on 28 June (item 7) |
| ID | Commenter | Location | Status | Current Text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 25 | Peter Korn | Appendix C | Closed | the examples all list "Person who did the evaluation". | This will not always be appropriate - e.g. in a self-assessment from a corporation of its own website or web application, this should often be the corporation's name and not an individual name. | Resolution: Replace "Person who did the evaluation" by "Evaluator" and link it to its definition Rationale: "Evaluator" is an organization or person Related discussion: Eval TF survey #1 (ID 25) and teleconference on 28 June (item 8) |
| ID | Commenter | Location | Status | Current Text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 6 | François Junique | 1.4 | Closed | "web-site-part" unclear | clarity | Resolution: Updated section 2.1 Scope of Applicability Rationale: Revised the approach rather than the term Related discussion: Eval TF survey #2 (ID 7) and teleconference on 5 July |
|
| 7 | François Junique | 2.3 and 3.1b | Closed | Note: more should be investigated in particular in relation to 2nd option in 3 step 1.b | Resolution: No change Rationale: All the goals in section 3.1.b are full evaluations; The term "Basic Conformance" was changed to "Basic Report" to help clarify that (see also Comment #2, Comment #4, and Comment #38) Related discussion: Eval TF survey #1 (ID 2), Eval TF survey #2 (ID 4), Eval TF survey #3 (ID 7), mailing list thread on "Basic Conformance only after detailed evaluation?", teleconference on 28 June (item 3), teleconference on 5 July (item 5), and Eval TF survey #4 (ID 38) |
||
| 8 | François Junique | 2.5 | Closed | should be clearer if the intention is to verify website accessibility or conformance to WCAG2.0 | clarity | Resolution: No change Rationale: Involving users is optional, but is strongly recommended for conformance evaluation as well Related discussion: Eval TF survey #3 (ID 8) and teleconference on 5 July (item 6) |
|
| 29 | Aurélien Levy | 3.1b | Closed | The detailed review and in-depth analysis spoke about "errors". Is it success criteria errors or specific dom node errors? | Clarity. In case it's dom node errors I don't think it's useful to have information about every identified errors including counting the number of errors and their locations within the web pages specially if the error is a repeatable one. | Resolution: Description of "Detailed Report" changed to [[Identifies whether a website conforms or not, and provides further information about the conformance of each evaluated web page. This type of evaluation is particularly useful for instructing web developers and for acquiring statistics for monitoring progress over time.]] Rationale: The wording was imprecise and misleading Related discussion: Eval TF survey #3 (ID 29) |
|
| 9 | François Junique | 3.1d | Closed | the implicit/interpretable-from-reading concept of for-specific-disability might be dangerous | clarity | Resolution: Section 3.1.4 Step 1.d: Define the Context of Website Use has been rewritten (see also Comment #A2) Rationale: The wording was imprecise and misleading Related discussion: Eval TF survey #3 (ID 9) and teleconference on 5 July (item 7) |
|
| 10 | François Junique | 3.1e and 4b | Closed | both essential for agile adaptation to technology evolution and rather tricky as the techniques are said to unstable non-standardisable docs, nevertheless not all that clear regarding how to handle the passing and failing ones (plus the many "other advices" type ones) | clarity | Resolution: Sections 3.1.5 Step 1.e: Define the Techniques to be Used (Optional) and 3.4.2 Step 4.b: Use WCAG 2.0 Techniques Where Possible have been edited Rationale: Explaining the use of techniques and their relationship to conformance is essential Related discussion: Eval TF survey #3 (ID 10), teleconference on 5 July (item 8), and summary of various email threads |
|
| 31 | Aurélien Levy | 3.2a | Closed | Definition of "template" | Clarity. Is it the CMS template without any content ? If it's corresponding to CMS template, sometimes the commissioner can't have access to this kind of information (proprietary CMS, CMS without template mechanism). | Resolution: No change Rationale: The text states "templates available to the evaluator" Related discussion: Eval TF survey #3 (ID 31) |
|
| 23 | Peter Korn | 3.2d | Closed | It would also be appropriate to include Java in the list of auxiliary web technologies. | Clarity | Resolution: Add Java to the list of auxiliary web technologies in 3.2.d. Rationale: - Related discussion: Eval TF survey #1 (ID 23) |
|
| 11 | François Junique | 3.2 and 3.3 | Closed | not so clear how relevant for each of the 3 options in 3 step 1.b | clarity | Resolution: Section 3.5.1 Step 5.a: Provide Documentation for Each Step has been updated Rationale: Clarifies the relevance of these three goals Related discussion: Eval TF survey #4 (ID 11) |
|
| 12 | François Junique | 3.4 | Closed | use-case: what is this: disability-specific | clarity | Resolution: Section 3.4.1 Step 4.a: Check for the Broadest Variety of Use Cases has been updated Rationale: Clarifies the concept of "use-cases" Related discussion: Eval TF survey #4 (ID 12) |
|
| 13 | François Junique | 3.5b | Closed | the term "accessibility statement" seems to often used for some thing wide than a declaration of conformity and more for user informationt | clarity | Resolution: Section 3.5.2 Step 5.b: Provide an Accessibility Statement (optional) has been updated Rationale: Clarifies what is meant by "accessibility statement" (and the difference to WCAG 2.0 conformance claims) Related discussion: Eval TF survey #4 (ID 13) |
|
| 14 | François Junique | 3.5b and 3.5c | Closed | it might be beneficial to use a richer score than a "one-figure" score, e.g. for each technology used a matrix of wcag2.0-principles versus use-case scoring | Resolution: Added note for document reviewers (the public): [[Eval TF is particularly looking for feedback on this section]]; open an issue to continue discussing "scores" Rationale: from the RDWG symposium on Web Accessibility Metrics it seems that the more complex the metric, the less reliable and useful it becomes Related discussion: Eval TF survey #4 (ID 14) Remark: Section 3.5.3 Step 5.c: Provide a Performance Score (optional) has been updated |
||
| 15 | François Junique | 3.5b and 3.5c | Closed | where will be defined for 1st option of 3 step 1.b, what are the tolerable number of errors to still conform | clarity | Resolution: No change; open an issue to further discuss "tolerance metrics" (see also Comment #19) Rationale: The methodology evaluates conformance to WCAG 2.0, which does not provide a concept for tolerance Related discussion: Eval TF survey #4 (ID 15) and teleconference on 9 August (item 6) |
|
| 26 | Peter Korn | Appendix C | Closed | more example information would be helpful with respect to what the results should look like (in the penultimate bullet "Results: per guidelines, checkpoint...") | Clarity | Resolution: Added editor note: [[this section may change in later drafts depending on how the other sections evolve]] Rationale: We will work on this section in more detail in later editor drafts Related discussion: Eval TF survey #3 (ID 26) |
| ID | Commenter | Location | Status | Current Text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 24 | Peter Korn | 3.2d | Closed | "auxillary" | "auxiliary" | Typo | Resolution: Change Rationale: Typo Related discussion: Eval TF survey #3 (ID 24) |
| ID | Commenter | Location | Status | Current Text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 36 | François Junique | Closed | I don’t see where you plan to discuss handling/evaluating the different versions of a web site (based on different available style-sheets or dynamically generated content (UA or server side - non only js based) based on user profile/preference or device type or natural-language. | Resolution: Added description for Website in Multiple Versions in the updated section 2.1 Scope of Applicability Rationale: - Related discussion: Eval TF survey #4 (ID 36) |
|||
| 37 | Jonathan Avila | Closed | The scope of the methodology is indicated to apply to full websites. While we applaud this effort and agree that sites should not be cherry-picked for accessible pages or features -- it may not always be practical for entire websites. Thus, the idea of applying the scope to departments or smaller portions of a site will likely be important. For example, a particular purpose of the site such as the online forum or reservation section of the site and whole pages within may be targets of the evaluation. Similarly allowing websites to report on conformance while still have non-accessible third party content such as social media links/log-ins/ads is a good idea. While in principle third party content should also conform sites should be credited with their efforts to be conformant of what they have control over. | Resolution: Added note: [[Note: According to WCAG 2.0 it is possible to make a "Statement of Partial Conformance" for third-party content and languages lacking accessibility support. However, these parts must still be included in the scope of evaluation. More information on making conformance claims is provided in section 3.5.2 Step 5.b: Provide an Accessibility Statement (Optional).]] to section 2.1 Scope of Applicability, and section 3.5.2 Step 5.b: Provide an Accessibility Statement (Optional) has been updated and provides further clarification on "partial conformance" Rationale: Partial conformance relates more the reporting than to the scope of the website / evaluation Related discussion: teleconference on 31 May |
|||
| 39 | Jonathan Avila | Closed | The goal of "Basic conformance" as stated in the EM appears to be to offer an option to essentially self certify without having to perform real evaluation of the site. This level of conformance is likely to be incorrectly applied and would likely mean that the site would in fact not be conformant. In cases where some national legislation references WCAG this could allow someone to argue conformance without defensible claims. making an assumption on conformance, while often well intentioned, is not safe for sites that have not previously been evaluated. The working group should consider allowing this level of conformance only for conformance claims that are made after updates are made to an already conforming site. | This is a repetition of Comment #38 | |||
| 47 | Greg Gay | Appendix C | Closed | Dates are also important in a conformance statement. | Resolution: Added date to sections 3.5.1 Step 5.a: Provide Documentation for Each Step and 3.5.2 Step 5.b: Provide an Accessibility Statement (optional) Rationale: - Related discussion: Eval TF survey #1 (ID 47) |
||
| 48 | Alistair Garrison | 3.1.e | Closed | Step 1.e: Define the Techniques to be used | Could we consider making this step non-optional? |
1. We really need to check their implementation of the techniques (W3C, their own code of best practice or whatever) they say they use. 2. Testing seems only to be really replicable if we know what the techniques were they said they implemented - otherwise, two different teams could easily get two different results based on the cases above. |
Resolution: No change, keep optional Rationale: In many circumstances techniques may not be able to be provided and evaluation would have to proceed with out them. Techniques may not be available to the evaluator so that this clause cannot be required. Note also that techniques are not required for WCAG 2.0 conformance. Related discussion: Eval TF survey #4 (ID 48), summary of various email threads, and teleconference on 9 August (item 8) Remark: See also Comment #10 |
The following are open comments from the 6 March 2012 Editors Dradt. The full disposition of comment for this pre-publication draft is available at http://www.w3.org/WAI/ER/conformance/comments-20120306
| ID | Commenter | Location | Status | Current Text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| A1 | Amy Chen | 3.1.d | Closed | Take out "minimum set of web browsers and assistive technology to evaluate for shall be defined" and take out paragraph "It is important to also define the minimum set of web browsers and assistive technology to evaluate for." |
This should be in the product documentation what browsers and AT is recommended, not in the conformance documentation. It might be good to list what was used to test as an optional step, but not required. Conformance should be agnostic to the web browser and AT used. |
Resolution: Open an issue on "defining tools support" Rationale: This is part of defining "accessibility support" Related discussion: teleconference on 22 March |
|
| A2 | Samuel Sirois (in Eval TF Survey) | 3.1.d | Closed | "The context of website use shall not be used to distort or go around the Scope of Applicability of this methodology as defined in section 2.1" |
With this comment, I wish only to make sure that someone could not use the context of website use to reduce the scope of users to which a website is created to "sighted users, fluent in the language and culture in which the website has been written that uses mice on desktop computers and accessing the website with a specific version of a browser, on a specified operating system |
Resolution: Section 3.1.4 Step 1.d: Define the Context of Website Use has been rewritten (see also Comment #9) Rationale: The wording was imprecise and misleading Related discussion: Eval TF survey #3 (ID 9) and teleconference on 5 July (item 7) |
|
| A3 | Samuel Martín | 5 | Closed | I would also add that this methodology is mainly oriented to evaluate web sites under a coherent control... Sorry for the wording, maybe I'll explain myself showing what it does not (fully) cover. Some rich web applications allow visualizing different documents or content instances. Conformance with WCAG 2.0 may be assessed using this methodology, but it would not be a comprehensive evaluation, as the web application would also need to be subject to ATAG. Likewise, some rich web applications allow users ("prosumers", as they are both producers and consumers) create and publish their own content. These applications should be subject in addition to UAAG. |
Resolution: Added a description for web applications and expanded the listing of content types in Step 2.c (see also Comment #34 and Comment #40) Rationale: Important to better clarify the applicability of the methodology to web applications Related discussion: Eval TF survey #2 (ID 40) and teleconference on 21 June |
||
| A4 | Kerstin Probiesch | 1.1 | Closed | Because those goodness criteria are defined and internationally agreed in the scientific community already it would be at this very status of the document enough for me to address them in 1.1 Scope of this Document in a very general way (but with explicit mentioning them) and explain or work them out clearer and more specifically in future versions of our methodology (in Eval TF Survey) |
Missing explicit statement about the main goodness criteria for evaluations: Objectivity, Reliability, Validity |
Resolution: Open issue to discuss "how to better address objectivity and equivalent results in the WCAG-EM" Rationale: Lack of clarity of the added value versus the added complexity Related discussion: teleconference on 22 March |
|
| A5 | Detlev Fischer (supersedes Eval TF Survey) | Req 1.b | Closed | "In the case of "Other" the exact goal must be clearly defined." | This forth option should be scrapped or one or two examples be given, as for the other three options |
It is not clear what "Other" might be and when this would apply,, so it should be explained with examples |
Resolution: Removed in updated revision of section 3.1.2 Step 1.b: Define the Goal of the Evaluation Rationale: Was not used in later sections of the document nor was it well-defined |
| A6 | Emmanuelle Gutiérrez y Restrepo (in Eval TF Survey) | 1.1 Scope | Closed | audit the selected sample | evaluate the selected sample | Disambiguation and clarity for future translations. Or put it in the list of terms and definitions. Maybe can be more clear the term: "Conformity assessment" [http://en.wikipedia.org/wiki/Conformity_assessment] | Resolution: Open issue to further pursue the usage of the terms "audit", "assessment", and "evaluation" (potentially with EOWG at our upcoming face-to-face meeting in Lyon) Rationale: We use all three terms interchangeably throughout the document which may be confusing; on the other hand it is used to differentiate different aspects of "testing/checking" |
| A7 | Amy Chen | Requirement 2.c | Closed | Need to provide an example | Resolution: Close the comment for now Rationale: Sufficiently clear, but also a potential topic to discuss later |