w3c/wbs-design or
by mail to sysreq.
W3C
Site Navigation
Nearby
Results of Questionnaire Eval TF Review: 6 March 2012 Editor Draft of Website Accessibility Conformance Evaluation Methodology 1.0
The results of this questionnaire are available to anybody. In addition, answers are sent to the following email addresses: shadi@w3.org,E.Velleman@bartimeus.nl
This questionnaire was open from 2012-03-07 to 2012-03-14.
21 answers have been received.
Jump to results for question:
1. Support for publishing this draft
The current proposal is to publish this group-internal Editor Draft as a First Public Working Draft (FPWD) for additional input from the public.
Summary
| Choice | All responders |
|---|---|
| Results | |
| I support publishing this as FPWD as is | 11 |
| I support publishing this as FPWD; however, I suggest the changes in the comments section below (for editors' discretion) | 6 |
| I support publishing this as FPWD only with the changes indicated as [important to be addressed before publication] in the comments section below | 3 |
| I do not support publishing this as FPWD because of the comments indicated as [important to be addressed before publication] in the comments section below | 1 |
| I abstain (not vote) |
Details
| Responder | Support for publishing this draft |
|---|---|
| Roberto Scano | I support publishing this as FPWD as is |
| Martijn Houtepen | I support publishing this as FPWD as is |
| Vivienne Conway | I support publishing this as FPWD only with the changes indicated as [important to be addressed before publication] in the comments section below |
| Léonie Watson | I support publishing this as FPWD as is |
| Kathleen Wahlbin | I support publishing this as FPWD as is |
| Frederick Boland | I support publishing this as FPWD; however, I suggest the changes in the comments section below (for editors' discretion) |
| Detlev Fischer | I do not support publishing this as FPWD because of the comments indicated as [important to be addressed before publication] in the comments section below |
| Katie Haritos-Shea | I support publishing this as FPWD as is |
| Kerstin Probiesch | I support publishing this as FPWD only with the changes indicated as [important to be addressed before publication] in the comments section below |
| Alistair Garrison | I support publishing this as FPWD as is |
| Richard Warren | I support publishing this as FPWD; however, I suggest the changes in the comments section below (for editors' discretion) |
| Emmanuelle Gutiérrez y Restrepo | I support publishing this as FPWD; however, I suggest the changes in the comments section below (for editors' discretion) |
| Eric Velleman | I support publishing this as FPWD as is |
| Konstantinos Votis | I support publishing this as FPWD; however, I suggest the changes in the comments section below (for editors' discretion) |
| Michael Elledge | I support publishing this as FPWD as is |
| Wilco Fiers | I support publishing this as FPWD as is |
| Donald Raikes | I support publishing this as FPWD only with the changes indicated as [important to be addressed before publication] in the comments section below |
| Elizabeth Fong | I support publishing this as FPWD; however, I suggest the changes in the comments section below (for editors' discretion) |
| Sarah J Swierenga | I support publishing this as FPWD as is |
| Samuel Sirois | I support publishing this as FPWD; however, I suggest the changes in the comments section below (for editors' discretion) |
| Elle Waters | I support publishing this as FPWD as is |
2. Comments
(remember to include priority, location, suggested revision, and rationale for each comment.)
Details
| Responder | Comments |
|---|---|
| Roberto Scano | |
| Martijn Houtepen | |
| Vivienne Conway | comments have already been sent to the group by email |
| Léonie Watson | |
| Kathleen Wahlbin | |
| Frederick Boland | There are other techniques/best practices out there besides those officially published on W3C (for example, those being developed by the Web Accessibility Best Practices Working Group - link in: http://www.cita.illinois.edu/ - chaired by Jon Gunderson). Under Section 3 Step 1.e where it says "other techniques may be used too" should there be a link to an informative "resource" list (with disclaimer if appropriate) of other known techniques/best practices that could be considered as well, so that evaluators will have the most information possible to consider for evaluating websites? |
| Detlev Fischer | My comments: 1. Introduction "web applications and websites for mobile devices" Not sure this is beneficial. We don't know what UA will be used to view content. For touch devices, some WCAG criteria are not testable (keyboard functionality). Leave out? (The same applies to section 1.1 Scope of this Document - mention of mobile devices) 1.4 Terms and Definitions: "Elemental Web Pages": not quite clear what "elemental" conveys. Wouldn't be "Typical Web Pages" more understandable? 2.1 Scope of Applicability I read "the methodology always applies to a full website without exclusions or omissions of website parts", giving the example of a university website where it would not be permitted to exclude the library section. I find this unneccessarily restrictive. First of all, I think this is in contradiction to the WCAG section on conformance claims which states: "However, a conformance claim may be made to cover one page, a series of pages, or multiple related Web pages." http://www.w3.org/TR/WCAG/#conformance-claims But my real concern is not of a formal nature. I think this requirement will severely limit the applicability of the methodology and thereby, its practical usefulness and application in the field. The scope of the evaluation should make it crystal clear what is being evaluated and what isn't (by providing the URI's of sections), and any conformance claim placed on the site must make this equally clear. From practical evaluation cases, I know that there are often situations where legacy or third-party content cannot be made fully conformant - often in areas where lack of conformance may have only a minor impact. Typical are website that in some sections draw on unstrucured data entries from elsewhere as content, for example, violating SC 1.3.1 by having page breaks instead of p elements or some list content not marked up as list. In all other aspects, such web sites may be well designed, well structured, and very accessible overall. I a binary evaluation that is favoured in this group, this means that such a site would never even achieve level A conformance. The message sent is “Mend your ways and come back another time”. A good part of potential clients will simply think of the methodology as remote from reality and leave it alone. I think that setting the scope of conformance including, if applicable, the earmarking of sections that are excluded from the scope will mean that the methodology can be used in many cases where a conformance evaluation might otherwise be dropped simply because some part of the site cannot be brought into the scope of conformance. And this would be a real shame. I agree however that complete processes should be included fully (in the score, not necessarily with every step in the sample, see comment further down), so the aforementioned exemptions should not be parts of complete processes. 3. Evaluation Procedure Typo: "Requirement 1: he evaluation scope" Requirement 1.b: In the case of "Other" the exact goal must be clearly defined. Then there is nothing else on "other" and what it might be? Any idea? I think this forth option should be scrapped or one or two examples be given, as for the other three options "Detailed Review" and "In-Depth Analysis": should we not mention the option, independent of stating conformance per SC, of calculating an accessibility score as a means of giving a tangible measure of the distance to the goal of full conformance? I know that many clients will find such a score useful. The methodology may still abstain from defining how such a score would be calculated and label it 'merely informative'. Requirement 1e: "define the WCAG 2.0 Techniques that will be used to carry out the evaluation" Surely the techniques are used to implement content and functionality in a way that conformance to WCAG is ensured? The only way they are used in evaluation is by reference to the tests at the end of Techniques and especially Failures? This should perhaps be reworded. Requirement 3.a: All elemental web pages shall be part of the selected sample of web pages: Maybe one should qualify that if elemental pages are of the same design / based on the same template, selecting one of this type is sufficient? The statement of "all" seems of little value here since the definition referred to is rather vague. Requirement 3.b: At least two distinct web pages... I think this is unneccessarily specific and possibly misleading. On the one hand, if both instances selected are essentially the same the work would be redundant. If, on the other hand, just two web pages "from distinct types of web pages" are selected, this may be insufficient. At least for the reader / user of the methodology, the impression could arise that, say, for a site with seven distinct web page templates, selecting just two of the seven would be sufficient. And this would be bad. This is probably not the intention of this section, but it can easily be misunderstood. Essentially, sampling will need to fully reflect the complexity of the site under review. Any quantification is bound to lead to misunderstandings. Requirement 3.d: All web pages that are part of a process for each web page selected through Requirement 3.a, Requirement 3.b, and Requirement 3.c shall be part of the selected sample... I do not read into the WCAG statement that it is mandatory to evaluate each and every page or page state that is part of a complete process provided that the exploration of complete processes in Step 2 has concluded that the basic setup is accessible. If, for example, a step-by-step process uses the same template and just changes a few instructions and form elements (as in a test with yes/no options used to narrow down a problem space in some expert system), it might be enough to select a few key steps of the entire process. So the stipulation above seems too restrictive and cumbersome to me, especially if each page must be evaluated fully (and not just for selected elements). I think it will be quite obvious to readers that the methodology is not just a draft, but an unfinished draft. With step 4 and 5 just being editor notes at the moment, I wonder whether it would be wiser to spend a few more weeks fleshing out these sections before publishing the draft. I think it gives a bad impression of the sincerity of this task force if we publish things that are partially only sketched via editor notes... If there are other good reasons to publish this now in an incomplete state, I don't want to be in the way. But I'd much prefer having all sections fleshed out at least a little. So I am ready to reverse my recommendation ("Do not publish") if the majority feels it is advantageous to go ahead. |
| Katie Haritos-Shea | |
| Kerstin Probiesch | Priority strong: As already pointed out via Mail and in the survey of the WCAG WG. According to WCAG2: "Note: that all techniques are informative" (http://www.w3.org/TR/WCAG20-TECHS/intro.html) and the definition of "informative" in the glossary of WCAG2: "for information purposes and not required for conformance" (http://www.w3.org/TR/2008/REC-WCAG20-20081211/#informativedef). I propose to move Step 1e: Define the Techniques to be Used (Optional) and the text belonging to this point into somewhere in the Report Section. Guided with an explicit statement, that techniques are not the checkpoints. Reason: I fear that the current version will lead to confusion about the character of techniques, especially when they are combined with "Requirement" (even when it is optional). And I also think that it is not valide against WCAG2. Priority strong: As discussed in the list: When speaking about evaluation methodologies in general there are the three main goodness criteria of evaluation methodologies: Reliability, Objectivity and Validity. In the moment they are not explicitly adressed somewhere in the text. Proposal: Because those goodness criteria are defined and internationally agreed in the scientific community already it would be at this very status of the document enough for me to address them in 1.1 Scope of this Document in a very general way (but with explicit mentioning them) and explain or work them out clearer and more specifically in future versions of our methodology. There are some minor issues but I can live with them in this very early current status of the document and look forward for future versions. |
| Alistair Garrison | |
| Richard Warren | Priority:Mild. Location: Step 4. I would be a lot happier if we had time to flesh out step 4 (Audit) as this is such an important section which will also impact on other areas. I believe that work on section 4 will produce some refinement to other sections. However, so long as this document is recognised as a first, incomplete draft and therefore subject to much future revision I support publication at this time. Richard |
| Emmanuelle Gutiérrez y Restrepo | • priority: strong suggestion for editors' discretion • location: 1.1 Scope • current wording: audit the selected sample • suggested revision: evaluate the selected sample • rationale: Disambiguation and clarity for future translations. Or put it in the list of terms and definitions. Maybe can be more clear the term: “Conformity assessment” [http://en.wikipedia.org/wiki/Conformity_assessment] • priority: medium suggestion for editors' discretion • location: 1.4 Terms and definitions & Step 2.a & Step 3.a • current wording: Elemental Webpages • suggested revision: Essential Webpages or Inherent webpages • rationale: Disambiguation • priority: mild suggestion for editors' discretion • location: 1.4 Terms and definitions • current wording: Website developer • suggested revision: Website author • rationale: Disambiguation. Is not the same a content author than a web developer. • priority: mild suggestion for editors' discretion • location: 3. Evaluation procedure • current wording: Subsections numeration. • suggested revision: Each subsection should be numerated, as for example: 3.1 Step 1: Define the Evaluation Scope • rationale: Order and clarity. • priority: strong suggestion for editors' discretion • location: Step 1. Requirement 1 • current wording: he evaluation • suggested revision: the evaluation • rationale: legibility |
| Eric Velleman | |
| Konstantinos Votis | in section 1.2 target audience: Web accessbility consultants and evaluation services i think it would be better the evaluation service providers" because in this text we want to define the audience. Also the same for the monitoring and benchmarking activities it could be "organisations involved in Web accessibility.....activities. For the text: Policy makers, project managers, and other decision makers who need a standard i would propose the: Policy makers, project managers, and other decision makers who need a standardized way for performing accessbility evaluations - section 1.3 - Evaluating Websites for Accessibility - A multi-page resource suite that outlines different approaches for evaluating websites for accessibility : I am note sure about the definition for the text:Evaluating Websites for Accessibility---Is this a multi-page resource suite? - Section 1.4: become some of the terms are also used before the section 1.4 i would propose to be included in beginning of the methodology - section 2.2: make linkage of this text with section 1.3 - section 2.3 text automatically check: i would propose to include also semi-automatically check -requirement 1.b: the large scale evaluations as described in the basic conformance it could be also included to detailed review and in depth analysis. Also, in the detailed review you are talking about Web pages. I suggest to put Web sites or applications -requirement 1.d: i am little bit confused with the described requirement. Why do we need to define who uses the Website?I am not sure if we need it - step 1.e: I am not so sure about this step..This could be modified for including also techniques that can be assessed automatically and techniques that could be tested by a manual way - Regarding the selection of a representative template i ahve the following comment:What about Websites that are being developed through templates and these templates are the same for the most of the pages for these Websites? - As a general comment: i think that as a next step we have to create some templates and some examples for providing a clearer understanding to the evaluator who has no experience |
| Michael Elledge | I think it's great--I'm looking forward to others' comments. |
| Wilco Fiers | |
| Donald Raikes | 1.4: * Elemental Web Pages -> Common Web Pages (medium priority) Elemental makes it sound like it is a subpart of another page or is included in another page. * * Key Functionality (medium) Should include registering for a website as an example of key functionality 1.4 and 2.1: * website (high) Still needs more clarification especially for very large websites. For example: If a university president requests an evaluation of the university website (http://arizona.edu). According to 2.1 all departmental webpages linked to the main Arizona.edu page should be included in the possible sample pages even though the main Arizona webpage is “owned” by the administration and the various departmental webpages are “owned” by the respective departments, but because of the non-exclusion clause which says that you cannot say a site conforms except for this section then even if the library or optical sciences department sub site does nto conform, then the entire university site does not conform. |
| Elizabeth Fong | |
| Sarah J Swierenga | |
| Samuel Sirois | Comment number 1: Priority: Mild suggestion for editor's discretion Location: Section 1.4 Terms and Definitions under Key Functionality Current wording: Primary functionality of a website including tasks that users of a website carry out to perform functionality. Suggested revision: Primary functionality of a website including tasks that users of a website carry out to perform functionality. A single task may need a set of web pages to be completed. Rationale: I believe that we should make sure that the audience of the methodology understands that a task might need more than one web page to be complete. A key functionality should be audited as so (in it's fullness). Comment number 2: Priority: Medium suggestion for editor's discretion Location: Section 2.1 Scope of Applicability, fourth paragraph (Exception) Current wording: ...such as to the public and restricted area of a website or the front-end and back-end of a web-based tool, provided that this scope... Suggested revision: ...such as to the public and restricted area of a website, provided that this scope... Rationale: The wording "front-end" and "back-end" has a specific meaning for developers, which is not the meaning we wish to use in this methodology (as I understand it). For a developer, the front-end is the HTML markup + JavaScript to handle behavior. The back-end is the persistent layer (MySQL, PostgreSQL) and the server-side algorithms, generally written in PHP, Ruby, Python, etc. Because that "front-end and back-end" part didn't bring more to what the phrase already communicated, I simply removed it without replacing that part by anything. Copy edit: Location: Section 3. Evaluation Procedure, inside the emphasis box. Current wording: Requirement 1: he evaluation scope shall be defined according to... Suggested revision: Requirement 1: The evaluation scope shall be defined according to... Comment number 3: Priority: Mild suggestion for editor's discretion Location: Section 3. Evaluation Procedure, under Step 1: Define the Evaluation Scope, first paragraph Current wording: ...carried out together with the evaluation commissioner (who may be the website owner)... Suggested revision: ...carried out together with the evaluation commissionner... Rationale: Because it is already stipulated inside the definition of an evaluation commissionner that the evaluation commissionner may be the website owner, I think it is only redundant to include that precision again inside the methodology. Comment number 4: Priority: Strong suggestion for editor's discretion Location: Section 3. Evaluation Procedure, under Step 1.d: Define the Context of Website Use Current wording: There is no "current wording". This comment is about adding an idea to step 1.d. Suggested revision: The context of website use shall not be used to distort or go around the Scope of Applicability of this methodology as defined in section 2.1. Rationale: With this comment, I wish only to make sure that someone could not use the context of website use to reduce the scope of users to which a website is created to "sighted users, fluent in the language and culture in which the website has been written that uses mice on desktop computers and accessing the website with a specific version of a browser, on a specified operating system. Comment number 5: Priority: Strong suggestion for editor's discretion Location: Section 3. Evaluation Procedure, under Step 1.e: Define the Techniques to be Used (optional) Current wording: Requirement 1.e: The WCAG 2.0 techniques to be used during the evaluation should be specified. Suggested revision: Requirement 1.e: The techniques to be used during the evaluation should be specified. Rationale: Because the methodology do not depend on Techniques for WCAG 2.0, we shall talk about "techniques" and not "WCAG 2.0 techniques". Comment number 6: Priority: Mild suggestion for editor's discretion Location: Section 3. Evaluation Procedure, under Step 2.b: Identify Key Functionnality of the Website Current wording: Every sentence is written with key functionnality in it's singular form. Suggested revision: Every sentence shall be written with key functionnality in it's plural form. Rationale: A website might have more than one key functionnality. It might only be the fact that I am not totally fluent in English, but key functionnality seems to me as if it is in a singular form and then imposes my brain to search for The One key fonctionnality of a website, which is to restrictive in my belief. Comment number 7: Priority: Mild suggestion for editor's discretion Location: Section 3. Evaluation Procedure, under Step 2.d: Identify Technologies Used in the Website Current wording: Technologies Used in the Website Suggested revision: Technologies Used to display information to the user in the Website Rationale: Because developers (and other technologists) are part of the target audience of the methodology, we shall make clear that when we talk about technologies, we are talking about "front-end" technologies or, in a less geeky gobbledygook way, technologies used to display information to the user. With that precision, a technologist will understand that we are not talking about server-side technologies used to distribute the information. Comment number 8: Priority: Mild suggestion for editor's discretion Location: Section 3. Evaluation Procedure, under Step 2. and Step 3. Current wording: HTML, CSS, Flash, JavaScript, PDF, Silverlight and WAI-ARIA Suggested revision: HTML, CSS, Flash, JavaScript, PDF, Silverlight, applets and WAI-ARIA Rationale: Because we have been so precise in the enumeration, we shall include the forgotten "popular" one inside that list of elements. |
| Elle Waters |
More details on responses
- Roberto Scano: last responded on 8, March 2012 at 16:28 (UTC)
- Martijn Houtepen: last responded on 9, March 2012 at 08:36 (UTC)
- Vivienne Conway: last responded on 9, March 2012 at 23:25 (UTC)
- Léonie Watson: last responded on 12, March 2012 at 11:58 (UTC)
- Kathleen Wahlbin: last responded on 12, March 2012 at 15:33 (UTC)
- Frederick Boland: last responded on 12, March 2012 at 15:36 (UTC)
- Detlev Fischer: last responded on 12, March 2012 at 17:21 (UTC)
- Katie Haritos-Shea: last responded on 13, March 2012 at 00:24 (UTC)
- Kerstin Probiesch: last responded on 14, March 2012 at 08:42 (UTC)
- Alistair Garrison: last responded on 14, March 2012 at 09:22 (UTC)
- Richard Warren: last responded on 14, March 2012 at 11:02 (UTC)
- Emmanuelle Gutiérrez y Restrepo: last responded on 14, March 2012 at 13:20 (UTC)
- Eric Velleman: last responded on 14, March 2012 at 13:23 (UTC)
- Konstantinos Votis: last responded on 14, March 2012 at 13:56 (UTC)
- Michael Elledge: last responded on 14, March 2012 at 16:11 (UTC)
- Wilco Fiers: last responded on 14, March 2012 at 16:16 (UTC)
- Donald Raikes: last responded on 14, March 2012 at 17:32 (UTC)
- Elizabeth Fong: last responded on 14, March 2012 at 17:45 (UTC)
- Sarah J Swierenga: last responded on 15, March 2012 at 01:03 (UTC)
- Samuel Sirois: last responded on 15, March 2012 at 04:07 (UTC)
- Elle Waters: last responded on 15, March 2012 at 04:26 (UTC)
Everybody has responded to this questionnaire.
Compact view of the results / list of email addresses of the responders
WBS home / Questionnaires / WG questionnaires / Answer this questionnaire
Report issues
on GitHub project