W3C logo Web Accessibility Initiative (WAI) logo > EOWG Home Page > Selecting Evaluation Tools

Changelog: Selecting Web Accessibility Evaluation and Repair Tools

This page records change requests and changes made to the draft document Selecting Web Accessibility Evaluation Tools, which is part of the Evaluating Web Sites for Accessibility Resource Suite. Please send additions or corrections to wai-eo-editors@w3.org.

Last updated on $Date: 2005/09/16 19:09:44 $ by $Author: shadi $

About "Selecting Evaluation Tools" Document

Audience for this document


  1. Introduce the different types of evaluation tools and define terminology
  2. Highlight the tasks of Web developers and how evaluation tools can assist them
  3. Describe (existing) tool functionality which Web developers may be looking for

Open Review Notes

Notes as of 9 September, 2005

From discussions on the version of 7 September, 2005.

[done] copyedit suggestions from Shawn Henry
[done] copyedit suggestions from Andrew Arch
[done] edits from EOWG discussion on 9 September, 2005

Notes on possiblities in the future:

Previous Changes to the Document

Notes as of 11 March, 2005

From discussions on the version of 10 March, 2005.

[done] Overall Comments
[done] fix in-page navigation (copy edit bug in one of the titles)
[done] Section 1
[done] What Evaluation Tools Can Do: second bullet: "performing checks" rather than "evaluating checks"
[done] introduction: first paragraph: add something like "review" or "processes" to "during evaluations"
[done] What Evaluation Tools Can Not Do: last sentence: consider "acurate" instead of "precise"
[done] What Evaluation Tools Can Not Do: third sentence sounds a little too exclusive, consider something less negative
[done] Section 2
[done] consider adding h3 sub-headings to simplify scanning and readability
[done] Section 3
[done] consider a different word for "characteristics" if possible - editor discretion
[done] play around with titles of sub-headers to decrease the emphasis on categories
[done] In-Page Feedback Tools: "that may be perceived by some Web site users" sounds too confined
[done] In-Page Feedback Tools: last sentence: consider putting "reading sequence"
[done] Transformations Tools: mediate the limitations of automated accessibility checks - s/checks/checking
[done] Transformations Tools: remove "s" from title, replace word "mediate"
[done] Section 4
[done] Reliability: wrong order for false positives/negatives, consider "reporting wrong failures" as example for false positives
[done] Reliability: consider using just the examples for false positives/negatives and adding the specific technical terms in brackets
[done] Comments from Andrew Arch on 11 March, 2005

Notes as of 28 February 2005

From F2F discussions on the version of 24 February, 2005.

[done] Overall Comments
[done] fix in-page navigation to match link titles to the section titles
fixed links but shortened titles by taking out "Web accessibility" in each title.
[done] take out reference to "conformance" (evaluation tools can be used anytime during evaluation, not just for conformance evaluations)
trimmed document to talk about a more generic approach to web accessibility evaluations.
[done] take out "you" & "your" throughout the document
changed the tone of the document to more formal style, and therefore removed all occurences of "you"
[done] relate how eval tools can help during development throughout the document
ellaborated on the development cycle in a new section (number 2) and mentioned it consistently throughout the document
[done] play around with "using evaluation tools shortens the time in general, improves quality"
added both perspectives but did not over-emphasize "improving quality" to remain within scope.
[done] delete the word paradigm & check for jargon else where throughout the document
searched for and replaced all occurences of the word "paradigm" and chased other jargon too.
[done] take away emphasis on "Web developers" (e.g. in S3: configuration replace with userrs
where "developers" occurs it was explained, and otherwise the term "users" was prefered.
[done] Section 1
[done] "specifically, evaluation tools should perform the following functions..." but some just do one thing, e.g., transformation tools -- change from "should" to "can"
sentence reworded to how tools can assist users in evaluating sites.
[done] change "How Evaluation Tools Can Not Help You" to "What Evaluation Tools Cannot Do"
also the title of "How Evaluation Tools Can Help You" has been changed to match this.
[done] in how eval tools cannot help you P - replace "innacuracies" change to "limitations"
term "innacuracies" has been replaced with "limitations" throughout the document too.
[done] in how eval tools cannot help you P - use "false positives", "false negatives", "misleading results"
terms false positives and false negatives seemed a little too technical for this section.
[done] editors discretion: new section on "strategies for selecting" in introduction? ...put as early section example tasks - then build into the S2 - e.g., say non-coders might like transformation tools... if easy, build into S2: ... not have a specific section for user requirements, but buildin into Section 2 as examples where appropriate...
this paragraph grew to become a whole new section called "Considerations for Selecting Tools"
[done] Section 2

Note: a new section (number 2) has been introduced and this section is now section 3.

[done] emphasis on tools should be revised to match the section title "user interfaces"
section has been retitled to "Characteristics" and text has been trimmed to match.
[done] elaborate more on type of task & stage of development, match with "strategies for selecting"
elaborated the user tasks and development stages, and reflected on the previous section.
[done] revise headings to say more about the content, e.g., Report-generating Tools
all sub-headings in this section have been revised and made to a delimited list.
[done] in-page feedback tools do more than mark errors (e.g., WAVE) -- describe more
in-page feedback tools are now described more precisely and related to user tasks
[done] in-page feedback: second sentence: clarifying fix it so seems like tools are marking the page, not the developer
ambigous terms such as "editors" have been rephrased to better clarify meaning.
[done] page transformation: "experienced Web developers can effectively evaluate and improve the accessibility features of their Web sites." applies to all evaluation tools. take out.
sentence taken out and transformation tools have been decribed more precisely.
[done] in 4th subsection: flip first & second sentence, reword more precisely
transformation tools are now described more precisely and related to user tasks
[done] Section 3

Note: a new section (number 2) has been introduced and this section is now section 4.

[done] elaborate on different types of tasks & roles or users, relate to "strategies for selecting"
relied on ellaborate explanation in the new section (number 2) and refered to it consistently.
[done] make sure when you say different tools, it's clear different types of tools and not different ones of the same type -- consider putting this in one place & not throughout, best in "strategies section"
clarified this with examples in the next section (number 2) and used it consistently in the document.
[done] clarify whether tools or people, e.g., "plugin interfaces for editors" throughout section
rechecked all occurences and cleaned this up to my best knowledge.
[done] consider new title such as "features of Web accessibility evaluation tools"
this section is now call "features of Web accessibility evaluation tools".
[done] introduction: suggest users to ask vendors if answers are not readily available
sentence added in the introduction paragraph of this section
[done] accessibility - clarify more than just the UI, also documentation, reports, etc.
added example parts of a tool within brackets to clarify the scope of accessibility
[done] "for different user roles" > "for differrent reporting needs" & relate this to the tasks section earlier
rephrased item to clarify the meaning more and put under an overall category "configuration"
[done] consider question on "additional guidelines and tests" (e.g., 508, usability)
new question about accessibility guidelines and national policies was added.
[done] Automation: impacts the evaluation process, "which checkpoints" rather than how many, take out repair
has been summarized together with "manual checkoints" into "checkpoint coverage"
[done] collaboration: take out repair, reword sentence to explain EARL
the term "collaboration" has changed to "data export" and the item has been moved under "integration".
[done] "platform coverage" be more explicit about other platform considderations (e.g., hardware, network, configuration)
the term "coverage" has been removed and the item has been rephrased and moved under "integration"
[done] site coverage: explain term better
retitled to "Web site coverage" and put under "configuration". also ellaborated on the meaning.
[done] technology coverage -- reword, now too negative [see also minutes for suggestions]
retitled to "Web technology support" and rephrased some to clarify meaning and implications.
[done] change "user interface" to "usability". take suitable out of here
this item has been removed as it does not directly relate to features of evaluation tools.
[done] documentation should be split into of tool itself & linked resources
tool documentation has been removed as it does not directly relate to features of evaluation tools, and linked resources are implicit in the items "manual checkpoints" and "repair".
[done] precision: false positives, false negatives... . change title to "reliablity"
retitled item and added some clarification about the difficulty to measure this performance.
[done] repair - replace term "retorfit" with repair
term replaced and ellaborated more on how repair relates to the evaluation process.

Notes as of 4 February, 2005

From discussions on the version of 1 February, 2005.

[done] categories of tools in section 2 are confusing as they rather resemble general characteristics
[done] Comments from Sailesh Panchange on 3 February, 2005
  1. [done] user interfaces in section 3 does not sufficiently cover section 2
  2. [done] renamed 4th sub-heading in section 2 to "transformation tools"
  3. [done] reduced emphasis on the requirements of "experienced developers"
  4. [done] softened the tone so as not to seem to require eval tools to provide educational material
  5. [done] "Web sites" is an umbrella term for the content, edited first sentence of the intro
  6. [done] clarified that tools could help point out applicable checkpoints even if they can't evaluate them
[done] Comments from Chuck Letourneau on 3 February, 2005
  1. [done] clarified that tools can only execute automated checks to a certain degree
  2. [done] reformulated both the sentence as well as the paragraph as a whole
  3. [done] tools list will be updated (by ERT & EO) to better reflect this document
  4. [done] kept format for section 3, just reworded some entries slightly
[done] Comments from Sylvie Duchateau on 4 February, 2005
  1. [done] reviewed the usage of the terms "comprehensive" and "conformance"
  2. [done] clarified the sub-section about what tools can not do a little more
  3. [done] renamed 4th sub-heading in section 2 to "transformation tools"
  4. [done] tools list will be updated (by ERT & EO) to better reflect this document
[done] Comments from Roberto Castaldo on 4 February, 2005
  1. [done] clarified that tools can only execute automated checks to a certain degree
  2. [done] common criteria for section 2 is now the user interface of the tools
  3. [done] it is the accessibility barriers which we want to evaluate and repair, not the checkpoints
[done] Comments from Andrew Arch on 4 February, 2005
  1. [done] also wizard-based tools can sometimes check multiple pages
  2. [done] reworded 4th sub-section in section 2 to describe "transformation tools"
  3. [done] comment about a combination of tools is implicit and now explicitly also in section 2
  4. [done] clarified that tools could help point out applicable checkpoints even if they can't evaluate them
  5. [done] reduced the dependency on authoring tools and added browsers (as types of UAs) to the list
  6. [done] reworded question about site coverage to clarify that tools can support multiple pages
[done] Comments from Sailesh Panchang on 4 February, 2005
  1. [done] benefits and limitation sufficiently addresses this, this section has also been edited significantly
[done] Comments from Harvey Bingham on 4 February, 2005
  1. [done] clarified the sub-section about what tools can not do a little more
  2. [done] will add eventually when intro pages for these acronyms are developed
  3. [done] reworded question about repair in section 3 of the document
[done] Comments from Sailesh Panchang on 4 February, 2005
  1. [done] reduced emphasis on the requirements of "experienced developers"

Changes as of 28 January, 2005

From discussions on the version of 20 January, 2005.

[done] description of the question about "precision" too difficult to understand
reworded it in simpler language
[done] term "crawling" is difficult to understand, maybe better use "Site Coverage"
replace the term "crawling" with "Site Coverage"
[done] description of the education effect of manual tools is unclear and too difficult to understand
reworded and split this section into two types of tools, wizard-based and visual-feedback tools
[done] bullets in section 3 might be better formulated as questions to retain consistency
removed these bullets as they were too descriptive and added some more overall questions about tools
[done] describe the benefits of tools in the introduction document of the document
added two new sections, what tool can do and also what they can not to avoid the myths

Changes as of 17 December, 2004

From discussions on the version of 14 December, 2004.

[done] use the term Evaluation Tools rather than Assessment Tools to avoid confusion
all instances of the term Assessment Tools have been replaced
[done] highlight the benefits of tools more clearly (efficiency, automation, ...)
separated the tool categories and described their usage more in-depth
[done] more practical guidance on how to determine what to look for (shopping list)
added several questions to ask about specific things to look in a tool
[done] highlight the benefits of tools upfront, best in the introduction section
first sentence already addresses why tools should be used, other sections reiterate that
[done] tone may be a little too hesitant (several usage of the word "may")
reworded and made the overall tone more descriptive, especially in the last section
[done] stressing the usage of a single tool in the first paragraph may be misleading
removed this sentence and included more descriptions within the content
[done] ellaborate more on the meaning of transformation tools
redescribed transformation tools by explaining their usage rather than their definition
[done] links to the appropriate sections in the "existing tools" document
two documents are incompatible, need to wait until "existing tools" is revised
[done] section about the focus of the tools (text, images, multimedia, etc) missing
the tool categories are described more in-depth making this information more apparent
[done] section about server-side and client-side aspects missing
added question about the compatibility of evaluation tools in section 2 to address this
[done] headings in section 5 are difficult to understand on their own
section has been removed
[done] in section 5, add bullets beneath each sub-section with keywords of what the roles look for
section has been removed
[done] section 5 should be combined with section 3
section has been removed

Changes as of 15 October, 2004

From discussions on the version of 7 October, 2004.

[done] overall language usage is complex and difficult to understand
reviewed and edited for complex language and long sentences
[done] describe document structure and usage more clearly in the introduction section
introduction highlights the key aspects which are discussed more in-depth within the document
[done] retitle section "2. User Requirements" (and sub-sections) to reflect the user roles
this section has become sections 3 and 5 with more explanatory titles
[done] more practical information under each user roles about tool features to look for
examples have been added to explain the connection between requirements and features

Copyright © 2005 W3C (MIT, ERCIM, Keio ), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.