Evaluation tools/Revision 2021

From Education & Outreach

Requirements Analysis

2021 revision of the List of Web Accessibility Evaluation Tools through the EC-funded WAI-CooP Project.

Objective

The objective is to redesign the list of web accessibility evaluation tools. This includes matching the page to the latest version of the W3C style, as well as evaluating and adapting features and content of the page to match tool user and vendor needs. The list can be filtered using different features to facilitate finding tools that match particular needs or particular coverage of the WCAG guidelines. The tools and their descriptions are provided by tool developers, vendors, or others. The tools and the features for filtering are listed with no quality rating.

Important disclaimer: W3C does not endorse specific vendor products. Inclusion of products in this list does not indicate endorsement by W3C. Products and search criteria are listed with no quality rating. Tool descriptions, search criteria, and other information in this database is provided by tool developers, vendors, or others. W3C does not verify the accuracy of the information. The list is not a review of evaluation tools, nor a complete or definitive list of all tools. The information can change at any time.

Audience

Primary audience of this resource concerns:

  • Consumers
    • Web developers, designers, content authors
    • Web content auditors (e.g. accessibility evaluator)
    • Web content procurers
    • Web quality assurance testers
  • Providers
    • Tool providers (e.g. Commercial or non-commercial tool providers)

Secondary audience of this resource concerns:

  • Product/web managers
  • Policy and decision makers
  • Legal
  • Researcher
  • Trainers/faculty

User stories

With regard to the primary audience:

  • Web developer or designer who wants to evaluate guidelines during design and development to make sure the product or service they are working on is accessible.
  • Web content author who wants to make sure their content is accessible for all users and who searches a tool to support that goal.
  • Web content auditors (evaluators) and Web QA testers who want to audit existing websites and web content for accessibility guidelines compliance.
  • Web content procurer who searches for a tool to check accessibility during the procurement proces
  • Tool providers who want to communicate availability of an accessibility evaluation tool

Secondary audience of this resource concerns:

  • Product/web manager - selects tool to report compliancy to accessibility requirements
  • Policy and decision maker - select tools to report compliance to accessibility requirements that policies must abide to
  • Legal - selects tool to report compliance to accessibility requirements
  • Researcher - selects tool to complement current expertise with accessibility knowledge
  • Trainers – selects tool(s) to illustrate accessibility issues in web accessibility courses and to demonstrate how tools can be used to help evaluate accessibility

Resources

This activity will redesign the list of web accessibility evaluation tools to include:

  1. List of available tools, including:
    1. A sidebar with filters (features). The features currently include: guidelines version, languages, type of tools, supported formats, assists by.. automatically checks, license. Also WCAG Rules implementation level and other aspects to be defined.
    2. A main area listing tools that match the selected filters
  2. Database (or the like) containing the data about the tools
  3. Mechanism where tool providers can input and update information about their tools, including:
    1. Add your tool form: where tool providers can enter details (like the features) about their tool(s)
    2. Back-office for W3C staff to update the offerings based on the details entered by the providers

Approach

Refine categories and taxonomy

  • Including, but not limited to:
    • Tool Name
    • Vendor Name
    • Tool Description
    • Web Address
    • Release date
    • Version release date and/or information updated (date)
    • Guidelines
    • Assist by
    • Automatically checks
    • Tool Languages
    • Browser plugin
    • Supported format
    • Online Service
    • License type
    • Accessibility information
    • Information updated
    • Type of tool
    • WCAG-rules implementation
    • Also see below filter needs from the user interviews
  • Research specific requirements, such as from Web Accessibility Directive (WAD)
  • Add specific features to the filter list based on interviews with the target audience and on discussion with EOWG members
  • Define the workflows for submitting, moderating, and editing entries
  • Use WAI website components infrastructure, frameworks, and styles for final online version
  • Re-populate list according to revisited criteria

Requirements from interviews with users

We started with interviews of persons with different roles as described in the section audience. We interviewed a developer of an accessibility evaluation tool, two front-end developers, a web editor, accessibility lead, professor of design, lecturer design and user research and a product manager with focus on accessibility. We also looked into the comments on Github. During the development, we are also organizing hallway interviews to continuously collect feedback.

The requirements have been categorized as either “necessary” or “suggested”. This distinction has been made based on the frequency and severity of user comments and observations during use of the tool list. Necessary requirements contain user needs that are essential to the usability and effectivity of the tool list. Those requirements that have been marked as “suggested” are valuable for the design process, but: aren’t absolutely required for a well-functioning list, are in conflict of the W3C requirements, or were only mentioned/observed in a small number of interviews and thus more research would be needed to estimate their importance.

Finding a tool

  • Necessary requirements
  1. The page layout should guide the user towards their first action
  2. The page should offer a textual search function
  3. The list should allow filtering based on user goal, e.g. to check contrast
  4. Tools should open in a new tab, so that users can more easily try them out, before returning to the list to make a decision
  5. The list should allow users to sort the entries based on their enquiries; such sorting categories should include:
    1. (Optional) Most used tools
    2. Recently updated
    3. (Optional) Recently added
  • Suggested requirements
  1. Filtering should narrow down the choices to a small list that is easy to browse
  2. It should be clear which tools the user has already visited through the provided link
  3. The site should offer a way to compare tools based on their properties

Composing the tools list

  • Necessary requirements
  1. It should be clear how the list is filled and regulated, so users can more easily estimate the quality of the entries
  2. The page should show when the list has been last updated and validated
  3. The list should include the most popular and most used tools currently available to increase users’ trust in the quality of the list
  • Suggested requirements
  1. Tools should be checked for usability and reliability of the functionality they claim to offer, before adding them to the list
  2. While W3C wants the list to be a comprehensive overview of the tools available, users would much prefer a curated list selected by accessibility experts - especially to help guide new users

Information needs

  • Necessary requirements
  1. The information provided per tool should contain:
    1. Use case (tool checks general accessibility/checks for specific elements/simulates disabilities)
    2. Features (see “Filters”)
      1. Tool output
    3. Date of last updated
    4. Release date
    5. Author/company (when known to the user they evoke trust for the tool provided)
    6. (Optional) Visuals such as screenshots to aid recognizability and quick assessment of tools
  2. The site should help users of all relevant professions (e.g. web editors, UX designers, software developers,…) to find tools that will aid them in their tasks
  3. All text on the page should be in plain language and easy to understand
    1. (Optional) Preferably, the site should be accessible in the user’s preferred language
  • Suggested requirements
  1. The site should support and guide users through contextually relevant explanations
  2. W3C should guide users on how to effectively choose and use tools, including information on what accessibility evaluation tools can and cannot do.

Filters

  • Necessary requirements
  1. Filters should have plain titles and if necessary, a clear description
  2. Filter for guidelines should be simplified

Filters needed are:

  1. Which set of guidelines can the tool check, e.g. WCAG 2.1
  2. Supported devices (mobile/web/other)
  3. Supported Operating system (Android/iOS)
  4. Supported browsers for extensions
  5. What does the tool check
    1. Topics such as contrast, documents, video, etc.
    2. (Optional) Filter by specific guidelines, e.g. "text alternatives"
    3. (Optional) Filter by type of impairment (e.g. color blindness or low literacy)
    4. Paid/free trial, then paid/completely free/partial free
  6. Type of tool
    1. Does the tool provide immediate, one-time results, or can it be used for continuous evaluation
    2. Does the tool check general accessibility, specific elements or guidelines, or does it simulate disabilities
    3. Tool integration: website, browser extension, desktop application, app, server-side or other
    4. Tool itself must be accessible

Design & layout

  • Necessary requirements
  1. The site’s purpose should be clear to the user
  2. The site should be accessible according to the WCAG 2.1 guidelines
  3. The layout of content on the site should aid readability and overview
  4. (selected) Filters should always be visible.
  5. It should be clear when the list changes based on filter selection
  • Suggested requirements
  1. The design of the site should be aesthetically pleasing and modern
  2. Information shown should be tailored to the current task of the user - irrelevant information should be kept to a minimum (e.g. Hide disclaimer after reading)
  3. It should be easy to find the list through navigation on the W3C website, and users should be able to easily recognize the location of the list within the website

Adding and updating tools

  • Necessary requirements
  1. The site should provide concise and clear information about the submission & validation process
  2. The site should provide information on how users can update information after submitting a tool
  3. The list should enable users to easily update their list entries
    1. (Optional) Allow users to automatically update list entries through integration with (external) databases (e.g. Github)
  4. When adding tools users should be able to review the information entered before submitting the tool
  • Suggested requirements
  1. The submission form should inform users which type of tools can be submitted and enable users to submit information relevant to those types of tools
  2. Submitted tool descriptions should be reviewed based on how informative and easy-to-read they are
  3. The site should enable users to submit tools even if they don’t have all the information required

Proposed filters

Below are the proposed filters based on the input of the interviews and the existing tool.

  • Purpose
    • Automated testing
    • Manual testing
    • Simulate user experience
  • Product to evaluate
    • Website
    • Desktop application
    • Mobile application
    • Document
    • Source code
    • Other
  • Media type
    • Text / Language
    • Video / Animations
    • Image
    • Audio
  • Type of tool
    • Browser plugin
    • Desktop application
    • Mobile application
    • Web application
    • Command-line / CI
    • Programming tool plugin / IDE
    • Authoring tool plugin
  • Paid or free
    • Free
    • Limited
    • Paid
  • Scope
    • Page component
    • Single page
    • Sample of pages
    • Entire product
    • Restricted or password protected pages
  • Guidelines
    • We keep the current filters with WCAG 2.1 at the top
  • File format
    • Microsoft Office documents
    • Open Documents Format (ODF)
    • PDF
    • HTML / XHTML
    • CSS
    • SMIL
    • Markdown
  • Accessibility statement
    • Statement available
  • Implements ACT rules
    • Checkbox on/off
  • Operating system
    • Android
    • iOS
    • Windows
    • MacOS
    • Linux
    • Other
  • Browser
    • Chrome
    • Safari
    • Edge
    • Firefox
    • Opera
    • Other
  • Language
    • Any language that is offered
  • Checks
    • Color contrast
    • Page structure
    • Navigation and links
    • Plain text
    • Text alternatives
    • Media alternatives
    • Labels
    • Forms
    • Tables
    • Maps
    • Hover and Focus
    • Resizing page
    • Keyboard accessible
    • Compatible with assistive technology
    • Time (pause, adjust etc)
    • Error (prevention)
    • WAI-ARIA attributes
  • Output
    • Generate report of evaluation results
    • Display information within pages
    • Modify the presentation

Timeline

Submission Form:

  1. Starfish Review
  2. EO Discussions
  3. Monkey Review
  4. EO Discussions
  5. Butterfly Review - open until 05 June 2023
  6. Publication - 19 June 2023
  7. Email existing vendors with information on new submission form and timeline for new list publication - 19 Jun 2023

Selecting Evaluation Tools page:

  1. Monkey Review
  2. EO Discussions
  3. Butterfly Review - pending, to be open from 05 - 19 June 2023
  4. Publication - 12 July 2023

Evaluation Tools list itself:

  1. Starfish Review
  2. Monkey Review
  3. EO Discussions 16 September
  4. Butterfly Review open until 05 June 23
  5. Delete old data and testing PRs from list - by 19 June 2023
  6. Publication - 12 July 2023


Planning:

  • DONE: Squirrel Review (requirements analysis): M01-02
  • DONE: Eagle Review (concept draft – structure, interaction and content): M03-04
  • 20 - 31 May -- Survey for submission page (Starfish Review / complete draft)
  • 3 June -- EOWG meeting discussion
  • 3 - 14 June -- Survey for Starfish Review of Selecting tools page
  • 3 - 14 June -- Survey for Monkey Review (final draft - pre-publication review) of Tools page, submission page
  • 17 June -- EOWG meeting discussion
  • 10 - 21 June -- Survey for Monkey Review of Selecting page
  • 24 June -- EOWG meeting discussion
  • 24 June - 5 Juli -- Survey for Butterfly Approval (approval to publish) for Tools page, Selecting page and submission form
  • 8 July EOWG meeting discussion
  • Re-population phase: July 2022 - Jan 2024

Open issues

  • Add metadata to dataset for easier navigation
  • Change interaction design
  • How to support non-technical users?
  • Document what type of "tools" will be included and what not. Put it where we can refer to it. Maybe linked from the Submission Form? Ask for input from current and former maintainers on "gray areas" they have encountered: Shadi Abou-Zahra, José Ramón Hilera González.
Retrieved from "https://www.w3.org/WAI/EO/wiki/index.php?title=Evaluation_tools/Revision_2021&oldid=38006"