The Website Accessibility Metrics symposium brought together researchers and practitioners to scope the extent and magnitude of existing website accessibility metrics, and to develop a roadmap for future research and development in the field. Details are in the Background and Objectives.
Extended abstracts and slides from the symposium are listed below. See also transcript of the symposium.
- Paper 1: Integration of Web Accessibility Metrics into a Semi-Automatic evaluation process (Slides)
- Paper 2: Measuring accessibility barriers on large scale sets of pages (Slides)
- Paper 3: A Template-aware Web Accessibility metric
- Paper 4: A metrics to make different DTDs documents evaluations comparable (Slides)
- Paper 5: Lexical Quality as a Measure for Textual Web Accessibility (Slides)
- Paper 6: Attaining Metric Validity and Reliability with the Web Accessibility Quantitative Metric (Slides)
- Paper 7: The case for a WCAG-based evaluation scheme with a graded rating scale (Slides)
- Paper 8: A zero in eChecker equals a 10 in eXaminator: a comparison between two metrics by their scores (Slides)
- Paper 9: Context-Tailored Web Accessibility Metrics (Slides)
- Paper 10: Web Accessibility Metrics For A Post Digital World
- Paper 11: Towards a score function for WCAG 2.0 benchmarking (Slides)
Measuring the level of web accessibility is essential for assessing accessibility implementation and improvement over time but finding appropriate measurements is non-trivial. For instance, conformance to the Web Content Accessibility Guidelines (WCAG) is based on 4 ordinal levels of conformance (none, A, AA, and AAA) but these levels are too far apart to allow granular comparison and progress monitoring; if a websites satisfied many success criteria in addition to all Level A success criteria, the website would only conform to level A of WCAG 2.0 but the additional effort would not be visible.
Using numerical metrics potentially allows a more continuous scale for measuring accessibility and, to the extent that the metrics are reliable, could be used for comparisons. However, it is unclear how metrics can be developed that fulfill requirements such as validity, reliability, and suitability. For example, is a web page with two images with faulty text alternatives out of ten more accessible than another page with only one image with a faulty text alternative out of five? While such a count may be a fairly simple and reliable metric it is generally not a valid reflection of accessibility without additional information about the context in which the faults occur, but identifying this context may introduce complexity, reduce reliability, and raise other challenges.
More in-depth background and discussion on web accessibility metrics can be found in the RDWG wiki.
The primary objective of this symposium is to gather, analyze, and discuss practical experience with measuring website accessibility. These may include approaches for measuring 'accessibility in terms of conformance' (metrics that reflect violations of conformance of web content with accessibility guidelines such as WCAG or derivatives such as Section 508) and 'accessibility in use' (metrics that reflect the impact that accessibility issues have on real users, regardless of guidelines). The papers resulting from this symposium will constitute the basis from which to further explore a research and development roadmap for website accessibility metrics.
We particularly welcome discussion of the relationship of these two approaches and how to potentially combine them, as well as a discussion of any of the following types of questions:
- What sort of techniques can we explore to combine metrics that are computed automatically, semi-automatically (with input from humans), and manually (where the judgment is made by humans, even if with input from software)?
- How can we build an infrastructure (such as IBM Social Accessibility) that allows experts to store accessibility information (metadata) for use with metrics that are computed during subsequent audits?
- What metrics, or combination of metrics, can be used as predictors of accessibility?
- How shall we characterize the quality of such predictors in terms of properties such as reliability, validity, sensitivity, adequacy and adaptability?
- Which approaches can be embraced for validating, benchmarking, and comparing web accessibility metrics?
- How should we tackle metrics in web applications with dynamic content?
Further open research questions and ideas have been identified and we welcome contributions related to any of these too.
The Website Accessibility Metrics symposium was organized by the W3C/WAI Research and Development Working Group (RDWG). Contact Shadi Abou-Zahra (W3C Staff Contact) with questions.
- Giorgio Brajnik
- Markel Vigo
- Joshue O Connor
- Shadi Abou-Zahra
- Mario Bartusic
- Giorgio Brajnik
- Simon Harper
- Joshue O Connor
- Markel Vigo
- Yeliz Yesilada