Towards a score function for WCAG 2.0 benchmarking

A contribution of the eGovernment Monitoring (eGovMon) project.

Presented by

• Annika Nietzio
• Forschungsinstitut Technologie und Behinderung (FTB)
• email: egovmon@ftb-volmarstein.de

How to develop a Web Accessibility Metric

Experiences from the development of the Unified Web Evaluation Methodology (UWEM) for WCAG 1.0

UWEM indicator refinement process

• collection of requirements (crawling and sampling, mathematical and statistical properties, influence of features of the web content)
• theoretical analysis (dependencies, potentially conflicting requirements)
• experimental evaluation (comparison of result on real and synthetic data)
• selection of score function

Lessons learnt

• Experimental evaluation is vital.
• The score function should be tailored to the structure of the test set.

Differences between WCAG 1.0 and WCAG 2.0

• WCAG 1.0: test are independent.
• WCAG 2.0: dependencies between Techniques (and the derived tests)
• Logical combinations must be taken into account.

Example: 3.3.2 Labels or Instructions (Level A)

• Test: Check that the purpose of a form field can be identified?
  • H44: label elements
  • H65: title attribute
  • G167: adjacent button
• Logical combination:

if ((H44:cause=no_label | H44:cause=label_empty) 
   & (H65:cause=title_missing | H65:cause=title_empty) 
   & G167:cause=empty_button_as_label)
      return error

Suggested score function and next steps

Web page score

• Erratic number of tests per Success Criterion (or Checkpoint)
• Accessibility score should not dependent on number of tests.
• Solution: use Success Criteria as intermediary aggregation level.
• The page score is calculated as the average of the SC-level page results.

Web site score

Aspects to be taken into account in web site score development:

• How to accommodate results of tests that are applied on site level?
• How to deal with conforming alternate versions?

Future work: A unified WCAG 2.0 score

• A generally accepted practice for reporting WCAG 2.0 results does not yet exist.
• Scores from WCAG 2.0 tools are not comparable. Differences are:
  • granularity of tests
  • counting the instances
  • result categories (error, potential error, warning)
  • reports: absolute numbers, percentages, more sophisticated scores

A unified WCAG 2.0 score

• inter-tool reliability
• collaboration between tool developers and researchers
  • Define comparable tests on atomic level.
  • Follow the logical combinations defined in "How to meet WCAG 2.0".
  • Agree on indicator requirements.
  • Define score function.