Motivation

• Outline of our work in quantitative accessibility metrics since 2004
• We were not the pioneers...
• There had been some initial approaches since 2000
  • Sullivan and Matson (CUU'00): "Barriers to use: usability and content accessibility on the web's most popular sites"
  • Hackett et al. (ASSETS'04): "Accessibility of Internet websites through time"
  • and some more
• ...but there were some limitations at that time:
  • No normalized metrics
  • No score in terms of WCAG 2.0 principles (it was a draft at that time)
  • Failure-rate of semi-automatic checkpoints not considered

The Web Accessibility Quantitative Metric (WAQM)

• WAQM was proposed to
  • overcome the identified weakeness
  • adopt strenghts of state-of-the-art metrics
• Inherited practices
  • Apply the failure-rate for each checkpoint
  • Severities of violations were considered
• Innovative aspects
  • Normalized scores [0-100]
  • Provide scores for P, O, U, R as well as an overall score
  • Severities of violations were weighted: 0.8 for priority 1, 0.16 for priority 2 and 0.04 for priority 3
  • Failure-rate of semi-automatic issues was estimated

WAQM Algorithm

for i in each checkpoint in a guideline {P,O,U,R} loop
   for j in each type of checkpoint {auto, semi} loop 
      for k in each priority{1,2,3} loop
	     Ai,j=calculate_failure_rate()*priority_weight(k); 
      end loop      
   end loop
   Ai=(Ni,auto*Ai,auto+Ni,semi*Ai,semi)/Ni    
end loop
A=(NP*AP+NO*AO+NU*AU+NR*AR)/N

Validity and reliability

• On validity: how close are scores to actual accessibility?
  • A panel of experts evaluated the accessibility of 14 pages and gave a numeric score
  • A positive correlation between their scores and WAQM r(14) = 0.55, p < 0.05 was found
• On tool reliability: how replicable are results between tools?
  • We investigated tool reliability: LIFT and EvalAccess
  • Almost 1.5K pages were tested with LIFT and EvalAccess and then measured with WAQM
  • A strong rank correlation between all pages r(1363) = 0.72 was obtained although no correlation was found between absolute values
  • We proposed a method to obtain tool reliability by tuning some parameters in WAQM. We obtained positive results
• On guideline reliability: how replicable are results between guideline-sets?
  • Compare WCAG 1.0 and WCAG 2.0 in EvalAccess
  • Weak correlations found: (1449) = 0.38, p < 0.00 and Spearman's r(1,449) = 0.50, p < 0.00
  • Inter-tool and inter-guideline-set reliability is not always desirable because tools and guidelines are different