W3C homepageWeb Accessibility initiative homepage

WAI: Strategies, guidelines, resources to make the Web accessible to people with disabilities

5 December 2011

WAI R&D Symposia » Metrics Home » Proceedings » This paper.

This paper is a contribution to the Website Accessibility Metrics Symposium. It was not developed by the W3C Web Accessibility Initiative (WAI) and does not necessarily represent the consensus view of W3C staff, participants, or members.

A Template-aware Web Accessibility metric

  • Nádia Fernandes. University of Lisbon, nadiaf@di.fc.ul.pt
  • Rui Lopes. University of Lisbon, rlopes@di.fc.ul.pt
  • Luís Carriço. University of Lisbon, lmc@di.fc.ul.pt

1. Problem Addressed

Templates are highly used in Web development. Estimates indicate that 40-50% of the Web content uses templates[1]. Automatic accessibility tools report errors and warnings on pages built on those templates. Consequently, if a template presents an accessibility problem, that problem is reported as often as the template is used within the page/site.

From the developers’ perspective, standard accessibility evaluation tools provide obfuscating results. The same error is constantly repeated, producing unnecessarily long reports that confuse developers[2], concealing fundamental repairing issues.

Metrics for accessibility evaluation have the same issue. A bad result, deriving from a large number of errors in a page or site assessment, may be a consequence of a small number of problems in a frequently used template that can be rapidly corrected. Therefore, common accessibility metrics may be misleading.

We can decompose our research question in:

  1. What is the impact of templates usage in the reported number of accessibility problems of a page/site?
  2. How can we provide a simplified reporting of accessibility problems that better addresses the developers’ needs?
  3. How can we disguise a metric that answers the conformance towards accessibility guidelines but from the point of view of the effort to repair the accessibility of a page/site?

2. Background

Template detection is often used in the fields of Information Retrieval(IR) and page Clustering [3]. Although most work on templates ignores accessibility, its use has already been proposed as a mean to improve it[4]. In fact, if templates become accessible and widely used, there is a lower probability of having inaccessible pages. Otherwise, the possible errors will propagate, causing the previously mentioned issues.

Metrics, such as UWEM[5] and WAQM[6], are invaluable to assess page accessibility. These provide different perspectives of the accessibility quality. However, none directly addresses the developers' efforts to correct the accessibility problems, in relation to the common development process. Templates are fundamental for this process and must be considered.

3. Strategy

To detect templates, we propose the use of a simple algorithm to identify common elements amongst the HTML DOM trees - Fast Match algorithm[7]. It offers a reasonable estimate of the common elements for initial assessment. This algorithm was validated; the matches obtained were compared with the same matches conceived manually(100tests).

Then, we modified QualWeb, an automatic accessibility evaluator[8], to consider the algorithm. The tool accepts a set of pages, identifies common nodes and evaluates the pages. The evaluator itself did not influence the results; we could perform the evaluation with any other evaluator. We even perform similar evaluation in an accessibility evaluator of another project WCAG 2.0 Assessment Tool[9], but the assessments, from a templates' point of view, have not changed substantially.

Afterwards, we prepared a study comparing each Web page with its home page, to identify common elements. The results provide an indication of the percentage of accessibility issues that are detected on templates shared between those two pages, so that we can verify if templates have an important role in accessibility. For a deeper analysis, one should consider within-page templates and templates across several other pages of the same site.

To address reporting, we modified QualWeb to accommodate the template awareness. Reporting is aggregated in two sets: template(common nodes) and specific(unique page nodes). Consequently, problems(errors and warnings) are reported only once, if they occur in similar nodes. On the template set, each reported problem indicates the number of occurrences(of the common node/template).

To address the accessibility quality of a page/site, for a developer, we combine these sets’ assessment as follows:

  1. Α(pi)=αt + αs(pi)
  2. Α(S)=αt + i=0∑n[αs(pi)] (n - number of pages of the evaluated Web site)

The first equation indicates the number of accessibility problems of a page - Α(pi)-, combining the number of problems on the template set - αt- with those on the specific part of the page - αs(pi).The second applies to a site and thus sums the number of accessibility problems of specific part for each page. This way, a development team can know if the effort to correct the accessibility problems of a site/page is worthy.

The study for assessing the impact of templates was also used to produce an initial quantitative assessment of reporting and metrics. We selected four representative Web sites- Google, Wikipedia, Facebook and Amazon-, two Portuguese newspapers - DN, and Publico - and WordPress.

4. Major Difficulties

Major limitations:

  • The thresholds defined for the fast-match algorithm were not optimal for some sites, excluding some elements that are part of the template or the opposite;
  • Intra-page templates(defined inside the page itself, e.g. list, ads...) were not considered;
  • Homepages are sometimes not a good source of templates(e.g. Google). Consequently, we will have to study a better way to combine all the pages to consider the majority of templates existent on the site.

5. Outcomes

The results show the percentage of template vs. specific outcomes from techniques(i.e. pass, warn or fail), considering all the seven sites. The average for the template set of the accessibility outcomes(e.g. pass, fail, warning) is 38.85%(σ=7.48). Of those, 34.5%(σ=7.0) were warnings and 0.8%(σ=1.0) were fails. Therefore, about 35% of issues(fail and warning) would be addressed twice for repairing if templates were not considered.

For the selected sites, we obtained the following scores:
Web Sites %template(with problems) Possible problems without templates Α(S) Α(S)/Applicability Decrease
DN 36% 291476 192717 0.95 34%
WordPress 43% 320890 196706 0.92 39%
Google 19% 15183 12539 0.95 17%
Publico 38% 207531 131824 0.95 36%
Amazon 33% 128058 87720 0.95 32%
Wikipedia 39% 26985 17513 0.93 35%
Facebook 34% 273114 184677 0.93 32%

The results show a significant decrease in the scores. The results presented in each column(left to right) are:

  • the percentage of templates detected with possible problems;
  • the sum of accessibility problems of each page of the site;
  • the scores of the proposed metric, i.e. problems that should be addressed by the developer;
  • the normalization of the results of our metric(using applicability, e.g. accessibility outcomes), though we consider easier to understand the work to be accomplished without considering the normalization;
  • the percentage of decrease in the number of nodes that have to be checked by developers. That percentage would be even better if we considered more than the home_page as reference for detecting templates. Clearly, developers would be misled if standard metrics were considered.

6. Open Research Avenues

The main directions identified were: more accurate template detection (and backtracking); intra-page and extra-page templates; page clustering; another type of analysis; and developers’ assessment.

Acknowledgements

This work was funded by Fundação para a Ciência e Tecnologia (FCT) through the QualWeb national research project PTDC/EIA-EIA/105079/2008, the Multiannual Funding Programme, and POSC/EU.

References

  1. Proceedings: D. Gibson (2005) The volume and evolution of page templates. Special interest tracks and posters of the 14th international conference on World Wide Web (WWW2005). DOI:10.1145/1062745.1062763
  2. Proceedings:C. Law(2005) Programmer-focused website accessibility evaluations. Proceedings of the 7th international ACM SIGACCESS conference on Computers and accessibility (ASSETS2005). DOI:10.1145/1090785.1090792
  3. Proceedings:D. Chakrabarti (2010) The paths more taken: matching DOM trees to search logs for accurate webpage clustering. Proceedings of the 17th international conference on World Wide Web (WWW2010). DOI:10.1145/1772690.1772713
  4. Proceedings:L. Moreno (2008) Guiding accessibility issues in the design of websites. Proceedings of the 26th annual ACM international conference on Design of communication (SIGDOC2008). DOI:10.1145/1456536.1456550
  5. Miscellaneous:E. Velleman (2007) Unified Web Evaluation Methodology (UWEM) 1.2. Available from:http://www.wabcluster.org/
  6. Proceedings:M. Vigo (2008) Quantitative metrics for measuring web accessibility. Proceedings of the 2007 international cross-disciplinary conference on Web accessibility (W4A2007). DOI:10.1145/1243441.1243465
  7. Proceedings:S. Chawathe (1996) Change detection in hierarchically structured information. Proceedings of the 1996 ACM SIGMOD international conference on Management of data (SIGMOD1996). DOI:10.1145/233269.233366
  8. Proceedings:N. Fernandes (2011) On Web Accessibility Evaluation Environments. Proceedings of the 2009 international cross-disciplinary conference on Web accessibility (W4A2011). DOI:10.1145/1969289.1969295
  9. Miscellaneous:ACCESSIBLE Project