WAI R&D Symposia » Metrics Home » Proceedings » This paper.
This paper is a contribution to the Website Accessibility Metrics Symposium. It was not developed by the W3C Web Accessibility Initiative (WAI) and does not necessarily represent the consensus view of W3C staff, participants, or members.
Web Accessibility Metrics For A Post Digital World
1. Problem Addressed
This paper argues that, as we move towards a 'post-digital' world where use of the Web becomes normalised, there is a need to address Web accessibility measurement challenges within a wider real-world context. Strategy and policy that defines Web accessibility purely by the conformance of digital resources with technical guidelines can lead to a danger that 'good enough' solutions may fail to be deployed; they also fail to consider a wider measure of user experience in accessibility measurement.
We propose that metrics should draw on aspects of user experience to provide a more meaningful, real-world measure of the impact (or not) of accessibility barriers and therefore priority in addressing them. Metrics should also consider context in terms of the quality of effort taken by organisations to provide an inclusive experience; one option for doing so is the framework provided by British Standard 8878 Code of Practice for Web Accessibility. In both cases, challenges exist in the complexity of defining and implementing such metrics.
2. Background
Accessibility metrics provide a valuable role in establishing the extent to which a web resource can be effectively used by people with specific access needs. W3C WCAG exists as the de facto standard for measuring technical web content accessibility; it can be incorporated into other standards and policy, and act as a focus for automated evaluation tools.
However there are limitations to WCAG as a means of measuring accessibility of wider real-world online experiences (Kelly et al 2005; Sloan et al 2006). These limitations include the extent and nature of accessibility barriers covered by the guidelines, the relationship of guideline conformance to observed user experience (as found in e.g. Petrie and Kheir, 2007), and the focus on product rather than the process undertaken to create that product. This is potentially problematic for countries such as the UK, where legislation protecting the rights of disabled people focuses on access to information and services and the responsibilities of organisations to take action to promote equality, rather than technical conformance.
More recently, implications are now emerging of in a post-digital environment in which it becomes increasingly difficult to differentiate between Web resources and other examples of digital resources as use of digital means to deliver information, services and experiences becomes normalised (White, 2009). The emergence of mobile and tablet devices as platforms for Web content and experience delivery blurs boundaries further, while economic constraints place new pressures on the need to rapidly publish, appropriate and reuse digital content for different purposes. This leads to a pragmatic desire not to reject potentially useful content even if it contains known accessibility problems for specific groups of people, and where resource constraints limit the potential for addressing these problems quickly, directly, and fully.
For organisations who wish – or who are required – to measure progress in delivering accessible online experiences, there is a need for more sophisticated models of understanding and addressing accessibility issues, in a way that is sympathetic to current digital authoring practices and online experiences. This would allow accessibility policy to incorporate a conformance-driven model of measurement within the context of a more holistic measure of the level of inclusivity.
3. Strategy
Measuring accessibility should not be restricted to web pages; but rather the extent to which goals can be completed by disabled people relative to other members of the target audience; which may mean alternative equivalent routes, using alternative online resources if necessary. Further, this definition could incorporate more subjective, experiential issues, such as those presented by Hassenzahl (2010), that may modify the apparent impact of a specific accessibility barrier – either by lessening its impact due to a very positive use experience, or increasing its impact because of a negative user experience. In each case, the default WCAG priority of a technical barrier – and the urgency required to deal with it may change.
Techniques for measuring accessibility, usability and user experience of e-systems are discussed in Petrie and Bevan (2009), although there is no detailed discussion of areas of overlap between accessibility and user experience. However, the Techdis Accessibility Passport (http://www.accessibilitypassport.org/), designed to document accessibility of e-learning resources, provides a promising example of how the steps taken to optimise the accessibility of an online experience – beyond guideline conformance to include user engagement and reaction - can be documented, publicised and updated over time.
BS 8878 (BSI 2010), published in the UK, presents a framework for organisations to follow in the process of commissioning accessible web sites, from definition of accessibility, to procurement – whether internal or outsourcing – to evaluating the accessibility of the solution delivered. As a process standard focused primarily on managers and decision-makers, it is complementary to existing technical standards for accessibility.
4. Major Difficulties
The obvious difficulties in defining and implementing an accessibility metric that incorporates quality of user experience and the quality of the process undertaken to provide that experience are the complexity of the environment to be measured – i.e. not just a collection of resources that enable an experience, but also evidence of organisational activity taken to enhance inclusion. These require precise and logical definition; so measurement may take substantial time and effort, and will inevitably have a degree of subjectivity, which needs to be carefully controlled if measures are to be reliable and meaningful. Additionally, there is an obvious need for such a metric to take into account evidence of decision-making, in particular justification for failing to address accessibility barriers – while still encouraging Web resource providers to continue to focus on removing barriers wherever possible.
BS 8878 provides a framework that allows definition – and measurement – of the process undertaken by organisations to procure an optimally accessible web site, but is at present a copyrighted work and not freely available. In comparison to a purely technical WCAG conformance report, the nature of the data being gathered for measurement means that inevitably the measurement process is longer; but it also provides a richer set of data giving context – and therefore justification – to current levels of accessibility.
5. Outcomes
This paper presents a theoretical proposal, and as such no outcomes from our work exist. However, many of the ideas expressed in earlier work encouraging adoption of a more holistic approach to accessibility are reflected in the text of BS8878; while the Techdis Accessibility Passport provides an example of how rich documentation of accessibility of a resource could be achieved.
The logical outcome should be the formalisation of an accessibility metric that can be adopted at an organisational level or higher, as a means of defining effectiveness of an online accessibility strategy. This would incorporate technical guideline conformance and measures of user experience to disabled people alongside measures of the process adopted of procuring and providing accessible web content.
6. Open Research Avenues
Following the iterative development of an inclusive user experience metric, the next step would be to evaluate the effectiveness of such a metric. The authors are currently exploring (Kelly 2011) the potential of using such a metric to evaluate the online accessibility strategy of Higher Education institutions – specifically collections of academic web content they provide, such as institutional repositories of research output.
References
- BSI (2010) BS 8878:2010 Web accessibility – Code of practice.
- Hassenzahl M. (2010) Experience Design – Technology for All the Right Reasons. Morgan & Claypool.
- Kelly, B. (2011) Metrics, This Time For Web Accessibility. Retrieved 31 October 2011 from http://ukwebfocus.wordpress.com/2011/10/17/metrics-this-time-for-web-accessibility/.
- Kelly B., Sloan D., Phipps L., Petrie H. and Hamilton F. (2005) Forcing standardization or accommodating diversity? A framework for applying the WCAG in the real world. Proceedings of the 2005 International Cross-Disciplinary Workshop on Web Accessibility (W4A) (Chiba, Japan, 10 May 2005). New York:ACM Press, 46-54.
- Petrie, H and Bevan N. (2009) The Evaluation of Accessibility, Usability and User Experience. The Universal Access Handbook, C Stepanidis (ed), CRC Press, 2009.
- Petrie, H. and Kheir, O. (2007) The relationship between accessibility and usability of websites. In the Proceedings of CHI '07: ACM Annual Conference on Human Factors in Computing Systems. New York: ACM Press, 397-406.
- Sloan D., Kelly B., Heath A., Petrie H., Hamilton F. and Phipps L. (2006) Contextual Accessibility: Maximizing the Benefit of Accessibility Guidelines. Proceedings of the 2006 International Cross-Disciplinary Workshop on Web Accessibility (W4A) (Edinburgh, Scotland, 23 May 2006). New York: ACM Press, 121-131.
- White, D. (2009) PostDigital: Escaping the Kingdom of the New? Retrieved 31 October 2011 from http://tallblog.conted.ox.ac.uk/index.php/2009/06/19/postdigital-escaping-the-kingdom-of-the-new/.