Important note: This Wiki page is edited by participants of the RDWG. It does not necessarily represent consensus and it may have incorrect information or information that is not supported by other Working Group participants, WAI, or W3C. It may also have some very useful information.
User Evaluation Methods for Web Accessibility Evaluation
From Research and Development Working Group Wiki
Other contact(s): if applicable, name of other RDWG participant(s) who have particular interest and/or expertise in this research topic]
User evaluation, Evaluation, Conformance, Guidelines, User Testing
User evaluation is considered to be very important for Web accessibility evaluation, however not much known about how to properly conduct a user evaluation for Web accessibility with disabled users.
There are a number of disabililty advocacy groups who are performing user evaluation, however most of them appear to be focusing on assessment as it pertains to that particular group of users. It is costly and difficult to set up and maintain a user testing group which spans a greater number of disability categories, but some are doing it well. We need to know the difficulties of resourcing such a group and training them in the art/science of website evaluation.
Many website accessibility evaluations are carried out relying solely on automated tools, or automated tools with the addition of a manual check of the results. W3C and other literature specifically stipulate that an evaluation should not rely solely on automated tools. Automated tools may miss items which cause a person with assistive technology to have problems with a website, and some of the tools flag items as problems which do not appear to negatively affect such users. An effective user evaluation will provide detail on the problems faced by a user with assistive technology and often suggest additional resources for the website that will improve the user experience for all groups of users. Another problem is that alterating a website so that it helps one group of users may negatively affect users with other disabilities.
- Can we adapt and use traditional usability methods?
- What methodologies are most suitable for user testing with people with disabilities?
- What are the benefits of user testing in iterative development cycles?
- Most User testing is traditionally informal and lacks scientific rigour. What are the benefits of a more scientific/controlled approach to user testing, and how would this work in practice?
- How do we ensure that we cover broad range of disabilities, broad range of expertise, etc?
- What about alternative techniques such as remote user evaluation techniques (Power et al.)?
- How would you verify the results obtained by remote user evaluation?
- How do you set up and support a user evaluation team?
- Involving Users in Evaluating Web Accessibility
- A Framework for Remote User Evaluation of Accessibility and Usability of Websites
- End User Evaluations (Caroline Jay, Darren Lunn, Eleni Michailidou), Chapter in Web Accessibility: A Foundation for Research (Simon Harper, Yeliz Yesilada, eds.), Springer, 2008.
- Real World User Testing: An Assessment of User Testing Methodologies in Theory and Practice (PDF, 16.6 Mbs) Joshue O Connor, MSc in Computing (Assistive Technology)
Back to the list of topics.