Problem Statements

These problem statements have been written by the Silver Community Group and Silver Task Force for use in the Silver Design Sprint. They will be used to focus discussion on critical problems to be solved.

Definition of a Problem Statement

A useful artifact to inform all solution design activities is to first articulate the problem space. A useful format to compose this problem statement includes: the specific problem; the result of the problem; the situation and priority; and the opportunity.

Key Problem Areas

The problem statements have been grouped into three areas for convenience, but it should be noted that they are inter-related.

• Usability
• Conformance Model
• Maintenance

Usability

Themes from research

Information Architecture (Using)

• WCAG is difficult to master.
• WCAG is difficult for beginners
• It is difficult to find the information desired
• WCAG lacks current practical examples
• Documentation is spread across multiple documents

Wording/writing style (Understanding)

• Ambiguity in interpreting the success criteria
• Difficult to persuade others to meet WCAG
• Too technical or poorly written
• Difficult to read and understand
• Difficult to understand/connect real world impact of specific success criteria on people with disabilities
• Most of the trusted statistics on disability are generalized, infrequent, and not aggressively publicized, such as the WHO World Report on Disability

Too Difficult to Read

Specific problem: Undefined acronyms, specialized terms, pseudo-legalese, and complex sentence structure decrease user’s comprehension, especially for people in the development cycle who are less technical (project managers, designers, social media marketing leads), regulators, and international users who need translations.

Result of problem: When users give up on understanding accessibility standards, it increases the perception that accessibility is too hard to do. Developers give up and leave accessibility to the specialists late in the project to bolt on only the most basic accessibility features. This results in difficulty in translation.

Situation and Priority: Difficult to read influences an opinion or reaction that accessibility is difficult to implement. Typically, a version one or initial release of any project doesn’t include accessibility because it was considered too difficult to ship. This means that people with disabilities are excluded from new technology.

Opportunity: When guidelines are easier to read and understand, users – especially people in the development cycle who are less technical – are more likely to implement accessibility. When all audiences are considered in the language and terminology used in the guidelines, the likelihood increases that they will: reach a larger audience; be better understood; be easier to translate; be interpreted as easier to implement.

Difficult to get started

Specific problem: WCAG is so complex in structure and content with documents, layers and resources that it is overwhelming to people who want to use it as a reference. Among other issues, it also difficult to search for – across multiple documents, and search within. This can be intimidating for people new to the topic who are genuinely interested in and / or tasked with supporting accessibility.

Result of problem: People new to the topic who want to use WCAG go to other websites that provide condensed summary information and simple checklists. Often, these are less nuanced and add to the “checklist mentality” that does not provide good accessibility in the long run. Others turn to books and other materials in order to better understand the topic conceptually, believing that will enable them to better understand WCAG. Even if successful, this extends the learning curve and process.

Situation and Priority: Accessibility is seen by designers as a constraint that is too limiting or at odds with brands. Accessibility is also seen by developers as a burden of effort and/or that it can simply be added at the end (checklist mentality). Well-intentioned designers and developers often give up on accessibility because it is too difficult to get started.

Opportunity: Improving usability so that users can develop understanding and mastery of the WCAG material leads to faster and greater acceptance of accessibility. This also creates the opportunity to convince developers and project managers to include accessibility at the beginning of a project instead of the end.

Ambiguity in interpreting the success criteria

Specific problem: There isn’t a clear distinction on what is the “right answer”. Even accessibility experts disagree. As the technology and contexts that can make requests and output web content continues to expand, it becomes less clear over time, which contexts the guidelines apply to. A browser is now only one of many such contexts.

Result of problem: Differing interpretation can lead to less support and more time resolving disputes than shipping support. Additionally, developers can be forced to reduce the accessibility of an application so it can pass specific WCAG success criteria or conformance.

Situation and Priority: WCAG can be seen as more of a barrier to more complete accessibility because it doesn’t provide clear solutions to modern applications. Native app developers can say that WCAG doesn’t apply to them because it is more difficult to apply to applications, both native and web.

Opportunity: Increase acceptance by developers by eliminating underlying assumptions and structure of the static web and providing clear answers for complex applications will provide greater accessibility of web and native applications.

Persuading Others

Specific problem: Demonstrating that accessibility is not only important to people with disabilities, but that it also benefits the business as a whole can be a challenge. The fact that accessibility is required by law does not necessarily influence decision makers to invest in accessibility. There are many compelling reasons for this, but ultimately, it can be difficult to calculate or predict the business and human impact within any given industry.

Result of problem: Organizations that are insufficiently convinced will not allocate any effort to accessibility. By dismissing the importance, they may also be less inclined to include PwD in any product design, testing or decisions.

Situation and Priority: Accessibility has somehow become an argument with neutral or opposing parties that need convincing. While this is not directly a standards problem, there is much that Silver could do to help both accessibility experts and non-experts who go to W3C for accessibility on giving greater context to why the advice is needed and important so that people are better educated.

Opportunity: Improving the usability of Silver can help improve the general advocacy of digital accessibility. Improving the reach of Silver can help improve the awareness of accessibility considerations. More compelling information that is contextually relevant to the standards may also aid in convincing audiences of any type.

Conformance Model

Definition

“Conformance to a standard means that you meet or satisfy the ‘requirements’ of the standard. In WCAG 2.0 the ‘requirements’ are the Success Criteria. To conform to WCAG 2.0, you need to satisfy the Success Criteria, that is, there is no content which violates the Success Criteria.”

WCAG 2.0 Conformance Requirements:

1. Conformance Level (A to AAA)
2. Conformance Scope (For full web pages only, not partial)
3. Complete Process
4. Only “Accessibility-supported” ways of using technologies
5. Non-Interference: Technologies that are not accessibility supported can be used, as long as all the information is also available using technologies that are accessibility supported and as long as the non-accessibility-supported material does not interfere.

Themes from Research

• No monitoring process to test the accuracy of WCAG compliance claims (Keith et al, 2012)
• Difficulties for conformance (Keith et al, 2012)
  • Third parties documents, applications and services
  • Know-how of IT personnel
  • Tension between accessibility and design
• Specific success criteria for failure - 1.1.1 , 2.2., 4.1.2 (Keith et al, 2012)
• “Reliably Human Testable” , “not reliably testable” (Brajnick et al, 2012) average agreement was at the 70–75% mark, while the error rate was around 29%.
  • Expertise appears to improve (by 19%) the ability to avoid false positives. Finally, pooling the results of two independent experienced evaluators would be the best option, capturing at most 76% of the true problems and producing only 24% of false positives. Any other independent combination of audits would achieve worse results.
  • This means that an 80% target for agreement, when audits are conducted without communication between evaluators, is not attainable, even with experienced evaluators
• Challenges and Recommendations (Alonso et al, 2010)
  • “accessibility supported ways of using technologies”.
  • Testability of Success Criteria
  • Openness of Techniques and Failures
  • Aggregation of Partial Results
• Silver need to expand the scope beyond web to include apps, documents, authoring, user agents, wearables, kiosks, IoT, VR, etc. and be inclusive of more disabilities. (UX Professionals Use of WCAG: Analysis)
• Accessibility Supported allows inadequate assistive technologies to be claimed for conformance, particularly in non-native English speaking countries. (Interviews on Conformance)

Constraints on What is Strictly Testable

Specific problem: Certain success criteria are quite clear and measurable, like color contrast. Others, far less so. The entire principle of understandable is critical for people with cognitive disabilities, yet success criteria intended to support the principle are not easy to test for or clear on how to measure. As a simple example, there is no clear, recent or consistent definition – within any locale or language – on what “lower secondary education level” means in regards to web content. Language and text content is also not the only challenge among those with cognitive and learning disabilities. Compounding this, most of the existing criteria in support of understanding are designated as AAA, which relatively few organizations attempt to conform with.

Result of problem: The requirement for strict testability for WCAG success criteria presents a structural barrier to including the needs of people with disabilities whose needs are not strictly testable. Guidance that WCAG working group members would like to include cannot be included. The needs of people with disabilities – especially intellectual and cognitive disabilities – are not being met.

Situation and Priority: Of the 70 new success criteria proposed by the Cognitive Accessibility Task Force to support the needs of people with cognitive and intellectual disabilities, only four to six (depending on interpretation) were added to WCAG 2.1 and only one is in level AA. The remainder are in level AAA, which is rarely implemented. There is a failure of expectations and a failure to meet user needs.

Opportunity: Multiple research projects and audience feedback have concluded that simpler language is desired and needed for audiences of the guidelines. Clear but flexible criteria with considerations for a wider spectrum of disabilities helps ensure more needs are met.

Human Testable

Specific problem: Regardless of proficiency, there is a significant gap in how any two human auditors will identify a success or fail of criteria. Various audiences have competing priorities when assessing the success criteria of any given digital property. Knowledge varies for accessibility standards and how people with disabilities use assistive technology tools. Ultimately, there is variance between: any 2 auditors; any 2 authors of test cases; and human bias. Some needs of people of disabilities are difficult to measure in a quantifiable way.

Result of problem: Success criteria are measured by different standards and by people who often make subjective observations. Because there’s so much room for human error, an individual may believe they’ve met a specific conformance model when, in reality, that’s not the case. The ultimate impact is on an end user with a disability who cannot complete a given task, because the success criteria wasn’t properly identified, tested and understood.

Situation and Priority: There isn’t a standardized approach to how the conformance model applies to success criteria at the organizational level and in specific test case scenarios.

Opportunity: There’s an opportunity to make the success criteria more clear for human auditors and testers. Educating business leaders on how the varying levels of conformance apply to their organization may be useful as well. We can educate about the ways that people with disabilities use their assistive technology.

Accessibility Supported

Specific problem: ‘Accessibility supported’ was never fully implemented in a way that was clear and useful to developers and testers. It also requires a harmonious relationship and persistent interoperability between content technologies and requesting technologies that must be continuously evaluated as either is updated. Further, the WG “defers the judgment of how much, how many, or which AT must support a technology to the community”. It is poorly understood, even by experts.

Result of problem: Among the results are: difficulty understanding what qualifies as a content technology or an assistive technology; difficulty quantifying assistive technologies or features of user agents; claiming conformance with inadequate assistive technology; and difficulty claiming conformance.

Situation and Priority: Any claim or assertion that a web page conforms to the guidelines may require an explicit statement defining which assistive technology and user agent(s) the contained technologies rely upon, and presumably inclusive of specific versions and or release dates of each. One could infer then that a conformance claim is dependant upon a software compatibility claim naming browsers and assistive technology and their respective versions. This would create a burden to author and govern such claims. Additionally, no one can predict and anticipate new technologies and their rates of adoption by people with disabilities.

Opportunity: As the technologies in this equation evolve, the interoperability may be affected by any number of factors outside of the control of the author and publisher of a web page. Either “accessibility supported” should be removed from conformance requirements, or it should clearly, concisely and explicitly define and quantify the technologies or classes of technologies, AND set any resulting update or expiry criteria for governance.

Evolving Technology

Specific problem: Evolving Technology: As content technology evolves, it must be re-evaluated against assistive technology for compatibility. Likewise, as assistive technology evolves or emerges, it must be evaluated against the backward compatibility of various content technology.

Result of problem: There is no versioning consideration for updates to user agents and assistive technology. Strict conformance then typically has an expiry.

Situation and Priority: There is no clear and universal understanding of the conformance model or its longevity. Some will infer that there is always a conformance debt when any technology changes.

Opportunity: Consider conformance statements to include a explicit qualifier of time of release or versions of technology. OR consider a more general approach that is not explicit and is flexible to the differences in technologies as they evolve, identifying the feature of the assistive tech rather than the version of the assistive tech. OR consider a model that quantifies conformance as a degree of criteria met.

Maintenance

Themes from discussion/research

Process:

• Need working group processes that are more agile

Information Architecture:

• Updates and additions to existing guidelines end up proving difficult to include in WCAG, keeping the overall structure entirely intact (numbering & levels).

Scope of disabilities explicitly identified

• Focus on screen reader users

Scope of interactions & technologies

• Rapidly changing technology

Flexibility

Specific problem: Strict document structure, workflow and approval processes can sometimes ensure efficiencies and predictable outcomes. However, they may also stifle innovation and inclusion. Additionally, this process can be intimidating for some and limit participation by PwD.

Result of problem: Accessibility standards are slow to become recommendations. Browser and assistive technology updates occur faster. However, organizations do not ship and publish accessibility support in websites and apps according to what can be done, they do so according to what the recommended guidelines say. Content and content technologies can lag available support. Additionally, the contributing audience to proposals for standards in this process is small in number and narrow in diversity.

Situation and Priority: Being conformant to the recommended guidelines is always less than fully accessible and less than current technology can support. Limiting PwD from the process may result in the unintentional and counterintuitive exclusion of support for a need of a particular audience.

Opportunity: Being more flexible in format enables greater scaling of new and expiry of outdated guidelines. Being more flexible in process enables guidelines to keep pace with technology. [open source] Being more flexible in participation – particularly of PwD – empowers a community and enables more inclusive insights.

Scaling

Specific problem: Updates and additions to existing guidelines end up proving difficult to include in WCAG, keeping the overall structure entirely intact (numbering & levels).

The taxonomy and explicit numerical outline system scales linearly – which is confusing and can get unruly (example: The ARIA Role Attribute in the HTML 5.2 recommendation is 3.2.8.3.1 in the outline.) – does not accommodate any additional organizational method, such as contextually by similarity or dependancy, or relationally by keyword. This can also make translation more difficult. Additionally, ensuring that PwD can easily access the documentation via assistive technology such as screen readers, as there is no semantic method for document outlines that have such a list depth.

Accessibility guidelines were not created with the anticipation of scaling in a way that would combine, absorb or incorporate the related specs for web content, user agent and authoring tools or any future accessibility guidelines.

Result of problem: Any additional specification and criteria must be appended in a manner that may not be clear or appropriate or ideal. Updates to related specs that contain redundancies or dependencies – such as for user agents and authoring tools – scale independently on their own track.

Situation and Priority: The current outline and numbering system is easy to cite, but confusing to traverse, discover and understand.

Opportunity: Consider multiple referring contexts, multiple use cases, and iteration and addendums over time when designing a system and taxonomy that is as future friendly as it is user-centric.

Governance

Specific problem: It has been difficult for accessibility guidelines to keep up with trends, new directions, and expanding scope of technology. Using the current methods of accessibility guidelines development, the input of new ideas and implementation can take years of work. For example, some suggested user needs for certain populations were identified more than 10 years ago.

Result of problem: As designers, developers, technologists, advocates, policy makers and stakeholders discover needs, there is no immediate path to propose or contribute to the standardization of solutions. Relatively few people attempt to contribute due to the complexity and perceived exclusivity of the W3C working group and recommendation track process. International participation is also imbalanced.

As an extension of this, all derivative documentation that is authored and published by organizations and advocates (like WebAIM and universities) becomes out of sync with subsequent versions that are released, as we are seeing with 2.1.

Situation and Priority: Technology and human needs move faster than this current process. Qualified and valuable participation and contributions are missed. Despite it being a standard convention of W3C process employed in other recommendations, there has been no practice to expire or deprecate individual criteria or documentation that is outdated or no longer relevant (ROT: redundant, obsolete or trivial).

Opportunity: Success criteria and all supporting documentation should be as forward looking and future friendly as possible – anticipating common scenarios like new technology. If possible, they should also consider the introduction of new modalities like surface reaction and ultrahaptics. A backlog could be maintained that reflects issues along with their status. Similarly or alternately, proposals could follow an open source model.

Appendix: Links to Research

It is the intention of the Silver Community Group to move all the research onto the Silver Community Group website. Until that is done, this Appendix links to public version of the reports on the Google Drive used by Silver Community Group for collaboration.

Research by Silver Community Group

1. WCAG Success Criteria Usability Survey. David Sloan and Sarah Horton. March 2018. (not complete) This survey asked 200 participants to rank each WCAG 2.0 success criterion by ease of learning, applying, and teaching. The results were analyzed to create a list of most usable and least usable success criteria.
2. Conformance Survey. Kelsey Collister and Charles Hall. (Draft of Key Findings). Survey sent to people with expertise in WCAG. 49 participants answered questions on the effectiveness of WCAG Conformance and on knowledge of the five WCAG 2.0 conformance requirements. March 2018.
3. Interviews on Conformance. Jeanne Spellman and Jan McSorley. Interviews of advocates, lawyers and authors of WCAG 2.0. February 2018.
4. Interviews on Legacy of WAI Accessibility Guidelines (WCAG, ATAG and UAAG). Jan Mc Sorley and Jeanne Spellman. March 2018.
5. Reimagining Accessibility Guidelines: An Analysis of Audience Feedback Jeanne Spellman, John McNabb, Kelsey Collister, Shawn Lauriat, Jennison Asuncion. February 2018.

Research Partners of Silver

1. UX Professionals Use of WCAG: Analysis Shari Butler and Jeanne Spellman. The survey was done by Peter McNally of Bentley University. This document is the analysis of his survey data that he shared with the Silver Community Group. February 2018
2. Internet of Things (IoT) Education: Implications for Students with Disabilities. Scott Hollier, Leanne McRae, Katie Ellis, Mike Kent of Curtin University. October 2017
3. Web Accessibility Perceptions Survey and Interviews: Analysis Shari Butler. This work was done by students of Professor Eleanor Loiacono of Worcester Polytechnical Institute. The paper has not been published, so the analysis paper is presented here. February 2018.
4. Analysis of Student Papers on Silver Research Topics: Analysis Shari Butler and Jeanne Spellman. The students of Professor Michael Crabb of Robert Gordon University. Undergraduate honors final year students were asked to select a question from the Silver Research Questions list, design a study, analyze the results, and write an academic paper. Professor Crabb forwarded nine the "A" grade papers to Silver. Caution must be applied to all results due to low sample sizes present within the work.