Site Navigation: W3C Home > WAI Home > WAI Web Site Redesign Project > WAI Site Evaluation Summary

Findings

Executive Summary

Overall the WAI Web site was received well by the study participants. They described the W3C as "essential" to their Web development work and stated that they considered the W3C to be "THE authority" for Web development concerns. Participants uniformly stated that they currently used and would continue to use the WAI Web site in their professional work.

However, despite their enthusiasm for the W3C and the importance of Web accessibility information to their Web site use, we observed the majority of Web site users struggling to match the Web site's cognitive model with their own, struggling to interpret nomenclature, to locate information, to find information a second time, and most importantly, we observed them experiencing navigational difficulties that appeared to have a significant impact on their ability to harness the Web site's capabilities.

Fortunately, many of the nomenclature, navigation, visual design, and cognitive model difficulties we observed the study participants encountering may be easily addressed. In the report that follows, we describe our observations, the participants' successes, and their challenges.

Usability Test Task Results

At the core of participants' explorations of the WAI Web site were a series of "typical" tasks that Web developers were likely to attempt while using the WAI Web site. These tasks were intended to represent key Web site functions and features, and to facilitate participants' exploration of a range of WAI Web site information mechanisms.

Participants were largely unable to complete these tasks. With the exception of the WAI mission task, no more than 5 of 8 participants were able to complete any task without assistance from the test administrator.

While participants often struggled with the navigation and information architecture throughout the Web site, their problems were more frequent and prohibitive for tasks involving the guidelines, checklists and techniques sections (tasks 6 and 7). The table below details the task success rates and subjective ratings for each of the tasks.

Table 3 - Task Success Rates
Task Task Description % completed % completed w/ assistance % failed % who rated task as "easy"
3 WAI Mission 100% 0% 0% 88%
4 Quick Tips 63% 12% 25% 63%
5 Participation 63% 0% 37% 63%
6 Evaluation 0% 50% 50% 25%
7 Online Forms 38% 12% 50% 12%
8 Training Info 50% 25% 25% 50%

Key Findings

Finding 1: Participants uniformly stated that the Web site was "overwhelming" and that using it was "like walking into a fire hose."

Perhaps the most significant finding of the WAI Web site usability test was the observation that the Web site users consistently stated that they were overwhelmed by the information available to them. They stated that the WAI developers "hadn't parsed" the information for users, hadn't provided adequate "information and technique summaries" and asserted that using the WAI Web site was like "walking into a fire-hose."

They repeatedly suggested that the information they were receiving could be "summarized" into "manageable chunks" for easy, or quick use.

Participants uniformly stated that the WAI Web site information that was most relevant to their work was always "hidden." They described the process of using "a W3C site" as: "you go to the page, scroll down, scroll down, scroll down, about four or five screens, past all the useless stuff, and then you get to the good stuff. Which is great. But it's always in the middle part of a really, really, really long page."

Finding 2: Participants expected the Web site to be organized in a manner that supported task-based information seeking.

During the usability test we observed a clear discrepancy between the participants' Web development needs and the manner in which information was organized on the WAI Web site. Participants uniformly stated that their primary reason for coming to the WAI Web site was to locate specific, task-based information about Web coding techniques.

The current WAI Web site's information architecture put tactical techniques three or four levels, and in some cases, documents, "deep" within the Web site. Compounding this disparity, WAI techniques on a given topic were often dispersed across several document sub-sections, making it exceedingly difficult for participants to 1) locate specific techniques, and 2) ascertain whether or not additional related techniques existed on the WAI Web site.

For example: when we asked participants to use the WAI Web site to locate information about making online forms accessible, they stated that this task represented "exactly the type of question" that they would visit the WAI Web site to answer. (A few of the participants had, in the past, visited the WAI Web site for this purpose.) With a task in hand that was highly-reflective of their Web development needs, participants "dove in" to the WAI Web site with enthusiasm. They stated that they expected to be able to easily locate a single, authoritative, techniques-based set of information pertaining to making online forms accessible, in part because "this is a huge topic and very important." As they searched they envisioned a "Frequently Asked Questions-like format that would provide a simple, detailed answer to the question: 'How do I make online forms accessible?'"

In practice, the WAI Web site provided no unified source of form-related accessibility techniques, and the techniques themselves were a sub- sub-set of the WAI guidelines documents. The manner in which this information is presented on the WAI Web site simply didn't match the participants' expectation of how this information should be organized, or how easily it should be able to be located. We observed this discrepancy having a significant effect on participants' ability to locate practical information on the WAI Web site. In the words of one participant: "What should be on the top is on the bottom, and what should be on the bottom is on the top."

Finding 3: The distinction between, and the relationship among, guidelines, checkpoints, checklists and techniques were universally unclear to the participants.

We observed participants struggling to learn the "WAI vocabulary" in order to understand how to interpret the information they encountered on the Web site. It appeared to be crucial to their success with the Web site that they first understand the distinction between guidelines, checkpoints, checklists, and techniques. Few participants achieved that level of understanding, and nearly every participant made mention of the Web site's ability to communicate this information as "insufficient." This method of organizing the information struck participants as "all about WAI and not about the information." Other participants noted that this distinction was "confusing" and "arcane."

Finding 4: Participants' ability to locate task-specific information was significantly hindered by the "front matter" at the start (and close) of the guidelines documents, the checklist, and techniques documents.

Participants consistently expressed negative reactions to the organization of WAI's Web-based documents. In particular we observed that the initial sections of WAI guidelines, techniques, and checklist documents, (typically featuring the document's development status and history), were a significant barrier to participants' ability to identify the relevance of the document, comprehend its content, and to navigate within the document.

Participants uniformly stated that whenever they attempted to read a WAI document, they had to first, in one participant's words: "scroll past a ton of irrelevant information about who wrote the document, why, when it was written, and when it was modified." Participants stated that they believed this information "must be of value to someone" but that it "interfered" with their ability to locate "the important information."

They uniformly suggested that the WAI development team consider ways to parse the Web site's information rather than giving it to them "all at once."

Suggestions were made to consolidate guideline document meta information (it's draft history) and place it behind a single link, to consolidate technique information around common topics (such as "making online forms accessible"), and consolidating the guidelines themselves to provide the most specific, tactical information "up front" and reserve the detailed discussion for secondary pages. The material found at the bottom or end of documents received similar unfavorable responses. Said one participant: "last updated- and all this dollar business [dollar signs], that's not really helpful."

Finding 5: Participants stated that the WAI Web site lacked both "Getting Started" overviews and specific coding techniques

The usability test participants stated that they primarily expected (and needed) two types of content from the WAI Web site: 1) a "Getting Started-style" overview of Web accessibility and 2) "practical coding tips and tricks for creating accessible Web sites." They felt that the inclusion of Getting Started information would help orient them to the issues involved in Web accessibility and that detailed introductions for, and examples of, implementation techniques would support their ability to implement "good Web accessibility."

As for what fell "in between" these two specificity extremes, participants stated that such information might be something they would read when they "found time," but that it "wasn't their focus." As it stands, a significant portion of the information present on the WAI Web site appeared to our moderators to fall into this "middle" area: neither general enough to provide a thorough overview of the issues pertinent to Web accessibility (largely because "getting started" resources proved difficult to locate), nor specific enough to be useful to the participants, largely because they had significant difficulty locating information about specific techniques.

Finding 6: Participants stated that the WAI Web site was "essential"

It is worth noting that, while participants experienced significant difficulties navigating the WAI Web site, and significant difficulty locating the technique information they prized most, they uniformly stated that the WAI Web site contained "helpful and essential" information, and that they greatly "appreciated" its content. Their negative comments about the WAI Web site largely centered on its overall complexity and lack of information architecture and visual style consistencies. While they stated that they would like a "simpler, more efficient" Web site, they uniformly valued WAI's information and linked it directly with their ability to meet professional Web development demands.

Next: Pre-Test Interview Findings >>

Back to top