Post-Test Interview
After participants completed the tasks, we asked them a series of summative questions about their experiences using the WAI Web site. Participants first gave a "letter grade" (from A, exemplary, to F, failing) for the entire Web site.
Grading the WAI site
In general the participants were pleased with the content on the Web site, but were frustrated by how difficult it was for them to find information. In addition, the participants expressed concern with the technical nature of the information itself, stating that it was difficult to comprehend, and that they were concerned it was impenetrable to "someone with less expertise in Web development."
The table below summarizes representative participant responses.
| P# | Grade | Comment |
|---|---|---|
| 1 | C | It has a lot of really good information but it is hard to get to and decipher. |
| 2 | B-/C+ | All the information was there, but it was hard to follow on some topics. Also, depending on the audience, if you are gearing towards developers, then I would look for keywords specific to developing sites. It needs keywords, instead of leafing through the whole site. |
| 3 | C | It is hard to navigate, words are repeated over and over. I feel al little bit frustrated navigating it. I feel like I went in circles. There was no easy way to get home. |
| 4 | B+ | I think it covers everything that I want to know. It has good content. It suffers from not using enough color for navigation. Navigation on this site is often different from 99% of what you see out there. |
| 5 | C | The interior and guts they haven't modified much. It feels like the page gets developed as information becomes developed. The overall site is not taken into consideration. There's too much information. There's so much of 'this is our news' 'this is how we put stuff together,' and not 'this is what we have for you.' |
| 6 | C | Depending on the topic, some is written well for people outside of the technical realm, but once I get to the W3C guidelines it gets confusing. |
| 7 | B | (No comment) |
| 8 | B | I do listen to the entire page normally, but I think things could have been organized better. It would be easier for a screen reader if you didn't have to read through everything and instead have the meat of the stuff first. |
We also asked participants to name three words or characteristics to describe the WAI Web site. Many participants used the same words to describe the site, such as "information-rich," "technical," and "overwhelming." The table below includes responses from all of the participants.
| P# | Words or Characteristics | P# | Words or Characteristics |
|---|---|---|---|
| 1 | Overwhelming Confusing Exclusive, like a club I don't belong to |
5 | Overwhelming with information Content-rich Organized |
| 2 | Text-driven Information-rich Slightly difficult to navigate |
6 | Readable Technical |
| 3 | Low budget Overwhelming with information |
7 | Useful Informative Insufficient |
| 4 | Information-rich Well organized Stark |
8 | Extensive Challenging Technical |
We asked participants to also describe three things they liked best about the WAI Web site as well as three things that they liked least about the WAI Web site. Again, participants spoke positively regarding the wealth of information available, but felt the information was difficult to find and was often too technical. The table below details participant responses.
| P# | Likes | Dislikes |
|---|---|---|
| 1 | Getting started Quick Tips Amount of information available |
Web site is technically exclusive to novices Hard to interpret information Lack of organization |
| 2 | It is informative The W3C stamp shows its reliable |
Lack of a search specific to the WAI Content seemed cramped on some pages Missing some information like FAQs |
| 3 | Soothing colors Easy to tell what is a link and what is not |
Too many different layouts on different pages The repetition of words made me feel like I was reading too much |
| 4 | Checklists Techniques Slideshows |
Poor navigation Lack of information about joining a group Poor content separation |
| 5 | Amount of information Commitment by WAI to make things accessible |
Difficulty navigating Unclear organizing principles Not knowing when I am going outside the WAI |
| 6 | Language is good Use of headers Font size is good |
The logo is confusing because it links to the W3C Needs more consistent headings |
| 7 | Easy to navigate Pages are easy to read Useful information |
No search for just the WAI Some navigation on lower levels is difficult |
| 8 | Alt tags in correct places Site was accessible |
Too many similarly named links Before you get to read information you are looking for there is pages of unnecessary text No search feature |
Participants offered one significant change they would like to see on the WAI Web site. The most commonly cited changes were to improve the navigation, offer a getting started section and to add a search feature that was specific to the WAI Web site and not the entire W3C. Some responses are listed below.
- Improve the navigation by offering an option for people you just want to lean to make forms accessible. I would like there to be a more user-friendly presentation.
- There should be only one section on every page for the navigation.
- I want a how to get started section. There should probably be a button that would kind of give me as a reader how to surf the page, if you are new to Web accessibility we suggest you first read this, and first read that. The homepage seemed to only have half of it. It had links, but there was so much stuff to link to.
- I would add a search for just the WAI and not the entire W3C.
Lastly, we asked participants whether they would return to the WAI Web site in the future. Despite the difficulty many participants encountered when searching for information, every participant strongly affirmed they would indeed return to the Web site, largely because the content is so valuable. Some representative responses from participants are listed below.
- I have no choice but to return. The site is great for accessibility information.
- Yes, I would return. The site has a lot of information that could be helpful to me. It answers questions as to making sites more accessible or what people in the industry are doing.
- I would return to the Web site, because it has all the content that I need. The information is in there, it just is not easy to get to or interpret. It's there, and if I really need it then I will probably find it.
Post-Test Evaluation Survey Results
We asked participants to fill out a survey that asked them a series of questions about the Web site. They then rated their level of agreement or disagreement with each statement. Survey responses are provided below.
| Participant | 1 | 2 | 3 | 4 | 5 | 6 | 7 | Average |
|---|---|---|---|---|---|---|---|---|
| The homepage is attractive | 3 | 4 | 5 | 2 | 4 | 5 | 5 | 4.0 |
| The overall site is attractive | 3 | 2 | 4 | 2 | 4 | 4 | 5 | 3.4 |
| The site's graphics are pleasing | n/a | n/a | 3 | 2 | n/a | 5 | n/a | 3.3 |
| The site has a good balance of graphics versus text | n/a | 3 | 2 | 2 | 3 | 3 | 6 | 3.2 |
| The colors used throughout the site are attractive | 6 | 4 | 6 | 3 | 4 | 6 | 3 | 4.6 |
| The typography is attractive | 6 | 5 | 6 | 4 | 6 | 6 | 6 | 5.6 |
| The homepage's content makes me want to explore the site further | 4 | 6 | 6 | 6 | 5 | 6 | 5 | 5.4 |
| It is easy to find one's way around the site | 3 | 4 | 2 | 6 | 3 | 3 | 6 | 3.9 |
| You can get information quickly | 2 | 4 | 2 | 7 | 2 | 2 | 5 | 3.4 |
| It is fun to explore the site | 2 | 3 | 5 | 6 | 3 | 5 | n/a | 4.0 |
| It is easy to remember where to find things | 4 | 5 | 2 | 6 | 3 | 4 | 6 | 4.3 |
| Information is layered effectively on different screens | 4 | 4 | 3 | 6 | 3 | 3 | n/a | 3.8 |
| The homepage is attention-getting | 6 | 3 | 1 | 3 | 3 | 3 | 3 | 3.1 |
| Information is easy to read | 2 | 3 | 3 | 6 | 3 | 6 | 6 | 4.1 |
| Information is written in a style that suits me | 2 | 5 | 3 | 6 | 3 | 3 | 6 | 4.0 |
| Screens have the right amount of information | 5 | 4 | 3 | 5 | 2 | 5 | 6 | 4.3 |
| The site effectively communicates the company's image | 5 | 6 | 4 | 6 | 2 | 4 | 6 | 4.7 |
| Information is relevant | 5 | 7 | 6 | 7 | 6 | 7 | 6 | 6.3 |
| The site is designed with me in mind | 3 | 6 | 6 | 6 | 4 | 3 | 5 | 4.7 |
| The site's content interests me | 6 | 7 | 6 | 7 | 7 | 7 | 6 | 6.6 |
| The site's content would keep me coming back | 6 | 7 | 6 | 7 | 6 | 7 | 6 | 6.4 |
| The site has characteristis that make it especially appealing | 3 | 5 | 3 | 6 | 4 | 3 | 5 | 4.1 |
| The site reflects progressive, leading edge design | 3 | 6 | 4 | 2 | 2 | 2 | 3 | 3.1 |
| The site is exciting | 3 | 2 | 4 | 4 | 2 | 2 | n/a | 2.8 |
| The site is well-suited to first-time visitors | 4 | 3 | 2 | 6 | 1 | 2 | 5 | 3.3 |
| Ths site is well-suited to repeat visitors | 6 | 4 | 5 | 6 | 2 | 5 | 6 | 4.9 |
| The site has a clear purpose | 6 | 6 | 7 | 7 | 3 | 6 | 7 | 6.0 |
| It is always clear what to do next | 3 | 4 | 2 | 5 | 1 | 2 | 7 | 3.4 |
| It is clear how screen elements work | 3 | 6 | 4 | 7 | 5 | 3 | 6 | 4.9 |
| Mistakes are easy to correct | 4 | 5 | 4 | 7 | 2 | 5 | 6 | 4.7 |
| Average | 4.0 | 4.6 | 4.0 | 5.2 | 3.4 | 4.2 | 5.5 | 4.4 |
Key
| Red | Yellow | Green |
|---|---|---|
| 1.0-2.9 | 3.0-4.9 | 5.0-7.0 |