agenda in e-mail list archives: http://lists.w3.org/Archives/Public/w3c-wai-eo/2004OctDec/0010.html
JB: The agenda is that we need to update evaluation resource suite this quarter. This is separate from a more massive update next year with coordination with WCAG 2.0 release. put some starter issues out for discussion and reaction piece. Outreach updates for starters and brainstorming on evaluation resource suit
SLH: Giving a keynote at a conference in Rochester, NY. Title is Web revolution through higher education. The role that higher education can play in web accessibility
No other outreach updates.
Evaluation Resource Suite
JB: How many people have participated in discussion on eval resource suite?
HB, CC, DS, SD, AA, JT: yes
JB: One of the parts of the eval. res. suite came out of a tools list. The eval and repair tools working group had worked on this. We attempted to provide practical guidance. To put best practice on summarizing results and eval. sites more consistently. Suggest best practice for multidisciplinary team. We had gotten good response when we sent it out for comments. One of the things that had nagged at me is that we don't deal with the role of people with disabilities in the evaluation. It would be helpful if we can update the suite. Finish integrating comments that have come in, clean it up, add one or two pieces of advice, such as dealing more clearly as to how to involve people with disabilities. One of the things we can't deal with now, is that we can't make it more precise. Because WCAG 2.0 will have precise testing criteria. We can focus on the precision of the test. We can say less, if WCAG and eval and repair WG say more. The question for today is for this quarter, what do we want to do with this suite? I sent out background that lists questions for discussion. The second e-mail also lists a draft page which came out of staff discussions and also some work with European outreach project. This might provide an interesting piece for some of our discussion. Start discussion by people who weren't here when we were working on it, if they have general questions on suite fits together or looks the way that it does. A second starter question, would be, for people who were involved, what are the things that you most want to do to this?
SLH: One of the things that I want to do is fix section on usability testing.
CC: Numbering system Under preliminary review, no. 2 two notes in a row, both parens. Consistent brackets or parens. Consistency of punctuation.
JB: Let's work on substance.
JB: Other substance issues?
JB: The second level, is labeled conformance but we talk about comprehensive evaluation. The UK disability rights report asked about usability. We felt that there should be a distinction between conformance and how comprehensively evaluated. Instead of having two levels, have 3 levels: preliminary, conformance, comprehensive. The third level would incorporate users. Any reactions?
AA: We run some danger that there is a 3rd level beyond conformance testing. A lot of people doing it because they have to, rather than want to. They will take minimum step, conformance testing. People could say I've conformed. It depends on how we prepare it. Warning flag.
DS: I feel that same way.
JB: disincentive
DS: Implies that you're not going to do all of it.
CC: User testing is needed in preliminary and conformance. User testing should be integrated into entire process. You need to do it in several steps.
JB: Let me clarify.In the preliminary, the goal is something that someone in an organization might do if they hear about web accessibility. Could do it sitting down at their desk, just to get a flavor of how off they are or not. Wouldn't have to downloadsoftware or have technical knowledge. Wouldn't have to work through networks of friends or colleagues. This would change how we conceptually view it.
CC: Could it be not a requirement?
JB: We worked hard to pare down preliminary review.
SLH: It would be good to keep in mind the reality and the ideal. The preliminary review is more like I need to see if my site is accessible. What can I do quickly? It's important to have this information. Rather than doing a new site where it's important to do eval. through the process.
JT: I agree.
JB: I want to play devil's advocate: conformance vs. comprehensive. How to deal with practical realities. A lot of people who are doing evaluations have expertise. There are organizations that are working to define it more carefully. In Europe there are groups that are focusing on eval. methodology. What does a technical evaluator evaluate? different tools. My understanding is that a lot of people aren't including users as part of conformance approach. I would be curious to hear if people are doing things regularly. Some include user involvement, some don't. Can we keep them grouped together? Should everybody be doing?
AA: This is our bread and butter (daily work). Three quarters involve users. Spend 2-3 hours with users. Not everybody wants this. Sometimes we do expert technical evaluation. This is what we call it to differentiate from what we do.
CL: This is how I deal with evaluations.
JB: What if we were to have quick, technical, comprehensive evaluations?
CL: This is a pretty good idea. Not everyone cares if they comply with W3C. They just want to be accessible.
NL: Technical review is misleading. They are all technical reviews. We do the best possible way is with expert. Then, we break into second layer. 508, WCAG, usable by people with disabilities. Technical will not cover that. In the middle could have conformance or compliance.
SD: Too many frameworks of review. Talking about users in the preliminary review is not mandatory, but recommended. If say that there are 3 types and people don't have users, then won't be able to do a review. If people say won't have a user, won't be able to conduct review.
SLH: If don't include people with disabilities, then it's not a comprehensive review.
SD: It's already included. Don't have to include users in preliminary level.
JB: When discussing this suite before, I thought that people who were doing a lot of evals, weren't including users. Maybe this has changed. This may have encouraged people to do that. It is philosophical. If it is promoted as a standard, should accomplish it's goal on the face, should be able to be evaluated on a technical level. If site complies, then does it mean that it has met technical requirements and we just encourage to include users.
SLH: a lot from industry on this. Important to separate the two. It's important to be usable, but first want it to be technically accessible. There are strong arguments for acknowledging that.
AA: Would technical requirements override usability?
SLH: No, they shouldn't need to, generally. But most can't do it all right away. First, make sure that it is technically accessible. I come from the usability world and agree, yes, it's better to do it all together. But won't always happen that way in reality.
DS: I hear what you are saying. Generally speaking, it is acknowledged in the documents. Have to go through various levels of conformance. How will the document be changed. I'm not sure that breaking it up, will express this.
JB: If we don't use word conformance, could say technical, comprehensive evaluation. The requirements angle is becoming more of a factor. This is raising the issue. There are groups who see the need to deal with compliance and then work it better. Separating things out might not discourage this. I'm trying to balance what Sylvie was saying, the more we could potentially confuse people who come for practical guidance.
NL: If we provide explanation of benefits and limitations of each method, people will understand.
JB: You can test to see if complying, but won't give info on usability by people with disabilities.
SLH: How usable is your implementation of it.
WD: Does it make any sense, releasing a page to people with disabilities without meeting the guidelines. It's painful. Knock all things that are wrong before releasing for review by people with disabilities.
JB: Clean up code and then let's do a comprehensive review.
SLH: In support of this, do all this stuff first. It does take more time and energy to involve users. Don't bring in users too early only to find out they can't get beyond step 1 and everyone's time is wasted.
AA: Won't do user testing until fix it up. Won't get beyond home page.
DS: Good way for users to develop skills. Use them right off. Use people with disabilities right at the beginning.
CL: I've just convinced a client to user testing before comprehensive eval. They were interested in getting user perspective.
JB: What if we were to have 3 levels. My guess, not matter we do, we will need to develop a subpage on how to involve people with disabilities in web site eval. That was near the top of our wish list. We want to provide more guidance. What if we were to say, we are mapping out a few options, you may want to look at guidance on involving people with disabilities. This might be the baseline. We explaining and describing different options that people could use and provide guidance on using options.
DS: We can say all things without implying that conformance is less than perfect. We know that this is a real world. If build into this a sense that there are certain limits.
HB: Harvey agrees.
JB: If we call it a technical eval, that is one idea.
SD: If have a technical eval, can say that web site conforms to WCAG? If you want to know about conformance, have to do a third level with users?
JB: With WCAG 2.0, we would approach differently.
SD: I try to work with someone who already knows the document. Does conformance disappear?
JB: We don't use conformance consistently. I'm not convinced that it's mapping out conformance eval. Good question. I don't know the answer.
AA: Somebody could do tech. review if knew enough about it.
SD: Someone could do a technical review, can't identify all issues without users.
AA: I work in an organization where a lot of staff have disabilities. I understand what they are facing everyday.
JB: Concerned that if include one person with a disability, think that makes review legitimate. May be able to review just for their own disability? Would they also be able to review for someone with cognitive disability or who is deaf? need to involve people with disabilities in comprehensive eval. Also need to think carefully what our assumptions are and how we explain it. When we have talked about this before, there's a strong consensus that we want to explain and promote people with disabilities in evals. We need to be careful.
DS: I support that. We cover mobility, deafness, vision, dyslexia. Each one of those people can't talk for somebody else. This is a profound question.
JB: Can't completely figure this out now. This is the hardest question. There are other questions to look at. There is a 1st draft document. We don't have a consensus yet.
WD: I think that we may have a consensus. Everyone agrees that have to involve people with disabilities. Question is how to phase, when, where to make site completely accessible.
JB: Start by saying, for the revision want to explore 3 levels: quick, technical, comprehensive eval.
JB: We would want to address concerns. Disclaimers for each level. Strong encouragement and reinforcement we recommend 3rd level which is user involvement. Does this capture what we having been discussing?
AA: yes. Taking into consideration Doyle's comments.
LC: yes.
DS: yes
PP: yes
SLH: yes. Want to share concern that we might be taking on too much here. This is a whole area in and of itself. How can we adequately cover it here?
JB: Shadi will be helping with editing.
CL: yes
WD: yes.
HB: go for it.
HBj: It sounds good.
JT: good
SD: yes need to have user testing page.
CC: I am comfortable.
NL: Need to include usability testing.
JB: I'd like to move to question 4: make the selecting eval. tools page part of the suite?
JB: There is a conceptual draft. Guidance on picking tools for review. Shadi will be working with eval. and repair tools WG.
CL: It is useful and almost necessary introduction to the eval. tools list. There is no intro on that document. This is a good introduction. Would not be out of place.
AA: concurs with Chuck.
JB: confusion about resource or content?
HBj: It looks good. But, want to read it closely. Not easily read.
JB: We spent a fair amount of time on selecting and using authoring tools. Turned some of the information into checklists about own software or purchasing software. I wish some of it were available in a list format. People may want to see what they can get out of it in a few minutes. Here's a first draft, people can read it in depth and look at a few requirements. Here's a pretty complete concept, what are the goals? audience? readability level?
HBj: If I need an eval. would should I be looking for? Some kind of checklist or listing. Choose a tool that does so-and-so.
JB: Section 2 is user requirements. How different are the requirements of the different groups? Different requirements for user profiles. Should this be the core?
SLH: Don't you have these requirements for all? Why are they broken out?
NL: When I looked at W3C, if I were a person who didn't know abou these tools, I would be overwhelmed. A checklist should help--positioned at the beginning so know what to look for. There are so many tools on the market. It's important to choose tools and the company that will consult with you. Experts who will be partners with you. IBM may have developed expertise. But, if don't want to develop this within the company, need to look for consultation and tools.
JB: You are not missing a checklist here. Even though user profiles are so broken out, it's not even addressing including consultation expertise and need for on-going monitoring. Would make sense to collapse user profiles and add in other things hat were mentioned.
DS: I have a different perspective. I would like to see that we develop expertise within the bank. I don't the applications as well as others. I look at list. Mozilla is missing. A big organization should be aware of open source Lynx. There is astrong need for accessibility demands be built into the enterprise.
NL: I agree with you. Should have expertise within the company. May not always have this luxury. Companies may have to go outside.
JB: How do we take what you are both saying and get the perspective inside this document? Does this mean an extra section at the back: considerations for on-going monitoring. Or, do we build into implementation planning resource suite?
AA: Already got a section in eval. document section 4. Build it into this document.
JB: I put a question about that section into the discussion questions for today. No. 5. Let's do a tangent on that. Do we want to move consideration for contexts? I hear some othe practice in some large organizations. May be some guidance we should capture. Have some of this discussion about considerations on its own page.
NL: I think that it will be good. It is an evolving trend to build accessibility into IT. Best practice is to have in-house expertise.
DS: I agree with Natasha.
HB: Small organizations won't have this. Should we advocate for resources to do this?
JB: What we will do with the suite, moving considerations to separate page and expanding discussion? Anyone speak against that? Explore moving considerations onto a separate page. Want to avoid scope-creep. We could capture requirements as a wish list and then figure out what is realistic and then what we want to do in 2005. Reactions? We have a tentative decision moving considerations to specific page. And expanding to discuss integrating accessibility expertise into organizations.
JB: We were trying to get consistent comments for recording comments. We came up with something that looks good. But don't have any idea if anyone is using it. What could we do to make it using it more usable. Any reaction?
PP: It's a very good template. This template may be written by [?]. We have a good law but not good guidelines. We are not so good at explaining. It's not so good to follow techniques in our law.
JB: Template encourages you to follow good practice. Other reactions? We could make the template downloadable.
CL: I was considering using it as a basis for an upcoming project. It may not be all exactly appropriate, but it's a good template. AA: We don't use it in the fomat, provide summary, background, page selections, who involved, etc. It's a fairly standard way of reporting.
SD: Yes. We use a similar format. JB: Should template be more portable, downloadable?
HBj: Didn't we talk about this? Henk and Eric made the tool/form.
JB: We were not able to complete this. That came out of the EO discussion. Henk and Eric had done an online form of this and it had other stuff. Have it available in two versions: downloadable template, online form version. Would need something on own server to back up form. Could be combined with eval. and repair? There's a huge accessibility party in Amsterdam today. Some people involved in another project may be coming into our discussion. WART Web accessibility review tool. This was an online form in W3 space. It allowed recording of review. But there were problems with the tool. Not guidance on what to do what, technical checks, summarized supposed faults without guidance, generate a complaint for the web master of the reviewed tool. Complaint letters looked like they can from W3C. This might be interesting to look at in the future, if developed more carefully.
NL: There are so many different tools. They are complex. I don't just see how you could create template. I think that what we have is good, it's a template outline.
CL: I agree with Natasha. It's a template of a template.
JB: This is what eval. template includes. If you're not from a large organization, they may to use it as is.
SLH: They would know that.
JB: Best thing would not be downloadable. Want to make sure that we're thinking about small organizations. I talk to a lot of non-profit organizations and don't know how to evaluate their sites. What can we help them do?
HB: Sending e-mail to responsible person. Making limiting number of suggestions and use eval. tools.
JB: They might look at technical comments. Then, might look for free tools and do a few more changes. It's a bug fix and awareness approach.
HB: Charity organizations appreciate this.
JB: Would it make sense to have additional subpage --have orientation basics. Part of an advocacy resource suite. The web accessibility reporting tool combined crude review and something that is action oriented. Would this be useful as part of this suite? Or outreach or advocacy oriented packet? Let's say that you're trying to use someone's web site and it's not accessible. Why don't I just shoot them an e-mail? Is there anyway we could combine anything with eval. resource suite with suggestions with how to use eval. information as part of outreach. Maybe this belongs to another suite.
NL: another suite
HBj: Did we talk about this before? standard letter that says there are problems and could you do something about it.
JB: I don't think that it is out of our deliverables list. It could be on wish list.
SLH: I thought we talked about it when we talked about user materials recently.
HBj: about 3 years ago.
JB: I will put it on wish list.
HBj: Could do something for advocacy. A combination of checklist and this template about eval. report. It would be nice to have something. Doing actual checking and then putting into a piece of paper. It takes a lot of work. Communicating about what you found. Review Teams page. This one had been more challenging than eval. processes. We are describing an ideal. We talk about composition of review teams, expertise, important considerations. Is anyone using this?
AA: No. This is probably an ideal. We have a team of 4, with several user testers. One individual takes responsibility and then will consult with the rest of us.
JB: Isn't this close to this?
AA: yes. We engage people who cover that.
JB: relevant ideal?
AA: yes
WD: yes
SD: I haven't looked at it for a while. I should read it through.
JB: (description of document) It's pretty broad.
LC: Lower on priority list.
WD: Comment on suite, as a whole. Whether you're in large, small organization. Beginning or doing it for awhile. For someone who is doing it for the first time, need where am I in the process?
JB: Should the front where am I in the document? This would break links. Do we something that orients people more before details? I would like to write an outline about priorities of what we would like to change. What is realistic for this quarter and what we should do in the future. Could people look at draft document that Shadi developed and we could go into more depth in an upcoming call.
15 October 2004