– DRAFT –
Silver Task Force & Community Group

Defining Conformance - addressing GH Issues

Jeanne: We want to work on today is to dig into some of the conformance topics that the silver task force and community group have been asked to work on, as part of WCAG 3 schedule.

Jeanne: The chairs suggested if we start looking at the issues that have been filed on conformance that do not have a group working on them.

Jeanne: I have a list of them in the sent agenda. That's a good place to start. Any Q on what we will be working on? Or comments?

Janina: Are we going through a github list? Can you announce issue numbers?

Jeanne: Yes. And I'll put a link to each one as we start.

<jenniferS> * Jennifer

Jeanne: Shawn is on vacation, if anyone notices that there is a q, bring it up.

Chuck: I'll monitor queue.

<jeanne> https://github.com/w3c/silver/issues/38

Jeanne: We have an interesting, old one. Issue 38.

Jeanne: About measurable test coverage. Should it be a silver requirement? Wilco, you made this comment 2 years ago.

Wilco: A while ago.

Wilco: I'll need a minute.

GV: Can you summarize the issue?

Jeanne: Challenge in WCAG 1 and 2, it's impossible to say anything about conformance, unless the page is fully tested. <reads issue>

Jeanne: More details on that, I can also read the whole thing. but it's an interesting place to start. We've talked about sampling as selecting specific pages.

Jeanne: I can see the need for large, dynamic orgs to be able to test portions of a page. Could we also allow that as well?

GV: This is an easy one to address. We are trying to create the ruler, not the rule. For example. If you are seeing if lumber in shipment meets weight, you create the legal weight the board needs to be.

GV: Process of how you test the entire truckload by random sampling is outside of the definition of a pound or foot or accessibility is.

GV: We are trying to say what constitutes a page in accessible. As you say, it's a sample. There's no one that does 100% sampling of anything. That's not a q we need to deal with. It is something that is good for a testing sub-committee.

GV: Maybe they craft best practices. The guidelines are the measure. We don't have to worry about sampling. We'll get deep into statistics. I don't think it needs to be part of determining what constitutes an accessible page.

<sajkaj> q+

Wilco: We are doing that, Silver has pulled it into scope. How many errors are acceptable, taking into account the severity. There is also talk about sampling in the sub group.

wilco: If I remember correctly. One of the challenges is to get an understanding of how much it does and how much we have already done. Those are real questions that would be useful to have an answer to.

Wilco: If we don't, everybody does it a different way. One orgs conformance claim will be vastly different.

<Zakim> Janina, you wanted to react to Wilco

Janina: What Wilco said and yes we put it in scope, part of challenge is to figure out process for defining conformance, you have reasonable expectation that if you run today and 6 months, you are in same range.

Janina: How do we say that process that you are applying the ruler, that we have consistent way of talking about view or page.

<Zakim> JF, you wanted to ask about sampling at page level versus site level

JF: To ask about sampling. The way Wilco explained, I imagined sampling images on page. I thought of sampling across the entire site, not sampling just the page.

JF: It seems to me that the page or collection of pages, that we cannot be in a position to determine that 90% of the images of alt text are sufficient, because the 10% are the most important.

JF: I think it's one of the existing gaps in WCAG 2.x today.

JF: A conformance report presumes to cover the entire site. I think we need to define at the site level not page level.

GV: We didn't require anybody make a claim. You don't have to make a claim that you meet WCAG. You can meet w/o making a claim. There isn't a need to define in the same way to define the claim, you can write your claim any way that you like.

GV: Requirements could come from the buyer. I'm thinking about webapps and sampling site and web pages. I think your "critical" statement on images is key.

<JF> +1 that would also be a great way to game a score

<sajkaj> +1 to importance of critical errors

GV: In silver there's a nice critical error concept that helps handle that. If we get into sampling, I'd assume we'd use standard industry processes.

Shadi: Brief response to Greg, and q. Greg, issue is the policy or law requires you have accessible web site, and someone is checking that. That's what Wilco is pointing out, people come to different conclusions.

Shadi: Regardless of WCAG requiring conformance claims, someone can come along with a different result.

Shadi: Wilco, we did observe with WCAG 2. question to me is, would sampling improve that, or is it inherent, or is it that the requirements are written so broadly, such as 1.3.1, covers so many things...

Shadi: Makes the statistical issue that if you pick different pages you will come to different results. If we do imagine requirements to be specific and testable, maybe sampling will be reduced, maybe people will get similar issues.

Shadi: When people do their own evaluation.

Jeanne: The topic is focused on one issue, I do think we need to have conversation on side issues, like sampling.

Wilco: Responding to John's question. I do think that this will work. I don't think one page at a time and only one page works that well anymore, given dynamically loaded in, pageless sites. We don't have an answer on that.

Wilco: Greg, I do agree with you, I'm not the biggest fan of us not putting everything into a single doc. I think WCAG 3 can be split up, cleaner approach.

Wilco: I still think that there's a need for testing things that are bigger than single pages.

Rachael: I agree with JF, trying to define conformance and explain what we mean by it. We've identified a core question, where does conformance go as far as measurement.

Rachael: ...a way of defining sampling and scope. At holistic level, we aren't just talking about websites. And at smaller, more granular level.

JF: Wilco, I don't disagree, we are in same place. My concern, your example is 90% sampling images. That could be used to game score. If I add 100 spacer.gifs, I can raise my alt text score.

JF: Too simple to game. I am concerned about scoping looking at that, whether we call it page or view. We've had that challenge for a while.

JF: We had long debate about "collection of pages". It comes down to scoping what has been evaluated. You have evaluated what you have scoped out. Defining a good scope is the challenge.

Jeanne: Reading some of existing comments.

Jeanne: Leonie Watson... <reads>

Jeanne: ...what happens if important stuff is on page that isn't covered...

Jeanne: Emma... <reads>

Jeanne: ...could a website make claim with full automation coverage...

Jeanne: Makoto <reads>

Jeanne: ...Japan uses representative sampling of pages...

Jeanne: Makoto, want to add anything?

Makoto: In Japan we adopted 2.0 as national standard. We wanted to have mechanism to make a claim for websites, so we defined web page only, we had to do something extra.

Makoto: We discussed additional guidance on how to test website or web pages. We do have the option of using random sampling.

Makoto: I think it works well. It can help this discussion.

Makoto: If you have any questions, let me know.

Jeanne: I'm suggesting we figure out next steps on issue.

<Zakim> JF, you wanted to respond to gregg

<Zakim> jeanne, you wanted to read comments on the Issue

<Zakim> SuzanneTaylor, you wanted to suggest that there are 2 steps that should be part of any written recommendations about measuring testing coverage (the accessibility review of a sample which is often sent to an expert and then accessibility QA which is often built into a product team's process)

Suzanne: If we define sampling, we should also define the QA process. What makes sampling work well, you identify QA process to identify as many pages as possible and that is what makes it work.

<JF> +1 to defining the sample

Suzanne: If we just define the sample, that can become the standard, but the QA part is what makes it work.

<JF> although I'd say "documenting" the sample

Jeanne: Good point. Do we have any suggestions for next steps?

Jeanne: Conformance options group is working on representative sampling.

Janina: We are still talking use cases. That has worked for us now. All started by specifying use cases. There is a use cases doc, I'll put in IRC.

Jeanne: One next step is for people to look at existing use cases for what we have.

<sajkaj> Conformance Options Sampling Use Cases:

<sajkaj> https://docs.google.com/document/d/1YgiOg3CZz-LAVxRT0CWUTWHzyVa3UrjqdU4NvoyUZ_8/

<JF> +1

<jeanne> chuck: Makoto has put forward a model. The next step could be drilling down into the Japan model and understanding it.

JF: I think Suzanne raised an important point. Defining, I'd say "documenting" the sample is a gap that we have today that I hope we can close in WCAG 3

Jeanne: Any other ideas on next steps?

Janina: Proposal about splitting into multiple documents, maybe that is something we should consider. We had a lot of conversations before I was involved, there's merit in that approach.

Janina: Not the first group to take a big problem and break it up.

Jeanne: I don't see we have an answer yet, trying to capture next steps. Summarize next steps in github issue and note that it was discussed in today's call.

Janina: If we were to say that we were willing to split, we'd need to talk AGWG through that. We can't decide here. It's a big shift, don't know how we'd go about that. Are Wilco and I are the only ones who think it has merit?

Janina: If you do decide to go that way, you'd want to know sooner rather than later.

Jeanne: This is going a long way away from this issue. I'd prefer to give that back to the chairs. For future AGWG meeting. Would rather discuss in that forum.

GV: Hopefully we can start to find some central reference document that captures the key issues. Pros or cons. Noting in minutes that it was discussed... which are short clips, you won't get richness.

GV: It would be helpful to see if we can't have people try to capture the "nubbins" of what was said, and then pass on the summary and synthesis.

GV: Otherwise we'll just keep discussing. Can we condense it down? Then moving up to AGWG.

Jeanne: Are you volunteering?

GV: Past chair, and new to the group. Might be challenging. Not in best position.

GV: may not be wise to do myself.

Rachael: Addressing Janina's, I added multi document approach to our future topics for AGWG. Q on timing is the issue. Breaking up now may cause immediate issues.

<Zakim> Rachael, you wanted to address next steps on Janina

JF: Chuck suggested spending time on Makoto's doc. Makoto I think you captured in your doc an accurate reflection of our current conversation. +1 to spending time with Japanese model.

GV: Which is the document?

<Makoto> I'm happy to share what we got in Japan.

<JF> Conformance model of JIS X 8341-3:2016 - https://docs.google.com/document/d/1SVDWemejSSBPPqJl4t_KBGeXWsFNWjHv0JW1y6RWgdg/edit#heading=h.ltnf1nkl7yg0

Jeanne: Makoto is referring to a writing that explains and translates Japan's model.

Rachael: Next steps. My summary is that someone will add notes to the issue. Were there other next steps?

<Rachael> Next steps: Review japan model, adding notes to github issue

Chuck: Reviewing Japan model was one that has momentum.

Jeanne: At same time that Makoto wrote his overview we also did the silver conformance group a couple years back proposed how to adopt WCAG EM to WCAG 3. That could be next step doc to review.

Janina: You found that for me recently.

Jeanne: This is...

Jeanne: I'll send to list once I find.

Jeanne: I didn't see any other next steps.

Jeanne: We were going to ask people to submit use cases to the conformance options document. Janina, did you drop a link to that?

Janina: Yes I did.

Jeanne: My thought is that we can move on and review another issue. May be larger than we can address in one day.

Jeanne: I don't want to lightly discuss and not go into details. May not be good use of time today.

GH issue 304

Jeanne: Issue 304, conformance claims need feedback mechanism.

<jeanne> https://github.com/w3c/silver/issues/304

Janina: Q is feedback to who? W3C? Other than collecting interesting experiences, I don't think it would be useful. I don't think that W3C want's to go into this space. So to whom?

Jeanne: <reads github issue>

Jeanne: ...learning from users feedback and experiences is useful...

Jeanne: <continues reading>

Jeanne: <continues reading>

Janina: I think we can close, this is covered in maturity model.

<Zakim> Chuck_, you wanted to ask if this is similar to peer review?

Jeanne: No, they are wanting every accessibility statement to provide feedback on each statement.

jf: I have been talking about using ERL and having linked conformance statements as part of protocol. I do believe that we have opportunity to have a standardized mechanism for making conformance claims.

jf: I hope that comes out in the end.

jf: That we have a mechanism for attaching declarative statements. EARL (Evaluation And Reporting Language) was created for this. I think we can have an entry with a feedback URL. Could contain feedback mechanism.

<shadi> Developing an Accessibility Statement https://www.w3.org/WAI/planning/statements/

Shadi: Side note, there is a WAI resource on creating accessibility statements, different from conformance claims.

<Jemma> https://www.w3.org/WAI/planning/statements/

<Jemma> Developing an Accessibility Statement

Shadi: That does include considerations of EU requirements. Also suggests providing feedback options. Q is to what degree is this a technical requirement for WCAG?

<SuzanneTaylor> +1 to having this live in the maturity model

Shadi: Sounds like maturity model, not really a measure of accessibility.

Suzanne: I agree that it's good to put in maturity model. In technical standards it will be rushed, and will be faded out. As part of maturity model, would be handled in the right way.

Jeanne: That's not to say that Chuck's idea is not a good idea. Peer review is an idea.

<SuzanneTaylor> Chuck: it's fine to include that as idea, but it is not my idea, just an interpretation of someone else's

Chuck: Not really my idea, just my interpretation of the issue.

Jeanne: I think we can close this issue as addressed in Maturity Model, but need to confirm.

Jeanne: Suzanne, do you know if it is included?

Suzanne: I do not.

Janina: I do recall such, not certain though.

Jeanne: Here's the maturity model doc. If someone can review and determine, we can start the next meeting with this item.

<jeanne> Maturity_Model draft: https://docs.google.com/document/d/1Y5EO6zkOMrbyePw5-Crq8ojmhn9OCTRQ6TlgB0cE6YE/edit#heading=h.yih1dxei45vc

<Zakim> Rachael, you wanted to suggest approach

Rachael: Let's write up a draft response and go through approval process.

Jeanne: Exactly! We can work on that next week.

ACTION: Suzanne will check if MM includes 304 and we will response next week

Jeanne: Thanks for participating. Will work on more next week.

<Jemma> bye. sorry for joining late.

<jeanne> s/ * Jennifer says for menopausal women gold is preferable because it doesn't get hot, thereby cooling the body.//

<jeanne> s| * Jennifer says for menopausal women gold is preferable because it doesn't get hot, thereby cooling the body.| |