Meeting minutes
Clarifying goals for protocols
<Jaunita_George_> *wcag3-protocols
Jaunita_George_: There are two views of protocols. The first is that protocols are extra, above and beyond standards. The other view is that protocols help define conformance at some level.
… I'd like to start by coming to a common understanding.
Bruce: Is the protocols page wiki correct? I've been following some of this work tangently. I'm not clear what our definition is for protocols.
… I may be overloading the term due to previous group work
<bruce_bailey_> https://
Jaunita_George_: I may not be characterizing this correctly.
There is a view that protocols go above and beyond conformance and another that they contribute to conformance.
JF: My vision. When you say standards, I hear requirements. In WCAG 3, we have requirements that will meet ACT format.
… Those will be measurable, repeatable. On the other side are protocols. Protocols can support those but the defining feature for a protocol is that its educational in nature. Gives policy makers information on the right thing to do. Hard to evaluate subjective determination. Informed opinions. Make relatively good decision due to being informed.
… something like plain language gives you information but doesn't say "don't use multi-syllable words"
… point is to communicate effectively. We have a goal and a document on how to acheive it but doesn't say you must do this.
<Jaunita_George_> Rachael: We have talked about the fact there are actually four levels of tests
<Jaunita_George_> ...don't want to oversimplify the types of tests in WCAG 3.0
<bruce_bailey_> Rachael: Whether someone has conformed to the guidelines or not
Rachael: ACT is working with Scoping on how to write tests for test scenarios.
Jake: When ACT group started, focused on purely objective tests. That has changed. There is now a lot of subjectivity in ACT that can't be tested automatedly. New rules are way more subjective. ACT moved its direction. Maybe would be good to explain a bit more. Not what it was 2-3 years ago.
… there will be more subjectivity in act tests. There are even more.
Jaunita_George_: It would be a much smaller number of requirements to talk only about protocols and what ACT was a few years ago. There are two concepts here that were expressed earlier. I'm not sure if we want to focus on both or only 1.
<Zakim> MichaelC, you wanted to differentiate protocol from guidance
Michael: The way I'm hearing things discussed. We know we don't have a consensus definition of protocols. The definitions in dictionaries don't match. I hear there are two different goals and we are using the same word for two different goals. What I hear John describing as protocols I define a guidance. What you are doing that is in service of the goal.
… What I have been using the word protocol for is the steps used to achieve the goal. Both are needed. Steps without the goal are meaningless.
<JF> Operational Protocols means the administrative policies and procedures of an EMS System or that provide guidance for the day-to-day operation of the system. https://
Michael: Goals without the steps are also not useful. We need to clarify terms and decide on what we are focusing on.
JF: I pasted in the definition for operational protocols. [reads above]. That is as close as I can to the concept. They define the goals. They may provide steps or user stories and examples. The fact that its a step by step process. May be applicable or not. The commonality of a protocol is that it provides guidance. You have a situation similar to this. These are the things you are looking at. We can give then the tools but at the
end of the day, they have to evaluate the situation and arrive at a decision. Its a contextual decision. If we use the same tools the final destination should be relatively close and meet the goal.
… uses experts creating alt text example.
<bruce_bailey_> JF: avatar: Testable Measurable Repeatable
<Zakim> bruce_bailey_, you wanted to endorse what chairs see as protocols
bruce_bailey_: I had the wrong impression coming into the meeting. I agree with testable, measurable, repeatable I don't think that's the biggest challenge facing WCAG. In 2.0 we had a suggested way to make conformance claims that hasn't been very popular. We've tried to create objective test criteria. The 100% all the time conformance model has been a challenge.
… I think protocols is exactly what's needed.
<Zakim> MichaelC, you wanted to say ultimately I´m intereseted in the parts that impact conformance claims
MichaelC: For me, what I'm interested in solving is making it possible to include guidance in WCAG within the conformance model so organizations can claim conformance against something that can't be repeatably tested. Perhaps we define the guidance and put in how to fit into that format. The guidance isnt' helpful unless we have a to evaluate it. That "test" is the steps that need to be taken so someone can determine whether the
guidance has been followed or not.
… anything that falls outside of the conformance model is not a priority for my time.
<bruce_bailey_> michaelC: protocols is sequence of steps for a process to follow WCAG
Jaunita_George_: If outside guidance is used to help support standards, we will need to be careful what we choose. Just stating something was done on the website. Smaller orgs might do so. Larger orgs may treat it as a loophole.
JF: Michael, we don't "test" for protocols. We "evaluate". Its subtle but important. A test assume we end up with the same results. An evaluation is a subjective determination of what the results look like.
… outcomes will be different but if we can evaluate whether they have used a protocol, then we have something we can evaluate our work against.
… Part of the problem is that we don't have a conformance model flushed out.
… I envision an conformance model that allows protocols to score based on points.
… evaluation is harder because we have to take it on faith. We don't have the ability to do a rigid test.
JakeAbma__: I would also like to respond to the Michael's remark. He said that guidance is not part of the conformance model. It depends on how we make the conformance claim. I sent part of an ISO standard. ISO is all about processes. The first two paragraphs are [reads paragaraph]
<JF> +1 to this Jake - this is the idea
JakeAbma__: its all about documentation.
… provide evidence to support the claim.
… need to document it. The checklist is about what you need to document and how.
<JF> the "public assertion" piece in my pitch
Jennifer: Going forward, our WCAG guidelines will have things that are objective (boolean). Then there will be subjective things. References GEL. Showing your homework. I don't think protocols should be about whether someone could pass a protocol. Should meet minimium levels of conformance through something that isnt' a protocol.
<bruce_bailey_> Jennifer: protocols used to show your homework
Jennifer: not baseline.
<Zakim> MichaelC, you wanted to clarify what we test or evaluate and to note work on protocols is hopefully part of filling out the conformance model
Michael: For me the difference in separating the concept of the guidance, which is what we want to make better, from the protocol, which is the steps to follow the guidance. I also use test and evaluate as you do. The reason we have protocols is to evaluate what we dont' know how to test.
… We use the protocol to provide the steps that need to be done and test that the steps have been followed. Did you follow them well is another question. A protocol could have evaluation as part of it. In addition to training staff and keeping protocol front and center. The test would be whether you perform the evaluation. The test is on the steps because its on the only thing that is testable.
… I don't think we can get tied up on the lack of the conformance model. We need to focus on this.
<Jaunita_George_> ack
Jaunita_George_: It does need to be something that can be used as an evaluation tool. It is also to address SC that can't be included in current model. We have to be careful not to create a loophole.
<bruce_bailey_> +1 to MC that work on Protocols can advance absent a specific 3x conformance model
JF: In my mind it keeps getting linked to the conformance model.
+1 to MC and Bruce
… you have 100 tests. You run 100 tests. You run 80. You lost points.
… we have subtractive tests. Jaunita had talked about loopholes. Protocols are additive.
… you dont' want to adopt it, ok. Note quite voluntary but why wouldn't you want to adopt. When you talk about baseline, I'm not sure it doesn't get adopted into baseline. Participating in protocols should still add.
… carrot and stick. We have to provide mechanisms to encourage people to strive.
… we just put more stuff out there.
<Zakim> Rachael, you wanted to say I think we are all in agreement, propose resolution and to ask why we can't "test" whether a protocol has meet the documentation requirement
Draft RESOLUTION: Protocols evaluate success based on whether an organization meets documentation requirements or the steps required
<Jaunita_George_> +1 (from Jeanne)
<Jennifer> -1
<bruce_bailey_> +1 for me
<Jennifer> Suggestion: An accessibility protocol is a list of considerations used to evaluate the accessibility of commonly subjective success criteria.
Michael: I agree with John's point that we may want to use test for that resolution.
<Zakim> MichaelC, you wanted to go back to feeding into conformance and to say I see protocols as our way *out* of AAA
JF: Other way around. We want to evaluate things we can't test.
MichaelC: The protocl makes it a test.
… the stuff leading to the outcome. The result is somethign you evaluate but the steps can be a test.
<Jennifer> No, the protocol does not make it a test, in my opinion. The protocol is how you show your process / your homework for how you met the subjective success criteria of WCAG.
<bruce_bailey_> I think this describes a protocol: https://
JF: I can read your public assertion and documentation. Me as a third party. I can read the statement and see what your claim is and then look and evaluate whether you met it or not. You evaluate against the claim.
… the entity claims to have guidance. I can then go and read protocol and content and then evaluate.
<Jennifer> @bruce_bailey - I don't agree that the section508 list equates to a protocol. That is more of a list of questions to document accessibility efforts.
<bruce_bailey_> i am okay with "evaluate" in draft resolution. I am not currently in favor of replacing "evaluate" with "test"
MichaelC: Perhaps use a different term
<JF> I am with Bruce
Draft RESOLUTION: Protocols address success based on whether an organization meets documentation requirements or the steps required
<bruce_bailey_> i don't think protocols should / need to use the word "success criteria"
<JF> =1
<JF> -1
<Jennifer> -1
<bruce_bailey_> i might now be thinking of protocols at too high level a meaning
<JakeAbma__> -1
<bruce_bailey_> address success is very different than address success criteria
<JF> how does one test documentation?
Rachael I think the core question is whether once we have documentation, we either 1) only test the documentation or 2) evaluate the results against the documentation
<JF> Protocols are not requirements
Jaunita_George_: I am not comfortable with organizations just testing documentation based on whether it first conformance.
… lets focus discussion on what to do next.
<Jennifer> -1
<JakeAbma__> +1
Jennifer: JF is here. Can we verify the original intent. The original intent was to evaluate the subjective. You take some other document, pick that, check the work using that, document the work doing that, doesn't get reviewed unless there is legal issue. Other than that, its just did you do your homework?
Next steps for the working group
Jaunita_George_: What are our next steps as a working group. Draft resolution but still two ways this could go. Do we want to focus on both ways in parrallell? We develop proposals for the larger working group? Is there a draft resolution there to get consensus?
<kirkwood> agreed with jennifer subjective it is also needed for processes and barriers that are two complex or even too new to be covered
<MichaelC> +1 to parallel work so both approaches can mature
<MichaelC> and be compared better
<Jaunita_George_> Draft resolution: Should we split the group to focus on both ways to use protocols?
+1 to both approaches
<MichaelC> +1
<Jennifer> -1
<JakeAbma__> -1
<Jennifer> * it's not clear to me what the "both ways" is, as that wasn't captured.
Jennifer: request for clarifying both views.
<Jaunita_George_> ack
<Jaunita_George_> • View 1: Protocols are an above or beyond any type of conformance, aka “extra credit” • View 2: Protocols are a way of meeting the guidelines, and how they fit into conformance is TBD
<Jennifer> +1 to JF
JF: Again, I struggle with testing a subjective evaluation. You can't test something subjective. If people want to try taht. Lots of ideas emerging. One of the things to keep in mind is that industry wants less complexity, not more. WE are making it more complex than it has to be.
<kirkwood> you can test with a group poll/survey. for ‘subjective’ criteria
JF: is it scalable to large industry. Going in opposite direction.
<Jennifer> I will definitely vote -1 for View 2. Protocols shouldn't be a way of meeting the guidelines — only a way of showing how the subjective guidelines were met — they're subjective, they can't be T/F met.
Jaunita_George_: Get folks thoughts but we may not be quite ready get.