– DRAFT –
Protocols-052722

Evaluating Procedures Proposal

https://docs.google.com/document/d/1W_5H0MCoKzGaD9XCxgzdqZ-1TiVCXHVipE_vNnG2DOQ/edit

jg: <reads from proposal>

jf: what is goal for this review?

jg: send the proposals to the WG

don´t expect consensus of the subgroup, we have two areas of consensus

jf: will we review first?

jg: can do some cleanup before submitting

rbm: I see next step as solicit review from the parent groups of both proposals

we can refine them if it´s productive

jf: I see lots of questions open

jg: because we have two areas of consensus, we expect that

taking to the larger group will give us more points of view

don´t think we will be able to agree here on a single proposal

per RBM we can start to prepare the editors´ notes of outstanding issues

Points for Protocol Proposal

https://docs.google.com/document/d/1UgoMz3OPyoEVLbU4uCU5F5K6aEM7E1rii6oCaWqQy50/edit

<jennifer_> I thought I asked a question about the two approaches, and the response was that the two approaches would work in tandem.

jf: my understanding of conformance is that you perform ACT-style tests

failing tests lowers the score and potentially conformance level

losing points is demoralizing

think protocols can be used to (re)gain some points

define what success looks like given unknown parameters

promising to adopt best practices docs gives you points

<Zakim> jeanne, you wanted to ask that a concern be listed that protocols can be gamed to pass a minimum on a site that isn't accessible.

protocol gives content creator info to make well informed decision

jeanne: should note concern about raising score where it @@

<Zakim> Rachael, you wanted to clarify what would go forward with this proposal

jennifer: would otherwise fail a minimum bar of objective tests. It makes it more vulnerable to gaming

rbm: we´d add all concerns to either proposal

Discussion of proposals

jf: my concern with evaluating procedures proposal allows self-documented protocols

could lead to toothless protocols

orgs do this all the time but it doesn´t necessarily improve accessibility

jg: the evaluating procedures proposal includes requirements for self-documented protocols

and states that they must be public so they can be inspected

jf: who vets them?

jg: protocols relate to specific outcomes and are measured in context of that

jf: don´t see how we evaluate stuff that can´t be measured

it´s still subjective

jeanne: how does Points for Protocols provide the rigour? I see the same question applying to both

<Zakim> jeanne, you wanted to ask how Points for Protocols addresses this?

jf: protocols are vetted by AG

hence subject matter experts

jeanne: that´s not clear in the proposal right now, think it needs editing

jf: AG doesn´t have to author all protocols

but it reviews them before they can be used to gain conformance points

jeanne: that needs to be written up

<jaunita_george> https://docs.google.com/document/d/1UgoMz3OPyoEVLbU4uCU5F5K6aEM7E1rii6oCaWqQy50/edit#

rbm: might be useful to compare the proposals in table form to find the similarities and differences

<jennifer_> +1

<jeanne> New document

jennifer: I think we should come together with a harmonized proposal

<Rachael> Just a note that multiple proposals at the exploratory phase is built into our process

need to be nice to each other, hear one another, approach with best of intentions

jf: @@ haven´t decided what we´re doing with conformance

public declaration of adopting protocol

accomplishes @@

the public promise means you have to follow through on it

in a legal situation

ack \

jg: accessibility statements aren´t generally enforceable

jf: one of the questions is, what will regulators accept

<jennifer_> * Jennifer needs access

<Rachael> Chair note: We are working our way towards conformance conversations but we are still not there yet so pre-discussing has not proven to be a productive use of time

jg: do we want to develop a single proposal, or submit multiple?

jf: are we even working on the same thing?

maybe we´re working on different things that both use word protocol

<jennifer_> * Jennifer: Thanks, Jeanne. I refreshed before, but now I do have access.

for me, the question is what @@

mc: I think we won´t form consensus on a single proposal, thing broader input from the WG is needed

raising questions and feature comparisons is useful

jennifer: think we should be able to come to agreement

rbm: discussion of conformance keeps side-tracking us

there are unknowns that we need to work around, until we´re able to get to them

<JF> +1 to Jen

jennifer: all work should halt until conformance sorted out

rbm: we tried addressing conformance first, but didn´t have enough base information to have non-circular conversation

the chairs have to find a way to work through things

having multiple options is ok

we can compare, discuss, merge, etc.

people may strongly favour a proposal, but we have to develop an overall consensus

chairs are handling this with a strict schedule, which we have to use to be able to move forward

jennifer: can we put a bracket around conformance-related aspects of proposal

rbm: thinks the points for protocols proposal has done that

it will be part of what we forward to group

we need to make progress, we´re recycling discussion

jennifer: think a group should be able to work together

and evaluating procedures group has developed something different

jf: we have two unique ideas, both have some value, neither are fully baked

should ask the WG to evaluate them independent of each other

with both using the word ¨protocol¨ that´s harder

see conformance being addressed in the evaluating procedures proposal as well

<Zakim> JF, you wanted to note that an end state does not have to be one or the other - it may be both

mc: think we were ¨bracketing¨ conformance but hadn´t actually put brackets in

think we need to have placeholder conformance info for the proposal and as fodder for later discussion

it´s ok to rename if there is concern they are being presented as exclusive of each other

jg: what about renaming?

jf: ¨development protocols¨ and ¨evaluation protocols¨

jg: development would be points for protocols, evaluation would be evaluating procedures

<jaunita_george> Proposed Resolution: Bring both proposals to AGWG as Development Protocols (Points for Protocol) and Evaluation Protocols (Evaluating Procedures) and work to get a table together outlining the differences/similarities ahead of the AGWG meeting we'll be discussing it.

<JF> +1

+1

<jaunita_george> +1

<Rachael> 0

<jennifer_> 0

<jeanne> +1 for two propo9sals, but I don't like the new names. I don't think they accurately describe the proopsals

<jennifer_> Agree with Jeanne — I don't understand them at all.

<Rachael> What about procedures and protocols?

RESOLUTION: Bring both proposals to AGWG as Points for Protocol and Evaluating Procedures and work to get a table together outlining the differences/similarities ahead of the AGWG meeting we'll be discussing it.

jg: we´ll prepare to present these to the AG WG

meanwhile let´s fill out the table of similarities and differences

jg: let´s meet next week (12:00 pm Eastern) to review the table