Meeting minutes
AC: everyone type "present+" please
Conformance introduction
AC: Welcome
AC: focused on provisions more in 2025 and need to focus on conformance now
[showing slides]
… want to get us all on the same page
… members of subgroup are here also
… goal of conformance is that it can be used by orgs to talk about accessibility for their site and can be used by regulators
… key issues
… people think conformance = Accessible
… we say otherwise, but it is a source of confusion
… products (and sites) rarely meet WCAG 2 conformance
… makes it hard to use in procurement
… regulations apply WCAG's page-based conformance to complete sites and apps
… often not realistic
… confusion of conformance with compliance to a law
… laws can add/remove items, include other mechanisms to meet
Slideset: https://
<Charles> one of the biggest issues i recall from our original silver research was understanding the text of the guidelines
<ShawnT> I have to often remind people that conformance and compliance are different things
<Lisa> +1 to Jenny
JD: points out that there is a disconnect between our conformance details and what regulators use
Wilco: issues due to 3rd parties has many different possible causes and solutions. Question of who is responsible.
<Charles> the biggest confusion i hear between conformance and compliance is not understanding that conformance is a snapshot moment in time. one can be in and out of conformance fluidly for any page with dynamic content.
LS: Speaking to conformance = accessible. Conformance models don't directly speak to the user experience and needs as well as they should.
… as a result, many users are overlooked.
<Zakim> kevin, you wanted to say that responsibility is not within conformance but compliance
<Adam> +1 to kevin
KW: Back to Wilco's comment. Don't believe that the conformance model should speak to who is responsible - it just addresses the level that a site meets the provisions.
<Lisa> some users should be many users and user groups
<Jon_Avila> WCAG understanding conformance requirements says Statement of Partial Conformance - Third Party Content
<Jon_Avila> Web pages that will later have additional content added can use a 'statement of partial conformance'. For example, an email program, a blog, an article that allows users to add comments, or applications supporting user-contributed content. Another example would be a page, such as a portal or news site, composed of content aggregated from multiple
<Jon_Avila> contributors, or sites that automatically insert content from other sources over time, such as when advertisements are inserted dynamically.
<Jon_Avila> In these cases, it is not possible to know at the time of original posting what the uncontrolled content of the pages will be. It is important to note that the uncontrolled content can affect the accessibility of the controlled content as well. Two options are available:
GV: Rather than talk about responsibility, we can say that we assume that some users have AT. In the cognitive space the availability of AT has been an issue, but not all blind users have access to AT.
… We have assumed that when we make content programmatically determinable that people can access it and that is an assumption
SAZ: is this intended to be an exhaustive list? Do we need to add that these are examples at the bottom of the list?
AC: We had a lot of things in the subgroup docs and I drew these from there.
… wanted to get agreement that these are major issues
CH: Key issue from research was overall for WCAG about understandable language and this impacts everything including conformance
GV: Last one. Many people don't understand that AT are user agents.
WR: Wondering how important this list is perfect
<shadi> +1 to wendyreid
AC: primarily this is framing but hoping we can agree
<Detlev> +1 to Wendy!
AC: WHERE to deal with issues is an important question
… in W3 we intend to have other docs, e.g., policy guidance, WCAG-EM-style document for testing, etc.
<GreggVan> +1 to alastair. since it is unlikely that will agree on everything (wouildnt that be nice) -- the meetings main output must be that we find where we do agree and document it.
AC: in the subgroup we had lawyers come in and they liked the idea of having additional guidance
GP: Think that there is often a misconception. What is missing is documentation about processes, trainings, whatever. With assertions we were trying to address this.
… right now WCAG is a snapshot of the current state
… agree that policy docs should help
<wendyreid> +1 to giacomo's point
JA: in ADA Title II rulemaking in first round. it was a challenge since there was no way of scoring. We should provide a framework that would help regulators choose.
… there is a risk that what we create won't be adopted unless we provide a way to facilitate adoption
GV: Wherever we say policy we should say 'policy and enforcement'
<Zakim> GreggVan, you wanted to say not just policy - but policy and enforcement
<nat_tarnoff> +1 to GreggVan
WR: Thinking about GP's point. May be helpful to understand who our primary audience is
… We have more than one audience, and we should understand what our priorities are
… how do we address the secondary audience needs? Perhaps it shouldn't be in the same document.
AC: We have looked at that. This is part of why we are considering a separate policy document
<ShawnT> What about thinking about putting this into play Accessibility Roles and Responsibilities Mapping (ARRM): https://
AC: Think that we may be able to agree on high-level solutions.
… proposals have included
… 1. prioritizing paths/tasks
… 2. automated testing
… 3. usability testing
… 4. authoring tool methods
… 5. some sort of progression score
… undoubtedly more possibilities
<nat_tarnoff> +1
AWK: Seem like a good thing. Good to get input from broader group
Wilco: Please explain the auth tool one?
AC: One idea would be that if you are making a claim about an area that used generated or 3rd party content that you provide a good tool for users to add the alt
GV: These are high-level topics not solutions
… dealing with only part of a site is always easier and I wouldn't agree that is a solution.
<kirkwood> +1 to Gregg
<nat_tarnoff> Not mention because it is a law, doesn't make it right.
<kevin> +1 to Gregg
GV: not sure about the automated testing one
AC: Automated testing can give you wider coverage and more easily identify issues
WR: wanted to call out that for the auth tools thing, people should know that there is an effort to reboot the authoring tools work
<GreggVan> great to hear.
WR: hopefully updated guidelines very soon
<hdv> +1 Wendy
CH: different concern with first bullet. Paths and tasks are hard to define.
… does this mean prioritizing paths and tasks for more testing or to be different in some way? but not at the expense of other content?
AC: yes
<Zakim> alastairc, you wanted to comment on 1st item
GP: About first bullet - everyone working on accessibility knows that people use this approach when getting started
<kevin> +1 to Giacomo
GP: concern that companies will use this as an excuse since they are "still working" on the main paths.
<Charles> an example challenge of defining a path/task: ecommerce checkout. guest versus account paths still are not static. each has dozens of sidequests.
<kevin> AWK: Debate now or later on survey?
AC: More discussion on these in survey
KW: thinking about class of product as a conformance category.
… SVG has 4 classes for conformance - generator, viewer, etc.
… share GV's concern about making a broad claim based on a small sample
GV: Can we swap "strategies" and "solutions"?
… classes are a sorting mechanism and we would need to make sure that we are comprehensive
<Zakim> GreggVan, you wanted to ask "can we change "solutions" to "strategies"
SK: two items
… 1. key challenges. User generated content is important due to the dynamic nature of products and new content is impossible to test
<mgifford2> Back to an earlier point. I do have a concern about WCAG-EM vs the American style ACR approach (VPAT, GPAT, and most recently OpenACR). I do feel like we are confusing them too often on both sides of the Atlantic. WCAG-EM is important for pushing organizations to comply with their websites or web services. American ACRs are focused on procurement and are critical for pushing industry incentives. It is really a matter of where pressure is
<mgifford2> being applied. I do think that having standardized measurement of deployed sites is great (WCAG-EM), but we also need to be able to push those vendors which are building tools like Content Management Systems. OpenACR is built to try to build on WCAG-EM, but be focused on procurement.
SK: 2. related to WR's point about ATAG. Regulators are typically unaware of other resources, like having the auth tool methods in the same document for this reason
CH: thinking about KW's point - by prioritizing paths we are hinting at having a "conformance claim scope"
… for automated testing there isn't another bullet about whether we are using sampling and how
GV: First two bullets - they seem dangerous for conformance
<Jon_Avila> Perhaps we could take out the "rather than whole product".
GV: add "for order of intervention" to bullet
AC: Part of the point is that these are strategies that people do use today.
AC: Survey
conformance Survey
Question 1 Scene Setting
https://
AC: do we want WCAG 3 to be able to replace WCAG 2.x as the standard of choice for regulators?
Lisa: Confused about survey. Did it change?
AC: Subgroup reviewed an earlier version, perhaps that was it?
<julierawe> (IRC booted me so am rejoining)
GV: Since people have difference preferences, there will be compromised
<Heather> wendyreid, I can scribe after you.
AC: a few people are unsure what the compromises will be
… comments on harmonization
… kevin points out that we are not regulators but should consider their needs
GV: we need to keep the focus on access by users with disabilities to all content
<Jon_Avila> My comment was just that we needed to create something that had some framework beyond 100% conformance - I think the ideas in the survey in other questions show good ideas to support that.
Lisa: Reminding you to refresh results, if you're encouraging people to update as we go
alastairc: I need to read things first, but will do
… move on to question 2
Conformance idea: Define "essential paths" and allow for site conformance for some paths
<nat_tarnoff> I looked to the other monitro, thanks though!
alastairc: This was drawn from one of the proposals, sort of a concept of an essential path
… taking in a definition of essential, and page titled criterion, talking about "purpose" in WCAG2
… the idea is you have more important paths, specifically for a website not user
… having three levels, level 1 is scoping down to essential paths, meeting provisions, content in all paths meet core requirements and essential paths meet supplemental provisions
… we had quite a few responses, two choice
… in conformance or associated documentation
… in the answers I was able to review ahead of time
… the main points I caught were [reading Detlev's point]
… how we do reliably define a path
… that seems like it should be objective that differnt people would apply in the same way to different products
<Charles> +1 to reliable definition of path
alastairc: different products using paths in different ways
… should be a compliance thing rather than a conformance thing
- Additional documentation will be difficult to develop & track.
- Useful for “partial conformance”.
- WCAG-EM would need updating.
- Need to define essential paths (probably different term).
- Every path on a page is there for a reason, if you remove it, the purpose of the page changes.
- Prioritization is understandable, but there should still be a clear expectation that the rest of the site will eventually be reviewed.
- Anything which allows for a conformance claim to be made against a whole based on a sample is problematic.
<Jon_Avila> Some regulations have specific pages that need to be covered - I wonder if we should consider essential pages?
GreggVan: Any time we say that conformance to part means conformance to whole, there's no logic to that
<kevin> +1 to Gregg
GreggVan: if one wants to say we want paths to be part of scoping, that is a different question
… it
… it's not to do with conformance, it's part of the claim
… that's scoping, not conformance
… a path through something is not the whole
GreggVan: It doesn't have any place in saying something conforms
alastairc: Going to see if I can play this back, if we're talking about conformance, specifically the term conformance, saying the whole of a thing conforms when a subset is including, it doesn't conform
<kirkwood> +1 to Gregg
GreggVan: Paths is a piece
… there are many kinds of a thing
… what does critical mean for example
AWK: I agree completely with part of what Gregg is saying, we absolutely want to make sure that the goal of someone who wants to meet WCAG3 is access
… but I like paths because we've done the approach where you have to meet everything and we've seen its not successful or better for end users
Two topics: 1) Having a sub-set of a thing to claim conformance, 2) Defining particular paths.
AWK: I'd like to see something like paths, paths in combination to assertions and recognizes their responsibilities
… providing something like an SLA for problems that people encounter
… people do this today
… they identify key paths and pages, and being clear they know there are gaps and commit to fixing those within a clear timeline
… acknowledging their service offerings
… most site owners can't or won't assess every nook and cranny of the entire site
<wendyreid> +1 AWK
AWK: can we do this in a way that ensures full access
<kirkwood> the reality is the “you can’t be 100% conforment” is true however in practice often essential paths and pages are more vigilantly texted for accessibility. We are the ruler not the measurer? no?
<Makoto_U> +1 to AWK
giacomo-petri: I mentioned previously I agree with the representative sample, I also acknowledge that for some products it is easy to determine the essential paths
… such as an ecommerce website, but other websites can be harder to determine
<Rachael> +1 to difficulty of determining essential
<Adam> +1 to giacomo-petri
giacomo-petri: It is due to the nature of the service
<kevin> +1 to exploring ideas about how to encourage improvements, -1 to this being part of conformance
<kevin> +1 to challenge of 'essential'
<Charles> i disagree that ecommerce example of path is easy to define. there are too many variables.
giacomo-petri: the ones most accessed by users, but this could be subjective, different audits may post different results
alastairc: One is being able to determine an approach that is a subset of the whole and can be used to determine conformance, and how would you do it
stevekerr: You mentioned the two topics, this is about the subset representing the whole
… being able to call it conformance as a whole or a part, I raised my hand in response to Gregg, having full conformance visibility is important
<Charles> +1 to stevekerr. this is partial conformance.
stevekerr: what about if you have 5 parts of a product, 4/5 are conformant, but now you have some transparency
… what parts of the products address user needs
<Zakim> Rachael, you wanted to ask how we would define critical paths
stevekerr: this product does not meet full conformance, but with this topic and others, is it not beneficial to be able to claim conformance to parts that make up the whole
Rachael: I've always really liked the idea of essential paths, the challenge is coming up with an adequate definition
… I like the idea of being able to scope to these
… we're focused on these areas and this is where we claim conformance
… my concern really is using part to declare whole
<Charles> +1 to Rachael path=scope not whole
<Adam> +1 to Rachael
<nat_tarnoff> Agree they are needed for scoping, but I agree with Rachael
ack Lisa
<laura> +1 to Rachael
<giacomo-petri> +1 to Rachael
Lisa: Having read the different proposals, paths are only used to define scope
I'm interested in the history also, don't recall it
Lisa: important features, can you send the email, can you label the email
… provide different statements for different features or parts
… in reality, that is already what everyone does
… are we allowing people to make accessibility statement for a scope
… the reason I like it is it allows people, including regulators, to do more on the really important stuff
… for the whole site, we have gold, automated testing on some provisions, on some other paths we have automated, user, professional testing, it sounds like a lot but its limited scope
… the power of the paths, it allows people to do more on the most important things
… do we need to define essential paths, we have some things like anything to do with health and wellbeing, rights, even legisalators clarify what they consider an essential path
Wilco: I find this an interesting idea, it also feels clunky to me, essential paths/pages, ultimately the goal is the whole thing is accessible
… where I find WCAG falls short is its a fixed set of guidelines without much flexibility, what I think we should consider
… what is the role of quality management systems, Accessibility for all, ISO 9001, EN 17161 Design for all, it isn't just about indiividual things, but processes that validate and ensure quality over time
… what's essential can change, can be defined by organizations as they grow and evolve
… avoid a snapshot style evaluation
… there's an intersection here, there's a lot to learn from security and quality
<Wilco> EN 17161 Design for all,
<AWK> +1 process+assertions+paths would be great
<Lisa> +1 to wilco
wendyreid: I agree with Wilco
… this is a good idea
… but I also think it would be too narrow for us to focus only on paths
… I think it’s possible to find a definition of “paths”, “parts”
… on large websites where you have millions of pages, e-commerce, etc.
<Charles> all paths are meandering
wendyreid: there are identifiable paths
… determined by use case or business case
… but also by “how many people go through these places?”
… focus on the most visited
… another thing to consider
… the process that goes into the generation of those pages
… products that use design systems, or component libraries, for example
… where the underlying systems have been, or could be, assessed for accessibility conformance
… so one could also assert these systems have been assessed
… and have really granular accessibility conformance reporting
… “we’re aware of this area, but not this area” and “here’s our remediation process”
… a lot of people have processes and use systems like these
<Wilco> +1 Wendy
GreggVan: Examples from real cases, it was decided the path didn't include ads, the ads had discounts
… PWD didn't have access to the discounts
<nat_tarnoff> That sounds like a scoping problem.
GreggVan: related products weren't included, the other was the state of Minnesota, the only necessary pages were services for PWD
GreggVan: People make strange decisions when determining these things
<Zakim> alastairc, you wanted to comment on defining paths, and whether it compares to selecting pages. and to also mention quality standards
alastairc: Playing devils advocate, right now in a conformance claim, people pick pages, does parts make a difference, on Wilco's point, quality standards
… it's come up several times, it turns into quite a different standard
… on the security levels, you're assessing a particular area, the risk for each
<kirkwood> if the elevator is inaccessible, is the building inaccessible?
alastairc: you pick what you do in that org, how to address, the choose your own adventure standards, do bring a much higher level of cost and effort
… at the end of the day, we'd still need the provisions
… that's the most useful bit
… we'd be potentially giving up a lot
… we didn't progress much further to see what it would be like as a quality-type standard
… third point, if we're trying to introduce flexibility, it's either the coverage of the pages or conformance units, how many conformance units are you including, which provisions are you saying you conform to
… how many for each conformance unit?
… the main push back I am hearing is being able to claim conformance for a whole
… when only testing a subset of the whole
kevin: That last point, I agree, I don't care what you put in the scope, as long as the scope is clear and makes it clear not everything is passing
… sensible for us to provide guidance on it
<Charles> +1 to kevin which is why we still need a ‘partial conformance’ claim
kevin: just off the back of Wilco's stuff, is there an aspect of this where there is something else, another standard or process standard
<giacomo-petri> +1 to kevin (not in our scope)
kevin: we have WCAG, this thing meets requirements and therefore accessible, and that only applies to the thing
<Lisa> +1 to kevin
<giacomo-petri> but essential as well
kevin: a separate thing is "this is a quality assessment process"
… that then allows for trying to make WCAG do everything
<kirkwood> Kevin made a good point about process standard
Makoto_U: I wanted to say that it is difficult to define what is an essential path, there is no issue as long as it's define in the conformance requirement
… it would be false to claim the site is compliant if part is compliant, it's a subjective judgement, but clearly defining what falls into the conformance claim, it is clear to them, we need to leave it to content owners
<kirkwood> +1 to Makoto
Makoto_U: for those that don't have legal requirtements or pressure, this would be helpful too
nat_tarnoff: I'm just wondering if there's layers to this
… we know that essential paths are important to getting the users what they need to do, but there might be things in the way that are important but not essential
… what conformance means is clear on what is an essential path may be, the end user can define that, they cover their essential paths, we cover other things in WCAG-EM, conformance ascends by parts
Detlev: Just to remind us, with WCAG 2, the object of conformance is pages, if we go beyond that to paths or processes, it's no different than claiming the pages within the path today
… why WCAG1 or 2 haven't been able to do that, the effort.
… the other thing is users don't look at a11y statements, they encounter a problem, they may then find the statement, but they just face a barrier, the user experience is the corner stone
In general, even procurement people who are buying a product and should be reviewing the conformance statements don't do it.
Detlev: to bring it into view to the user, you'd have to label things, but clients don't want to do this, maybe only in cases in legacy pages or older areas of a site
Wilco: The reason I brought up quality management wasn't to suggest WCAG gets one, or W3C needs to make one, a new version is already coming,
… I brought it up because the problems we raise are better addressed there
… how do we write a conformance model that works well alongside a quality model
… how can we offload some of these problems to those standards
<Zakim> alastairc, you wanted to comment on what's needs to be solved for this idea to get somewhere
alastairc: Keep it simple, refer to the quality standard
<kevin> +1 to not reinventing the wheel
alastairc: good point from Wilco
… what would be needed for this proposal to get somewhere?
… there's two topics, definition of essential paths, and proportion of a product is conformance based on a subset
… proportion of a product, what if we're not calling it conformance
… it's not level 1 of conformance, it's partial conformance or something like that
… you're stating your scope, this scope doesn't include everything, that might solve this problem
… a later proposal has a variation of this
<hdv> +1 to call it partial conformance
alastairc: on the definition side, I don't see how we could come up with a definition that covers a range of products, they have limited paths but many pages, or more limited things like a watch app
… if we leave that aspect to the person making the claim
… it clears the way for it
<Zakim> Charles, you wanted to comment on language. change ‘essential path’ to ‘your scoped path for a partial claim’
alastairc: Two aspects, we give up on trying to define what paths, but we give an informative description, or we don't give it a name like conformance
Charles: I got on queue to say the same thing, thinking of it as inverted. Conformance is everything
… a partial conformance claim requires a scope, the author has to define
… they have to define their partial conformance claim
<Zakim> GreggVan, you wanted to say "Interesting idea. that in CLAIMS we allow people to specify the SCOPE in more than just pages. THat is a claim can be Pages, or Paths, or Function, or classes - but the word "essential" or other subjective terms cannot be used. It must be clear what the exact scope is.
GreggVan: An interesting idea we allow people to specify scope in claims, it can be pages, paths, functions, etc.
… the word essential or other subjective terms cannot be used, there's no way of testing your claim
… I don't know what is essential
… there's no subjective term in that
<Charles> +1 to GreggVan on ‘essential’.
GreggVan: it could be a claim of conformance, I would think we probably should use partial conformance
… with the following scope, they'll not understand. I like partial conformance
<kirkwood> +1 to partial conformance
GreggVan: if regulations can change the scope needed, if the regulation doesn't require the whole site, allowing the claim to be more flexible, and owners have flexibility. Even small companies can have hugely complex websites
<Detlev> isn*t "partial conformance" the scope does not meet all requirements? That is different of conforming fully, but only in certain parts
wendyreid: I also agree that we can say partial conformance for this
… there are a lot of parts, even for simple websites
… I do worry about us getting bogged down in “what ifs”
… of people interpreting this in a negative fashion
… we should threat model people trying to get around it
<shadi> +1 to wendyreid getting held back by edge cases "what-if"
<hdv> +1 to wendy, don't need to spec our way out of finding loopholes
wendyreid: if we try to “spec” our way out of someone trying to find loopholes
<kevin> +1 to Wendy's point on avoiding to try to account for every bad approach
wendyreid: if we try to define a conformance model that tries to account for every bad actor, we’ll tie ourselves in knots
… we need to make this easy for people to create an environment of conformance
… can’t focus on edge cases
<Zakim> alastairc, you wanted to comment on normalising partial claims
alastairc: Goal is conformance, we should have notches on the bar leading to conformance
… it is a subset that is defined, we don't call it full conformance, we make it clear its normal and on the pathway
hdv: I want to +! wendyreid and loopholes, as a regulator we see problems but they are very rarely about definition-based loopholes, but it's because they are on their journey towards conformance
… we don't need to work those things into the conformance models
… in response to alastairc, regarding partial conformance, one thing we see is orgs doing a lot to getting a sticker or the equivalent
… they want proof to say they did the thing
<Charles> if normalizing partial conformance, it may require normalizing partial conformance claims. very few sites make claims.
hdv: we had a logo of a little green man to put on websites, we know its nonsense, but we saw people wanted the signifier
DRAFT RESOLUTION: Adjust paths idea to: 1) Define paths as a unit of conformance, 2) Have one or more levels of partial conformance that include paths.
hdv: we can figure out wording that equivalent to partial but encouraging to people wanting to improve their work
<wendyreid> +1
<Rachael> Suggested edit to DRAFT RESOLUTION: Adjust paths idea to: 1) Define paths as a unit of conformance, 2) Have one or more levels of partial conformance that include paths. 3) Expand on paths in external documentation such as policy and quality documents
stevekerr: On Hidde's point, on the topic of encouraging people trying to full conformance status, if we move forward with paths, is it something we put in a policy doc to prioritize full conformance
… provide recommendations on full conformance, how to prioritize critical paths, paths to full conformance
… helps people try to reach their goals and transparency
Rachael: Chair hat on, make a suggested edit to include a point on paths
… started a parking lot on these topics
alastairc: Haven't decided on a quality doc so more to talk about
kevin: How is path different from process in WCAG2
alastairc: I did try to define it differently, more of a name difference that people will take to have a wider meaning
shadi: Exactly my point, thanks Kevin, the thought here, essential or not, it was building on essential in WCAG2, we have squishy terms in WCAG2 that can be testable, maybe we can build such a definition
… definition that provide that clarity, not all content is equal, even if it is partial conformance, the point is, can we separate out primary tasks or things to be done on a website
… using WCAG2 type definitions might help ehre
… primary path or selected path
… subtype of processes that accomplish certain things
… ensure different people come to the same outcome
<nat_tarnoff> Attempt: Path - a set of pages or screens needed to complete a major task. Process - a set of pages or screens not considered a major task. Example Ecom: Purchase is a path. Adding to wishlist a process.
Wilco: Sorry if this is obvious to others, what problem does this solve?
… thinking about the historic problems slide, this doesn't click with those
shadi: This is responding to a couple of things, one of the issues with WCAG2 is that its harsdly met in practice because there is always something somewhere, I think this better reflects the user experience, there is a difference if the site says the primary functions meet a higher level
… there is a different expectation than all functions meet requirements
… trying to make these notches on the ruler more correlative to the user experience
… or to user expectations
… level 1 is less accessible or expectation, not necessarily by removing requirements, but where the requirements occur
… regulators can say how far, or timelines to reach different levels
… more notches on the ruler to allow regulators to provide the rule, more focused on the experience
Rachael: Intention is to finish off the queue and then go back to the draft survey, which had a proposal for a way forward.
<Zakim> GreggVan, you wanted to say 1) there is no path in W2 -- only process. so they are not the same -- Paths does not exist - for the same reason that it should not exist in conformance here - (only in Scoping - maybe) and 2) Essential is only used in exceptions -- and is something a site/author need to be ready to defend in adjudication (court, judge, jury, regulator).
GreggVan: 1) There is no path in WCAG 2, only process. 2) Essential was raised; essential was only used in the exception, not in the provision. The problem is that if you claim the exception, you'll need to prove it in court, or defend your decision that it is an exception. Cautions the use of using exception.
Charles: Has heard the use of essential, prioritized, critical... all of these words suggest an audience for whom that is true. I don't think that is our place in providing a definition of path within this resolution, whether we call it path or process. Don't think we should qualify that path; that's up to the author making the claim. Example is
… the upsell in an e-commerce site to add what you forgot.
alastairc: One aspect of the proposal was to use the terminology of essential from WCAG 2 and page title. We could use those in a site context; it's select a subset of paths that maybe a partial conformance. It would be a way to level on the way to conformance. We are trying to define essential paths as something that would work across all the
… different types of site and product that we need WCAG to apply to. I can see paths working as a unit of conformance where the regulator or the author is selecting what goes into a claim, but it's something which would need to be objective and work across the different types of sites.
<alastairc> I think parts 2 and 3 would be new
<Detlev> 2 sounds vague
Rachael: [Proposing draft resolution]; calls out that there is a scope for declaring conformance in addition to views called task flows; not sure if I see an difference between what we're talking about here or paths in general. Needs help understanding what the difference is.
<Charles> path / flow / process can be grouped but are not same
GreggVan: We were talking about paths being in scoping conformance claims; you have it as conformance for number one, number two, you have it in conformance, number three, you seem to have thrown it out. none of your choices are to have paths to be something that is part of scoping.
Rachael: Was thinking that Unit of conformance would be conformance; clarified draft resolution language, posted above.
<Detlev> thanks!
alastairc: Clarifying that it's saying that we would try out in the conformance model including meeting certain levels of provisions - that's what number 2 means.
Rachael: To clarify for people who joined late, wants to restate the goal for this 2-day period is to have a series of strategies that we as a group think we have enough weight or approaches that have enough likelihood of success that we can all agree on.
GreggVan: Are we agreeing to three different points?
Rachael: We would be agreeing to do 3 different points that seem to have come out of the conversation.
Wilco: Is this a decision from the group to do this, or that we'll explore this as a resolution?
Rachael: Explore is used in number 2. 1 is included in the current path. 3 is expanding on the external documentation.
<Jon_Avila> Does a path include the whole page they go through?
Rachael: agreed to say 'explore' instead of 'expand'
Wilco: Better, thanks,
GreggVan: We have it as a way to conform, or as a way for reporting a partial conformance. Maybe explore, it should be as part of claims.
Rachael: Reposted revision of Draft Resolution above.
<GreggVan> +1
<Lisa> +1
<hdv> +1
Rachael: Is everyone happy with that before we vote?
<JeroenH> +1
<wendyreid> +1
<nat_tarnoff> +1
<Makoto_U> +1
<Rachael> Proposed resolution: Adjust paths/task flows idea to: 1) Explore paths/task flows as a unit of conformance in scope, 2) Explore one or more levels of partial conformance that include paths/ task flows. 3) Explore paths / task flows in external documentation such as policy, wcag-em, etc.
<SydneyColeman> +1
Rachael: no other concerns heard.
<Adam> +1
<ShawnT> +1
<Jon_Avila> +1
<wendyreid> +1
<Heather> +1
<CClaire> +1
<BrianE> +1
<GreggVan> +1
<alastairc> +1
<Charles> +1
<joryc> +1
<Rachael> +1
<Wilco> 0 Don't mind this but don't know the value of this
<Detlev> +1 ( but we have been exploring for many years...)
<Monica> +1
<Gez> +1
<AWK> +1
<filippo-zorzi> +1
<jtoles> +1
<Jennie_Delisi> +1
<kirkwood> +1
Rachael: Acknowledging Detlev, that we have explored this previously.
<LenB> +1
Kevin: Not averse to this; thinks its fine. How long do we explore this for? What's the end point? How do we determine what is and is not valuable?
Rachael: When we talk about 'explore' here, that we are saying that we are going to build this into the proposal and present it to this group, and assess if it's better or not.
alastairc: Agree to Rachael's comment. It's not the proposal that is in the survey, it's an adjustment of that. The next step is to put something in the draft.
Rachael: No -1 in the poll; moving forward.
Rachael: RESOLUTION Adjust paths/task flows idea to: 1) Explore paths/task flows as a unit of conformance in scope, 2) Explore one or more levels of partial conformance that include paths/ task flows. 3) Explore paths / task flows in external documentation such as policy, wcag-em, etc.
RESOLUTION: Adjust paths/task flows idea to: 1) Explore paths/task flows as a unit of conformance in scope, 2) Explore one or more levels of partial conformance that include paths/ task flows. 3) Explore paths / task flows in external documentation such as policy, wcag-em, etc.
Rachael: draft straw poll. We were talking about this concept of essential tasks, paths or essential task flows, or whatever we're going to call that concept - do we ant to continue to explore. The concept of 'essential path'?
alastairc: If you've got a definition of essential path that was pulling through - is it essential to the website, is it part of the website. Does it seem to be feasible to have that kind of definition that is going to work across a variety of definition that is going to work across the interfaces that WCAG shoulda apply to.
shadi: Suggests that get started with the resolution and see how that definition of task flow evolves and gets used. Have a second definition of critical task flow, or similar. Pragmatically, let's start with that.
Rachael: Hears suggestion; wants to capture thoughts
<alastairc> In the current adjusted form, path becomes a conformance unit you can use (but claimants choice as to which are included), with advice, there is no sub-set going on.
<kirkwood> +1 to Gregg
GreggVan: Thinks Shadi has it right; we should not use the word essential anywhere except in exceptions. To say that critical paths are, there's no way of doing it; thinks we should just drop the terms critical and essential, except in the exceptions nd with the understanding that those are going to have to be adjudicated. Continue to explore the
idea of paths.
julierawe: Wants to double check that the resolution is to see if we are aligned to explore all three of these things?
Rachael: Yes.
<maryjom> +1 to dropping "essential" as noted in my comments on the previous question.
Charles: Agrees with Gregg to drop essential or critical from it, but that we need to define these things as group able or relatable. They're all similar, but they can be groups and have their own definitions.
Jon_Avila: 1) There are certain paths that maybe less commonly used, like resetting your password. It's hard to define a term like essential because they may fall into different categories. 2) Thinks that from WCG 2 and beyond, just because a path may be through a certain page, other parts of that page could be barriers that would prevent access to
that path on that page.
Rachael: When a path touched a page, the scope was to include the whole page - based on previous conversations.
hdv: Sounds like we might try to get a resolution to explore essential paths more, and whether the essential part is what we are going to call it. Doesn't think we need to go too much more into detail.
Rachael: Yes, that's what I'm trying to get to.
Important paths and we need to define what important means
<Rachael> Straw poll: Should we continue to explore the concept of "essential or critical paths" to enable requiring these as part of a conformance claim? +1 to support exploring essential paths, -1 to discourage exploring essential paths, 0 if in the middle
alastairc: There are concerns raised on this topic.
<Lisa> +1
<julierawe> +1
<GreggVan> -1
<shadi> +1
<Rachael> -1
<shoobe01> +1
Rachael: Puts in straw poll.
<Wilco> -1
<Charles> -1
<ShawnT> +1
<Adam> -1
<Jennie_Delisi> +1
<nat_tarnoff> +1
<hdv> +1
<wendyreid> +1
<laura> -1
<alastairc> -1, should be regulator/compliance thing.
<kirkwood> -1
<Makoto_U> -1
<AWK> +1
<SydneyColeman> +1
<kevin> -1
<Heather> -1
<stevef> 0
<maryjom> +1 ...but not using "essential" as it is currently defined
<janina> +1 because there are some paths we insist must conform
<BrianE> 0
<JeroenH> +1
<CClaire> -1
<Gez> 0
<Jon_Avila> +0
<Detlev> 0
<filippo-zorzi> 0
<LenB> -1
Rachael: Around the concept of essential. We are doing paths, we are only discussing the use of 'essential' to define a path or a subset of paths.
<giacomo-petri> 0
<shadi> +1 to janina
janina: I think what we're trying to say with essential and critical is that there are some kinds of activities that we really want to see them being fully accessible and we want them as part of any conformance claim. I think we need to turn our focus as to how we define the ones we really care about.
<maryjom> +1 to janina
<GreggVan> -1 unless there is a univeral definition of what essential or critical and everyone will know which parts are accessible if i say just "i did the essential" parts
Rachael: Thanks for the clarification; acknowledges that Shadi agrees with it. Will probably redo the straw poll
<AWK> +1 to Janina
<shadi> +1 to wendyreid
<atya> +1
<kirkwood> COGA path ?
<giacomo-petri> +1
<GreggVan> what does "types of " mean
wendyreid: Agree with you, I think based on our previous discussion, one of the things that is compelling is the idea of partial conformance and almost having a ladder (Tier 1, Tier 2,e tc). Having an iterative path, and building towards something.
<GreggVan> is a type definitive and comprehensive>
<Jon_Avila> Paths are really helpful as without them it's hard for a purchaser/user know what the conformance level actually means in a practical way to people
Rachael: Acknowledges general support to that.
alastairc: Chair hat off; there are a couple of ways this can go. If you're trying to find whatever it is (essential, critical, important) I don't think that's going to be objective enough to use in conformance. That doesn't mean we can't use the concept. It doesn't mean we can't have more levels and notches. The key for me is whether you want that
… term to be used in conformance, which may imply different people would define the same paths for the same product, and is skeptical that would be the case.
alastairc: Let's keep the building blocks simple, and allow them to be used in various ways.
<alastairc> hdv - but could we define a term (like essential) that defines which paths to include for each bar?
hdv: I think I want to make clear we're not talking about lowering the bar, maybe that's something we should say explicitly. Having essential or critical paths helps us show the path to full conformance by distinguishing levels of partial conformance is better, and and critical or essential paths or whatever we call them can help find out, I already see my regulatory colleagues do that today too, it's a pragmatic way to find out what affects most people.and of course some of this puzzle is to be solved by regulators, but our definitions for paths can help inform some of it and be useful.
specifically case.
giacomo-petri: Voted
<kirkwood> we need a deeper dive on accessible paths. (and affects on accessiliby on the cumulative mulitpage related accessibility issues)
giacomo-petri: Voted '0' because I'm in the middle. Echoing alastairc, essential is difficult as an objective term in conformance. It might be more difficult for larger companies to claim conformance, compared to small properties. Essential flows can be a way to demonstrate progress and conformance. That said, I am more in favor of still exploring the essential path and not for conformance.
<Zakim> GreggVan, you wanted to say "we should explore the idea of saying that "these paths" conform where the paths are defined by the author. esp as part of partical conformance. but we should not be using words that people would all think meant something different like "i did the *important* or *essential* or *critical* paths or even "these types of paths" since i dont know what they actually mean by those paths.
Charles: Building off alastairc point, my understanding of this straw poll of whether or not we pursue essential or critical. In part 3 of the resolution and not parts 1 and 2.
GreggVan: "we should explore the idea of saying that "these paths" conform where the paths are defined by the author. esp as part of partical conformance. but we should not be using words that people would all think meant something different like "I did the *important* or *essential* or *critical* paths or even "these types of paths" since I
don't know what they actually mean by those paths.
Rachael: Felt conversation was circling, closing the queue.
<alastairc> hard to define in a way that applies across the variety of things WCAG applies to, and would be reasonably objective.
<AWK> +1 Shadi
shadi: Hearing general agreement is that the challenge is how to define 'paths' or 'tasks' or whatever we call it. What I don't hear consensus on is what we call it at the end, or if it's part of conformance or not, or in what way. We agreed to explore that earlier; we need to double down and define that. Everything else will emerge later.
shadi: In WCAG 2, we have things like accessibility support where there's extensive guardrails and definition of how you define something and how you would probably need even more extensive definition. We need to see where it fits in and how else we can use that int he rest of the WCAG document.
<laura> +1 to wilco
Wilco: voted '-1' because it seemed like a prioritization problem. The ultimate goal is to reduce accessibility barriers as an organization. The other part of that is how impactful that barrier is. Thinks that having one part of this in the conformance model and not the other wish cue that prioritization problem in an uncomfortable way. These kinds
… of problems that I think are much better solved in a quality management system and don't fit well into WCAG.
jyasskin: 1) The goal to have website owners or regulators talk about paths probably affects the conformance model itself because the current conformance model is only on page at a time. 2) Does this group want to add more work, or just accept that regulators and website owners can talk about paths, but not actually make requirements or define what
… an essential path is?
Rachael: Goal is to both get it done and get it right, and there's a balance to that.
Rachael: Wants to understand where people are at on using the term 'partial conformance'
Partial conformance
<Jon_Avila> We have partial conformance today in 2- but partial is non-conformance.
Rachael: This is a side step, please keep points concise as possible.
Detlev: Not entirely ensure when we talk about partial conformance, such as a page that does not meet all requirements, therefore is partially conformant. Asks for clarity for clarity on the two terms.
<Rachael> "partial conformance" is when the scope does not meet all of the provisions.
<Rachael> "conformance of parts" is when the scope is smaller than the full site
Rachael: Asks that for the purposes of the conversation, that we need to come up with terminologies. Partial conformance is when the scope does not meet all the provisions; Conformance of parts is when the scope is smaller than the full site.
<Zakim> Rachael, you wanted to react to Detlev to define terms
GreggVan: If you're only claiming it for some pages, you don't have to do partial conformance, you just state those pages as being in your scope, and you say these pages conform. Within my scope, not everything conforms. Doesn't think there is a need to disclose any further than that, because when you make a statement of partial conformance, you
… would state what doesn't conform. You could say this is a statement of partial conformance which is a subset of non-conformance. It's a partial conformance either way.
<nat_tarnoff> If we only conform pages, GreggVan makes sense. If we conform for products, then Partial is non-conformant.
<kevin> +1 to 'partial' conformance to being a wide area
Jon_Avila: Partial conformance, wonders if there's a non-conformance. If there is one thing that metes one of the criteria, then you're partially conformant. I agree with Gregg that we would need some sort of way to explain what that means to make it useful. The whole point of understand nonconformance is for the user to understand or determine if
this is acceptable or not.
<Zakim> Rachael, you wanted to react to Jon_Avila to discuss scribe change
<Detlev> I'll pick up
<shadi> +1 to janina
<maryjom> +1 to janina
<wendyreid> +1 to Janina
janina: We need to think about partial conformance because that's the reality of the world. We're never going to get any site to fully conform, unless it's really simple and doesn't have a lot of pages. It's not a question of 'do we deal with it' it's 'how we deal with it.' I think we've taken that is about as far as we can go. There are places
… where we don't want partial conformance (password reset, credit card data, etc). We might come close, but may be arguably not sufficient.
<AWK> +1 to Janina
<janina> AG's FPWD of Challenges with Conformance: https://
<Zakim> alastairc, you wanted to comment on 99% of sites being partial, and we should make that normal, just need a score
wendyreid: +1 to Janina's comment. The challenge is that we're talking about it like it isn't the current reality. There's probably fewer examples of companies and organizations not taking this kind of approach than taking this kind of approach. You have to scope the work, regardless of where you're at today. The partial, path-based model, or user
… journey-based model is more much reflective of how people are actually working. It's really hard to do an entire website regardless of its size. We need to be mindful of how we explain the progression of this. The partial conformance model is the one that makes the most sense. Someone in a legal department at a company is probably grimacing at the
… idea of claiming partial conformance. How are people doing this current? How do we adopt some of those principles and define them into our conformance process?
<Charles> +1 to janina in that getting a part to conform encourages getting the next part
alastairc: Similar to Wendy, Out of the thousands of audits I've been involved in, and there was 1 for each fully and not, and all the rest were 'partial conformance' - maybe we need rebranding of that term, acknowledge that reality, and provide the differentiation for that 99% of cases.
<Wilco> +1 Giacomo
<laura> +1 Giacomo
giacomo-petri: Trying to solve something that is not in our scope for conformance. current regulations only refer to WCAG, so in our real world you’re either conformant or you’re not, and that alone almost determines compliance. The problem is that there should be additional ways to assess the accessibility of a of where we are now within a product, rather than being the sole measurement of accessibility. This is out of our scope. We should provide meaningful
… guidance for policy makers, but is not convinced that it should be part of a conformance itself, since any definition of partial compliance will still be somehow subjective.
Rachael: Curious to understand where people are on this concept of declaring smaller scopes as a stepping stone.
<shadi> +1 to hdv
<alastairc> Probably terms that don't have "conformance" in the term.
<stevekerr> +1 to hdv
hdv: Agrees with a lot of people that partial conformance is non-conformance. But while it is technically the same, the space between 0 and 1 is huge, almost all sites are partially conformant, so we need richer vocabulary to express where people are between 0 and 1We need to get closer to the reality of how organizations work today, how they improve accessibility, we need to be closer to that reality, like Wendy said. It would be better for regulators to understand how to compare different companies,
and to prioritize. It would be helpful to define the vocabulary to understand where people are.
<kirkwood> substantial, provisional, limited
<Zakim> Jennie_Delisi, you wanted to discuss branding impact of partial conformance
<Makoto_U> +1 to hdv
<kirkwood> reg terms
<maryjom> +1 to Hidde. There is a vast amount of work between non-conformance and full conformance. Plus delivery of content (whether web content or software) is so rapid today, reaching and then keeping full conformance is pretty close to impossible.
<giacomo-petri> ... product or company's processes; a WCAG audit should ideally serve as a validation of those processes or snapshot
<shadi> +1 to Jennie_Delisi
Jennie_Delisi: Supports Janina's concepts and HDV's. We saw an increase in WCAG 2.X maturation of adoption. They adopted it as this pass-fail concept. One of the branding concepts that Shadi brought up in the subgroup that I think might be helpful to consider because the word partial conformance speaks to that pas-fail. Only a single score. If we
… instead we could consider multiple rulers.
Heather: Anyone saying "fully conformance" I didn't trust as credible.
… Whether it's by task / workflow, it's a way to broaden the landscape for the assessment.
… it would help regulators make decisions about what they should be asking for.
Heather: Regulator sneed to understand what non-confornance means
… e.g. if company A says we have a process for taking in requests about WCAG issues, that's better than company B which doesn't make commitments about fixing things.
… sorry I was waiting for the eyrly scribe to scribe Heather...
thank you Detlev
<joryc> timelines.. and neither of that should be in WCAG, that should be in the regualtions.
<GreggVan> it is binary if you want to require it. Nutrition labels can be informative and vague because products are not required to meet some nutrition value. Whenever they are - then a specific number is specified for each dimension and you either pass or fail if you meet that threshold or not
BrianE: Not clear whether partial conformance means 80% 20% within a provision or between provisions
shadi: in support of Jenny, hidde and others...
… it is also the reality of users - like stars of hotels - that sets expectations, levels are communicating something - that is missing in WCAG
… maturity model / process oriented is the standard and WCAG conforms the outcome would not work without more granularity
Wilco: WCAG is useful in process-oriented environment because you can measure frequency occurence impact of issues - so you can then change - it is really a snapshot. This is what organisation need.
<Zakim> GreggVan, you wanted to say -- nutrition labels keep being raised and there are two ways to look at that. 1) there are 4 or 5 types of disability and you must conform for each or report which you do not conform for. (this might be legitimate but im not sure anyone would want to do that) or 2) there are dimensions and you pass if you do some of each but we are not saying how much of each you must do. If you do - then you are back to #1. In all cases
Wilco: to Hidde'S point: THe NL government has a label that includes other things, is there a plan, is there a follow-up etc - we need to look beyond the requirements in WCAG 3 - and that does not fit well into WCAG
GreggVan: We are mixing up requirements and informative information
… "nutrition labels" are informative, but they are not requirements, otherwise they would need to be binary thresholds
<laura> +1 to Wilco and Gregg
<Zakim> joryc, you wanted to say if we get into more granularity we would need a change to regulations to protect orgs who are more specific in their stated conformance, nobody is fully conformant and it is impossible to be so, yet anything other than full conformance yields legal risk. Also orgs will not universally be able to reveal their process nor promise resolution
GreggVan: 2nd thing We say: because no one can meet WCAG, we should change what it means - but it still informs behavior as a benchmark, enforcement is another thing. Everything under regulation either passes or fails. We should stop using informative examples for the conformance discussion
<laura> +1 to Jory
<Zakim> Rachael, you wanted to point to the fpwd
joryc: processes / roadmaps are a non-started from a legal perspective - processes are very flexible and not so useful from a conformance perspective - there should not be a stigmatization of levels or conformance - the reality is that companie don't talk publicly about meeting regulations
Rachael: three options...
<Rachael> 1) within the scope % of provisions met 2) smaller scope than whole 3) within a provision % met
Rachael: The percentage of how many items passed for a provision was considered too complex
… not acceptable - so mire movement on a smaller scope, not 3)
… thanks for the side-step.
… what to skip user testing now
… (Rachael sharing screen)
Rachael: Discussin levels of conformance like automatable / core / core plus
… reporting results from the survey question 4
<kirkwood> [
Rachael: The summary about those reluctant to have levels: limits of automation / changing or fluctuating levels
Rachael: Should automatic testing be a level?
giacomo-petri: Automation is not objective parameter - who is responsible to determine what can be automation?
<Wilco> +1 Gaicomo
giacomo-petri: it is essential for continuous monitoring, but we cannot mandate it - what platfor? Where is the validity?
<Zakim> joryc, you wanted to say automation is inadequate now, but may be as good as humans soon. Let's not make assumptions about its future capabilities
<kirkwood> do we consider ai to be automation?
joryc: Calling out that in many cases that in many cases is (will be) as good as human evaluation - we should focus on outcomes - the focus is can this method produce reliable results, not how you arrive there
<kirkwood> +1
<alastairc> +1, I hadn't even considered the cross-platform issues.
Jon_Avila: Agrees - it depends on various parameters - AI will change a lot of that. We cannot rely on it as a base though
<Rachael> Does anyone want to speak FOR using automation as a way of defining a conformance or partial conformance level?
<Heather> +1 to Jory - focus on outcomes and not process. Automation has a cost which may not be feasible for small businesses whose strategy is to hire knowledgeable developers.
GreggVan: Automation is undefined - different tools lead to different results - there is no correlation between what might be tested automatically and the iportance of issues
<laura> +1 to Gregg
Rachael: plus one if you agree
wendyreid: questino of when you do the automation - can happen during development, or on the productino website through crawlers, or explict scan - we need a definition of what places we mean when we talk automation
… for orgs building towards a11y automation is a tool in the toolchest, nit the whole story, but helpful especially for big companies - there you need it or it wont get done
<giacomo-petri> +1 to wendyreid
<LenB> +1 to wendyreid
Lisa: we have seen event-based automated testing - need to define relevance to user issues - its nit the kind or regex pattenr matching - future wull sho interestign tools
AWK: the goal was not to emphasize automated for its own sake, but make it part if the process - it is the lowest-hanging fruit - there's merit of it being a piece of it- the baseline should not be just automated. But we shouldn't think it cannto be trusted - manual testers also have issues
… we need to define what the rules are, ACT is a good starting point
<Charles> automation seems policy. not conformance. your org may decide to use automation within their strategy and policy.
Rachael: any comments to draft resolution?
<joryc> can you define "onramp within external documentation"?
<hdv> was just going to queue with same question as joryc, a reword would be helpful!
LenB: Expanding on Wendy's comments - turn everything into clear language - need to include clear language provided by automation tools
… it is not just checking code. Appreciate conversation
jyasskin
jyasskin: you don't want to constrain the requirement to what can be automatically checked
<Zakim> alastairc, you wanted to comment on what can we take from the idea/results
jyasskin: user-generated contend may be checked with automated ruls, the rest stricter
alastairc: Not much sympathy for automated as its own level - but particular requirements could be labelled as "should be automatable", regardless of tools used, like detect alt text in pages (nit whether it is correct)
Rachael: (reading revised draft of resolution)>
<GreggVan> +1
<shadi> https://
shadi: Resonding to Alastair: ACT rules implementation where vendors indicate which rules are implemented show the incentive to provide such information
@rachael That doesn’t speak to crafting provisions to be automatically testable to the extent possible
<alastairc> Yes - but ACT doesn't cover everything yet, we can probably provide a little preview of the theoretical automatability...
shadi: that covers some of that idea of Alastair's
<Heather> Propose: draft RESOLUTION: Do not use automation [testing] as a conformance level... (assuming we are talking about automation testing only)
Lisa: have to be careful not downplaying potential of tools in the future that may replace expert testing form today.
<joryc> +1 Lisa. I wouldn't bet against emergent tools not being to detect any kind of issue in the next 3 years
Lisa: if you say "you can't rely on tools" people will de disincentivised from developing things
<Rachael> proposed RESOLUTION: Do not use automation as a conformance level but explore it as an onramp within external or informative documentation such as labeling requirements as potential for automation
<alastairc> +1 on being agnostic to the tools
<julierawe> +1 to Lisa comment
<GreggVan> +1 to lisa comment
<Adam> +1
<GreggVan> +1 to resoultionh
<Jon_Avila> +1
0
<Charles> +1
<Heather> -1 is this only automation testing? if so include it in the definition
<shoobe01> +1
<LenB> +1
<alastairc> +1
<BrianE> +1
<julierawe> +1 to giacomo-petri comment about needing to constantly update what is automated
giacomo-petri: Like the idea of having a banchmark for tool vendors and labelling automatic checks - but can be change that over the future, since things will change?
<alastairc> giacomo-petri - is that a conformance thing? Sounds like other documentation...
joryc: Who is doing the labelling wehat is automatable?
Rachael: We cwould do it, but not in normative cintent
<alastairc> The idea was to say how automatable it is, not to call out whether particular tools are doing it.
joryc: There may be things that do better testing already right now - understanding what id possible needs to be stored in people's knowledge, it changes quickly
shadi: instead of us labelling what is automatable, we require vendors to show what they can automate
… adresses Lisa's concern, we would not pre-emt that choice, if someone can dempnstrate it cna be done
<julierawe> +1 to shadi comment about letting marketers say what they have automated testing for—doesn't seem like WCAG 3 should be getting into what is and isn't automatable
<GreggVan> "Do not use automatic-testable as a conformance level but explore it as an onramp within external or informative documentation with inviation to test-tool makers to report what they can test automatically (which will change continually) But do not say anything negative against using automatic testing since it will play an increasing role in accessibility.
<julierawe> +1 to wendyreid
<jyasskin> +1 wendyreid
<kevin> +1 to Wendy
<CClaire> +1 to wendyreid
wendyreid: The best we can do as spec creators is make clear what is a successful result for each criterion - because we don't know what tools there will be - its different on an org's level where automation can be built into the production process
<Rachael> draft RESOLUTION: Do not use automatic-testable as a conformance level but explore it as an onramp within external documentation including an invitation to test-tool makers to report what they can test automatically (which will change continually). Explore defining success and do not say anything negative against using automatic testing since it will play an increasing
<hdv> +1 wendyreid
<GreggVan> +1 to WendyReid
<Makoto_U> +1 to wendyreid
wendyreid: the thing we are looking for is user outcomes, how you get there in terms of testing methodology does not matter - but we should talk about waht is possible
<Zakim> GreggVan, you wanted to suggest "Do not use automatic-testable as a conformance level but explore it as an onramp within external or informative documentation with inviation to test-tool makers to report what they can test automatically (which will change continually) But do not say anything negative against using automatic testing since it will play an increasing
Heather: Include the word "testable" in resolution
GreggVan: one part is missing
Rachael: changed
<Rachael> proposed RESOLUTION: Do not use automatic-testable as a conformance level but explore it as an onramp within external documentation including an invitation to test-tool makers to report what they can test automatically (which will change continually). Explore defining success and do not say anything negative against using automatic testing since it will play an increasing role in accessbility
<Heather> +1
<giacomo-petri> +1
<julierawe> +1
<Adam> +1
<wendyreid> +1
<GreggVan> +1
<LenB> +1
<shoobe01> +
<Makoto_U> +1
<BrianE> +1
<Lisa> 0
<Francis_Storr> +1
<shoobe01> +1
<CClaire> +1
<Rachael> +1
<Charles> +1
0 can't think, scribijng...
<alastairc> +1
<kevin> +1
<kirkwood> +1
+.9
<Jon_Avila> +1
<janina> +1
<JeroenH> +1
<maryjom> +1
<stevekerr> +1
<jtoles> 0
<hdv> +1
<joryc> +1 with the notice that most testing tools will not be built by "Test tool makers" as we currently think of that category
<filippo-zorzi> +1
RESOLUTION: Do not use automatic-testable as a conformance level but explore it as an onramp within external documentation including an invitation to test-tool makers to report what they can test automatically (which will change continually). Explore defining success and do not say anything negative against using automatic testing since it will
play an increasing role in accessbility
<GreggVan> +1 ro Jory comment,
Rachael: Now question 3...
Usablility testing Survey #3
Rachael: Role of usability testing
Rachael: (reading the questions)
… more no's than yesses
… Slight preference for assertions
Rachael: No side said it is not as reliable for covering the whole scope; that the results of other checks and usability testing are different; should not be required
… high value in usability testing but cannot replace formal checks
… good to discover problems
<alastairc> The difference between "user testing" and "usability testing" is just because the proposal included "user" and I default to "usability".
Lisa: Objected to the question, my proposal had a component of user testing together with another form of testing, so a statement would include both, e.g. with automated ans user testing, or with expert testing. So thid provides guard rails - was involve in project with Shadi to develop methodology to make it reliable.
… if we would include it, we would increase usefulness / coverage for the end user
… there is a decoupling between conformance testign and user testing right now, currently. Our proposal would bridge that gap
… what would be a user testing result to knoe that you ment the outcome?
Lisa: If you did expert and user testing on critical paths would really help - policy makers may say health care also needs user testing
… would put it in a framework for policy makers to demand it in some circumstances, e.g. context of high importance for PwD
<julierawe> +1 to Lisa comment about importance of user testing to improve accessibility gaps
Lisa: so many users are being left out - so we should make inroads to closing this gap
Rachael: Pleas clarify ho thi is different from assertions (which cover user testing)?
Lisa: The assertions look lie 2nd class citizens - this would enable people to say this is a core thing to bring it in full front of a policy. No it looks a bit like triple A where it is likely to be overlooked. If it is part of a testing suite, it is more visible and can be made a requirement in certain situations
… we can build the framework, policy makers can the ndecvide where to require it
Rachael: So the question is, should user testing be made more prominent in our draft?
come back in an hour...
<Rachael> come back in 15 minutes.
We come back in 15 minutes!
Adam: Last context was "should user testing be made more prominent?" Picking up with that.
Charles: There's an inconsistent use of language. "Usability testing" vs "user testing". "User" implies that we're testing a user. Should be "usability testing" if we're talking about testing a product.
Charles: 2 main concerns. "or": Idea in survey question is that the scope meets requirements with expert testing _or_ usability testing. But usability testing can't be an "or" because the testing itself requires expert assessment in creating or evaluating the test.
Charles: Usability testing is even more subjective than the automation discussion we just had. Tons of methodologies. Scripts, paths, things person conducting test must know. All requires expertise.
Charles: When we consider usability testing, must have diverse users, and I agree. But people conducting test must also include people with disabilities.
<Zakim> alastairc, you wanted to react to Charles
alastairc: Terminology: I agree. I use "usability testing", but some bits of the proposal were copy/pasted and used "user testing". We're talking about "usability testing".
<Charles> user research and usability testing. not user testing.
<Makoto_U> +1 to Charles
giacomo-petri: Re automation: Usability testing is fundamental but should not be used as a metric for conformance. Woudl need to define what makes it repeatable and reliable enough to determine conformance.
giacomo-petri: Proposal defines a number of users. Feedback of a single user could significantly affect conformance even if it's individual. Structured assessment is more reliable.
giacomo-petri: Usability testing is essential to evaluate real-world cases. It's just not a way to meet conformance because of variability and context dependence. Works best as a parallel system.
<Makoto_U> +1 to giacomo-petri
<wendyreid> +1 giacomo!
<Adam> +1 giacomo-petri
GreggVan: First, we don't require any testing at all. We require that you conform. You can make a claim, an assertion, that you conform, but nowhere do we require that you test your site, or how you test your site or what method. Adding usability as something you "must" do to conform, would say you must test, and we'd have to say exactly how you must test your site.
GreggVan: No basis in any law or regulation for testing. We're outcome based, and testing doesn't guarantee an outcome.
GreggVan: Second, user testing itself doesn't require any change to the site. Lots of people do usability testing and then make no changes. Testing itself has no impact. It's good for making people aware of things.
GreggVan: Third, if you require testing of any kind, have to define exactly how it's done in order to count. E.g. testing with 1 user with 1 disability. Have to specify how many of which types of disability. Degree. Combinations. Blows up very quickly.
GreggVan: Finally, comment that assertions make it a second-rate requirement. Say we had a requirement to do usability testing. How do you conform to that? You make an assertion that you did usability testing, and the assertion is the claim. If you aren't required to make a claim of conformance, no way to check unless they said they did.
<Makoto_U> +1 to outcome-based
wendyreid: 2 things are true: agree with a lot of Lisa and a lot of giacomo. None of us would deny the value of usability testing for the quality of our products. Similar to automation discussion, it's not helpful for us to say "do X type of testing", or being overly prescriptive in how you measure achievement.
<Charles> usability testing provides results. those results provide insights. but insights can be cherry-picked by bias.
<Adam> +1 to wendyreid
<Rachael> +1 to not everyone can do user testing
wendyreid: If we say "you must do X testing", makes it really challenging to let different organizations achieve the criteria. Small nonprofit might not have the resources to do expert or usability testing.
<Charles> +1 to GreggVan having insights does not = taking action
wendyreid: Agree that it's valuable to do this kind of testing. In certain contexts it might be necessary. Leave that to the regulatory side. Could totally see a regulation like EAA say "here's how you meet generally, but health care industry must to user testing for appointment-making flows." But we shouldn't say that because WCAG should be as achievable as possible. Single developer to multinational.
<Zakim> Lisa, you wanted to say I think it should be part of conformance
<Makoto_U> +1 to wendyreid
Lisa: Disagree on one point. Want to be as achievable as possible, but not at the price of user groups being included in accessible outcomes. Clarifications:
Lisa: Have to have methodology. Have to say what's included. Don't emphasize disability but functional impairments. Contracts since people can have more than 1.
Lisa: Talking about this being a process that confirms certain outcomes, where outcomes are an aim of WCAG. If someone who's a screenreader user can submit their information using a form. Would be a way to confirm conformance of the form. Think of paths -- have to do a tax form -- talk about functional impairments: can you confirm that this form conforms to WCAG: accessible to people with these functional impairments.
Lisa: Alternative mechanism of confirming your path conforms. It's the best way for a lot of things. For COGA. In subgroup we talked about not all violations are equal. Not everyone agrees with this. E.g. alt tags on submit button are more important than in a footer.
<Zakim> Rachael, you wanted to say I'd like to see usability testing be prominent but not required
Lisa: Since footer isn't the primary task of the page. Helps close the gap. A way of conforming that doesn't require you to buy tools or a subscription. Allows you to focus on the disabled user with functional impairments on success of completing task.
<Charles> a usability participant can complete a task, but that cannot confirm anything outside of their instruction. nor can it confirm any particular method was used.
<Lisa> that is why it should be part of conformence, not just prominent in the document
Rachael: <chair hat>We'd talked as a group about exploring usability testing, even putting out a note. That history is there; ongoing conversation.</chair hat> Usability testing is incredibly valuable. Could be more prominent. Note. Highlighting in some form. Don't see how we require it. Gets really interesting in legislation and compliance. Have functional performance criteria. Dont' think those are sufficient as written in most legislation.
Rachael: Value in exploring how we encourage usability testing in various ways. Support them as alternatives in compliance.
Adam: Can we begin drafting a resolution?
Rachael: Sounds like we have some strong support for having it in; more strong support for having it not be a required level of conformance.
jyasskin: wanted to suggest something perpendicular to this
… it was suggested that we make this more prominent even thought it’s kind of an assertion
… that worries me as a third new idea
… and think this would be better handled in the policy document
<giacomo-petri> +1
<GreggVan> +1
<kirkwood> +1
Lisa: At no point was I suggesting that we require it at a base level of conformance. Would be an alternative way of confirming. Might be something like, Resolution is that we're not going to require it, but proposal was to make a way of conforming through user testing. With possibly other testing.
Lisa: Question isn't relevant to proposal.
Lisa: Maybe amend to "allow usability testing to be part of one method of confirming conformance." It's already allowed in some cases, but not others. E.g. can you use a form.
GreggVan: Nothing I'm saying has anything to say about the enormous value of usability testing. It's enormously valuable. Existing usability testing rarely includes disabled people, and it's often just one disability.
GreggVan: Lisa said it should be an alternate way of proving conformance. We don't currently have any way of proving conformance. Usability testing can't be an alternative to something we don't have.
GreggVan: Don't have any requirement that people prove it. Only in court.
<Lisa> +1 to wendy
wendyreid: Maybe a through path. Thinking when Lisa was speaking that while putting it into the conformance requirements raises challenges. We could still put it in the spec itself as a way to pass tests, as a way to achieve different outcomes. Option for us: when you have a specific requirement, provide more explanatory language around ... we have tests for each, but more information about achieving the tests.
<Charles> if a usability test confirms an outcome for a task it still may not confirm an outcome of a single provision let alone any other provision.
wendyreid: e.g. for "abbreviations explained", test is that explanation is available. Maybe an additional test that a user is able to find or understand the explanation. Could be ways to introduce that as a method. Sometimes usability testing might be more feasible than having an expert or automation. Showing that that holds equal standing to other testing methods.
<kevin> +1 to Wendy's idea to have usability testing as a viable test for a requirement
alastairc: Trying to figure out the potential proposal. Reading Lisa's proposal was "could do expert testing or usability testing", and treating those as equivalent or alternatives. Acknowledging GreggVan's point that we don't specify how people test, If we include that in informative documentation about how people test things, problematic. There's a level of requirements it's being put equivalent to. Testing for conformance and usability
testing are inherently different. Variability of results.
<Charles> usability testing is also usually conducted against a prototype versus a final production instance.
alastairc: Coverage is a problem. Scale of organizations. Either has to be really focused: e.g. on a particular requirement. Do it as methods? Not an efficient way of testing individual requirements. Not the best use of usability testing, which is best for gathering insights and improving things overall. Very focused or optional. Cost difference between manual and usability testing, would be rarely used.
Rachael: I find the idea of ... it's not a method, but a procedure ... having usability testing as part of the procedure for testing is interesting. Want WCAG3 to be as future proof as possible. Kiosks often don't have a screen reader. Not testing availaibility but alternatives. You can test that through user testing.
Rachael: Best practices could be under a requirement, and tie usability testing to best practices. Potential but more work. Chair hat off, I'm excited about exploring that direction.
giacomo-petri: Agree with Rachael and alastairc about exploring user testing in procedures. Should be very focused. Say you have a contrast ratio that's barely below the threshold. From a user testing point of view, might be ok, but technically doesn't meet the standard. User testing isn't objective enough in those scenarios. Makes it impossible to make a conformance claim based on that kind of user testing.
<Heather> +1 to Giacomo's comment
<alastairc> That is a problem, where we have to draw a line (e.g. contrast), usability testing will give results above and below that line.
<Makoto_U> +1 to giacomo-petri
GreggVan: Also think we should note ... if it's so important, why can't we get it in there. Best practice idea is good, but those are often tied to requirements. Tie this one to a guideline rather than a requirement. In COGA doc, really good things people should look for or do, that you can't tie to a requirement. But can tie them to guidelines.
GreggVan: Name the best groups to use in user testing. Types and degrees and combinations: 150-175. If the provision and best practice is to make it accessibile to people with cognitive disabilities, should not test with blind people. For each provision, give advice to test with people having these disabilities and look for these things.
<Rachael> +1 to guidance around usability testing on who to test with
GreggVan: That's where you get great advice about things to look for. Test with the groups that it's specifically targeted for. Gets over the requirement that usability testing represent the entire population. but each provision doesn't represent the needs of the whole world. Something here that helps. Not in requirements, but for thigns we can't figure out how to put into requirements.
Charles: Rachael used the word "procedure", which made me think of something included in GreggVan's comment. If we decide to provide informative guidance on the value of usability testing, and an approach to usability testing, and how to use it on specific provisions. We should have an actual test procedure to go along with that. Otherwise variability and nuance still applies. No way to consistently get measurable results from usability
testing.
Charles: Constrained type of test for a specific purpose.
Rachael: If we had those procedures, we'd want a note that was separate on usability testing that we could reference. With detailed information, so our normative document isn't too repetitive.
<Charles> a test procedure would be both for the administrator and the users
<Lisa> +1 to wilko
Wilco: Heard a couple times that usability testing isn't deterministic. Push back a little: if you get enough people, often 3-5, you're able to get a solid understanding of the issues people will run into. Reliability is not that of a traditional accessibility audit, but not bad either.
Wilco: Inter-rater reliability of traditional audits is also only about 80%
<Wilco> no i'm not
<Charles> +1 to Wilco‘s point that expert assessment also varies
GreggVan: Wilco, you're basing your nubmers on studies of homogeneous users. If we test 3 people, blind, deaf, cerebral palsy, not enough. Even within blind, is it low vision, very-low vision. Variance is directly dependent on variance of the population. Have to have a large number to cover variations. Can only have 2-3 of each type, but easily 100 combinations.
<Wilco> +1 Rachael, these are well established practices
Rachael: Chair hat off: each group needs representation of 3-5, and each category needs that. We need a document with methodology. Standard to divvy based on user groups. Original point: each provision doens't need to be test with every disability. Also wouldn't be the only way you'd test, just a way. Provides more flexibility.
<Rachael> +1 which is why we need clearer and more diverse functional needs
<Wilco> I did not say usability testing is deterministic
GreggVan: Just saying that even within the groups, there are many layers of difference. It's helpful but not deterministic. If we focus it, could be helpful with fewer people.
<alastairc> There's also still an interpretation issue, if some people get past a barrier but other's don't, how do you figure that as a result?
<Charles> it sounds like we would benefit from an ACT parallel for usability like AUT (accessibility usability testing)
Lisa: We've been part of projects making reminder programs and apps for people with dementia. We found that the number of 5 worked. Didn't lose participants because of low usability. Even with quite severe cognitive impairments. If you take something like MCI, have memory functional impairments, attention, processing speed. In those 5 people, you hit an enormous representation in the cognitive space.
Lisa: Don't think it's a huge number of people to test with. Even testing with 1 person really seemed to address 80% of the issues from the original design that wasn't made by experts.
Lisa: In a way can be easier for some designers to go on the testing, fix, test again. Instead of trying to understand a huge requirement document.
Heather: Keep going back and forth on the "sufficient representation" term. Types of disabilities, how many. Over time you'll test different people and different combinations. Thinking about getting customer feedback from people who use the product and disclose the disabilities in their support tickets. Consider that in the testing portion? In the wild, as more people use it. Good for feedback, but hard for a requirement.
<wendyreid> +1
<Lisa8> +1
GreggVan: What procedures? We have guidelines, requirements, nothing called "procedures"
Rachael: In informative docs, there's a test procedure section.
<SydneyColeman> can you repost the proposed language -- wifi kicked me out of irc
<SydneyColeman> thxxx
GreggVan: I'd include usability testing in the Understanding document as a way to understand issues and better meet the guidelines. That'll allow us to put much more in there.
Charles: Resolutions with "include", is this still exploratory? Or are we saying "include it"?
<alastairc> include where appropriate...
Rachael: We could change it to say to explore it.
Wilco: I had to read this 3 times to sort of understand it. You're saying "usability testing isn't required"?
<kirkwood> recommended?
<Wilco> Does that mean no assertion on usability testing?
GreggVan: We could add "is not required but". I think this is a great idea. Overcomes all objections about objectivity since it doesn't have to pass. We should also put as much as we can in the guidelines themselves. Allows us to catch everything we couldn't get in, and get it in front of people.
Rachael: In the Timeline
<Rachael> Resolutions to date: https://
<Lisa8> Draft RESOLUTION: Usability testing will not be required for general conformance but, for relevant requirments, we will include usability testing in procedures in the informative (understanding type) documents as a means of identifying issues and better meeting the guidelines.
Lisa: To Wilco's point, to conform to an assertion that mentions usability testing, you'd need to do usability testing. Want to add a word. ^
<Wilco> This sounds meaningless. What we're saying is usability testing isn't required except where it is.
<GreggVan> +1 for lisa add
Rachael: "Core conformance"?
Lisa: Don't want to say it's not in core conformance.
<Zakim> GreggVan, you wanted to react to Heather
Heather: When you do usability testing, have to consider user and assistive technology, and we're not mentioning any assistive tech in this resolution. Not sure how I feel about this, but raising it.
GreggVan: +1, but i think that'll be included in the writeup. Specific to provision: might say "usability with appropriate technology".
GreggVan: Also agree with Lisa's addition. Wouldn't be in the conformance section, but to conform to certain provisions might need to do usability testing.
I am confused by Gregg's explanation
julierawe: Phrase "for relevant requirements": maybe "provisions"?
GreggVan: Somebody said we shouldn't use provisions. But if we have recommendations and best practices, those are also provisions. Should be "relevant provisions". Might be attaching to guidelines. Shouldn't narrow down to requirements.
<alastairc> I was thinking requirements+assertions (and wouldn't be needed in assertions), but with best-practices we should say provisions.
Detlev: Don't quite understand separation of requirements from other provisions. If you want to conform, need to meet all requirements. In this phrasing, you wouldn't need usability testing to ascertain that you conform to all requirements. So usability testing would be something extra? Above the level of requirements? Bolstering requirements that would be asserted another way?
<Zakim> alastairc, you wanted to react to Detlev
alastairc: We have 3 types of things in the guidelines: requirements, best practices, and assertions. Assertions could say "we have usability tested this thing and acted on the results". But in requirements or best practices, you could establish you're meeting it by doing some usability testing. probably very specific. Not requiring anyone to do that. Just a way.
Lisa: Answering my problem: Could write that we're conforming to a guideline through user testing.
alastairc: If you make a conformance claim, you'd say "we meet this requirement", and you'd know that because you did usability testing.
janina: Nitpicky: Don't know about "general". Might be other adjectives. Don't know exactly what it means.
GreggVan: "Not included in conformance section"
draft RESOLUTION: Usability testing will not be included in conformance section but, for relevant provisions, we will include usability testing in procedures in the informative (understanding type) documents as a means of identifying issues and better meeting the guidelines.
<Rachael> draft RESOLUTION: Usability testing will not be included in the conformance section. For relevant provisions, we will include usability testing in procedures in the informative (understanding type) documents as a means of identifying issues and better meeting the guidelines.
Lisa: "General" was there because some assertions have user testing in them. It is part of conformance in some assertions.
Lisa: This might make it hard to refer to usability testing. We don't know what the conformance section will look like.
GreggVan: There's no testing in the conformance section.
Lisa: We don't know that. If we don't mention any testing, I don't have a problem with this, but if we do mention automated testing, I do have a problem excluding usability testing.
GreggVan: Is there an intention of putting testing in the conformance section?
alastairc:
alastairc: Not there now, but haven't discussed it.
GreggVan: "unless conformance section includes testing.
Lisa: Put "general" back in?
Lisa: Can look at the minutes to understand it.
<Heather> proposed to simplify this RESOLUTION: Usability testing with persons with disabilities will not be required for WCAG 3 conformance.
<Heather> WCAG 3 Provisions Understanding document may include a recommendation for topics to include in usability testing as a means of identifying issues with persons with disabilities.
what exactly is the "cnformance section"? is there one in each guideline? Or each requirement?
<alastairc> It's the section that says how you claim conformance: https://
thanks, Alastair
<Lisa8> -1 it is required in some assertions
<alastairc> assertions are not part of the conformane section
Heather: First, not requiring it. Second, For each provision, can document what to test.
<alastairc> "Each WCAG 3 Provision's Understanding document "
GreggVan: Don't think the second sentence works.
<Lisa8> -1
GreggVan: Instead of "general", could we use "overall conformance"?
<Lisa8> i can live with that
<Lisa8> Draft RESOLUTION: Usability testing will not be required for overal
Rachael: <chair hat>We have 2 things going on. Have agreement on 1, and not on the other. Suggesting we start with "draft RESOLUTION: For relevant provisions, we will include usability testing in procedures in the informative (understanding type) documents as a means of identifying issues and better meeting the guidelines."
Rachael: For the second, we need to talk a bit. Some of us don't want usability testing at the required level. Lisa wants to include usability in some assertions and require those assertions.
julierawe: We're having a lot of discussion about what can and can't be done with usability testing. Compare to discussion about automated testing. We don't know yet what form some of this will take. People are aiming for such specificity with usability testing, and it's so different from automated testing.
<alastairc> Well, usability testing and automated testing are quite different...
RESOLUTION: For relevant provisions, we will include usability testing in procedures in the informative (understanding type) documents as a means of identifying issues and better meeting the guidelines.
GreggVan: Reason is that automated testing was talking about levels. With this one, we're trying to keep as much as possible on the table. The other one was just off the table.
<Lisa8> +1
<GreggVan> +1
<janina> +1
<Makoto_U> +1
<ShawnT> +1
<julierawe> +1
<filippo-zorzi> +1
<GreggVan> +1
<Monica> +1
<Rachael> +1
draft
<Lisa8> +1
<janina> +1
<BrianE> +1
<ShawnT> +1
<alastairc> +1
<Detlev> +1
<julierawe> +1
<GreggVan> +1
<CClaire> +1
<Makoto_U> +1
<Heather> +1
<kirkwood> +1
<Charles> +1
<kevin> +1
<Rachael> +1
<Wilco> 0, this has been widdled down to meaninglessness
I never knew hat doh actually menas...
Rachael: For the second part, are we listing this in Conformance in some way? Don't think anyone has proposed to make it requirement. Would we ever have a provision that required usability testing at the most basic level of conformance?
GreggVan: Thought we were circling around "in order to conform at any level", might be in conformance level. Lisa said that if other testing is there, wanted to keep this open for discussion. I thought that a long time ago, we'd said that testing was non-normative and so can't be in conformance section. Wouldn't want to say "you have to test this in exactly the way we tell you to."
GreggVan: Might provide *a* way of testing, but never a required way of testing.
Rachael: Don't think we have a resolution on that yet. I've drafted one.
<alastairc> I don't recall a resolution to that effect, but generally we resolve to include things, not to exclude things.
<kirkwood> I do think Gregg brings up a good point
<shadi> +1 to janina
janina: Just because we've never done it, wouldn't want to conclude we can't. But can't see how. At the fundamental level, it would be RFC2119 MUST something. What's the "something" that would include usability testing? Maybe SHOULD. Can't quite formulate that either.
janina: Think what we did before needs to be what we do in the future. Doesn't mean we omit usability testing, but "how" we include it. +1000 to best practices.
<Charles> +1 to janina on aligning usability testing to ‘must’
Wilco: Think usability testing is phenomenally useful, but there's a scope problem. Organizations are required to have fully conforming websites. They already don't measure that; they measure samples. Automated tools. When you get to usability testing, there are even fewer pages you end up covering with those processes. Different layers. Automated can do whole sites; audits a couple dozen. Usability is smaller. How do you bring that together in
one thing?
<Zakim> Lisa, you wanted to say some sections of conformence section could be non normative
Wilco: If usability becomes a requirement, that means far fewer pages can conform because you can't run that many usability studies.
Lisa: Don't think we need to decide that now. Some sections can be normative with an informative subsection. Would we necessarily make the entire section normative? Even in a normative section, could say what you SHOULD do. Mention or reference to testing in a SHOULD section.
Lisa: in WCAG 2, we have partial conformance, that's not a MUST. Don't know what the section will look like.
janina: I think we're making progress. Maybe need to talk some more. Don't see a place for user testing in the normative section. Lots of room in the informative section. Hope to have future conversations.
janina: Lots of constraints to consider. I don't see how we could make a SHOULD work, but don't want to say never.
<Zakim> Rachael, you wanted to say there is a difference between being in the conformance section and being in am assertion referenced by conformance
<alastairc> Conformance is currently a pretty short section, and fairly short in WCAG2 as well.
<kevin> conformance but, for relevant provisions, we will include usability testing in procedures in the informative (understanding type) documents as a means of identifying issues and better meeting the guidelines./
Rachael: Chair hat off, I'm comfortable saying that testing shouldn't be part of the conformance section. It's how we level conformance, but not how we test pieces of assertions or content or requirements. I'm comfortable saying we can't require something that has to have usability testing at the base level.
Rachael: We should be able to say "we're not going to do things". We can always change our minds, but we need to be able to move forward.
GreggVan: Was going to call the question. Resolution to not include a requirement for any type of testing in conformance. How you test, we won't specify anything about testing. How you get there is up to you.
<Jon_Avila> What about usability testing in assertions?
Lisa: In my proposal in the conformance group, I'd suggested a framework, which in the conformance section, you could state how you tested. And people could check if it conforms to their policy.
GreggVan: You're talking about htings that would be included in a claim. Doesn't preclude saying that if you make a claim, you need to say how you tested for it. But not saying that if you make a claim you must have tested it in this way. Since we don't require that you make a claim, nothing about requirements for making a claim are requirements.
alastairc: Conformance section is quite short. Might be the shortest section. Useful that it doesn't specify too much.
<Lisa8> the proposal discusses testing in conformance even if it doesn't say how to test
alastairc: Think you were saying people can use usability testing for _them_ to know they're meeting requirements, but we want to do the slight of hand of not saying how people meet requirements.
Rachael: We never captured the previous resolution, so pasting that in.
Lisa: Depends on the wording. Proposal didn't say how you had to test. Gave a framework of saying how you tested. If resolution is that we don't discuss testing in Conformance section, then we're throwing out that proposal. If we don't insist on how someone tests, in Conformance, that's fine.
alastairc: WCAG2 didn't talk about EARL statements either, so we wouldn't put that into Conformance, but it could go elsewhere.
GreggVan: Thought Lisa had a good point. Provision said we wouldn't include any requirement. ... Maybe we shoudl say "in order to conform"
<Lisa8> ok
<kirkwood> +1
<Detlev> +1
<GreggVan> +1
<Lisa8> we get back to the problem of assertions
<Rachael> +1
draft RESOLUTION: We will not require usability testing in order to conform
<alastairc> draft RESOLUTION: we will not include any requirement for any kind of testing in order to conform
<Jon_Avila> +0 I wonder about assertions with usability?
kevin: Let's revisit this first thing tomorrow?
<Jon_Avila> I wonder if we could allow alternative conformance methods? For example, usability or technical?
<alastairc> Jon_Avila - assertions aren't affected
Rachael: Active people will email to finalize the wording.
various: sleeping? We'll make it work.
<Jon_Avila> Thank you all.
<CClaire> Thank you
<julierawe> Thank you