Meeting minutes
AP: the first thing that we will start the meeting with is introductions.
<Adam> agebda>
<ryomtoob> I'd like to give an intro
Introductions
RY: I'm Rachael Yomtoob. I'm no longer with Deque, so l've rejoined as an invited expert.
Announcements
<alastairc> probably all day mon & tue
<alastairc> with joint meetings on the thu / fri
AP: TPAC 2026, will be a hybrid event. located in Dublin. AG meetings Monday or Tues.
Kevin: re-chartering- short extension, which we got late in the day because there were, we got, because there were some, uh, last minute changes requested
… have requested permission to progress that for AC review.
HV: I'm pleased to say that we had a great working session last Friday, and that's me and Jeroen Holscher and Steve Faulkner.
… we're at a version that feels like we're ready for note. So we're currently in discussion with with chairs.
<bbailey> https://
PL: I just wanted to draw people's attention to you've received yesterday my usual WCAG 2.x Task Force backlog meeting email.
<hdv> Patrick_H_Lauke++
PL: one's a tiny bit special, because it includes an either-or set of pull requests.
at the start. So we've been deliberating
… we've got two PRs that basically fall on either side of the fence on how to interpret it. So yeah, at this stage we'd be interested if people could have a look at it, thumbs up or comment. just so we can get a feel
… for it.
Spike results: Progress to conformance score — https://docs.google.com/presentation/d/1XshX43F4167UtOU1XKDt7CHcpJyEm4iVhmZXJBp9840/edit?slide=id.p#slide=id.p
rm: we are, uh, implementing some of the comments back from the retrospective last week I did want to request that as people queue today to please go ahead and queue two, so queue plus to say, and put your topic in to help chairs manage topics.
Slideset: https://
ac: We had two day virtual meeting a few weeks ago on conformance. So, various ideas were discussed in how the WCAG 3 conformance could be, could be different from WCAG 2.
… 3 of those ideas were given to small groups, and this is one of them. It was a progress towards conformance score.
ac: And this is literally copy and pasted from, you know, uh, some rough drafting earlier.
that it could be... the idea was it would show progress on the way to conformance. It would be a score, probably a percentage score.
https://
ac: Each person could take, uh, would create a couple of tabs. We would copy across the, um, all the whole list of provisions.
… We would have, columns per page or per view within whatever product it was we were looking at.
… Each row had a provision. We were excluding assertions for the purpose of this
… each cell would be, uh, probably numbered, actually, but we were focusing on pass or fail, or not applicable was counted as not a fail.
… So, passing a not applicable were equivalent, um, so basically put in a 1 if it failed.
… Giacomo, who had instance data, he was also putting in instances, rather than just a pass or fail.
<Zakim> Charles, you wanted to ask the audience or context for: ‘Show progress on the way to conformance’
ac: so in general, this, the average score, we tried calculating using the average number of fails across conformance units.
… So if you had 6 pages, how many pages did that fail across?
… And then we've created an overall percentage for the scope of testing.
ch: The context question for this whole spike of work is, who is the audience or context in which this progress toward conformance would be made?
ac: this would be building on top of conformance.
… I think quite useful for internal teams, if you're working on your product, what have you got left to do?
… And, you know, how far away from that are you?
It can also be used for people like procurement
… But yeah, there are various audiences.
Gregg: On the data sheet, that's the one that broke down all the provisions and tried to list their benefits for each different disability and give them a weight.
… The goal there was to address a couple of issues
… including - What if you solve the easy stuff, but the stuff that's really important you skip? So that's why, you know, 1, 2, 3 rating was included.
… And the other one is, what if you solve all the problems for one group and leave the other group hanging?
… And that's the reason why it has 4 columns.
detlev: Wondering whether this spike group considered having a graded result?
<Charles> thought: in the US, orgs are risk averse. hence accessibility statements have “we strive to meet WCAG 2.2 AA” to strategically avoid any specific claim of conformance. it would take significant culture change to convince orgs to publicly disclose the progress toward conformance.
ac: yeah. So we did look at the first two case studies I'm will show , how to break down the instances.
… we did talk about that kind of severity issue, and I will come to that in the conclusions.
<kevin> Charles, it wouldn't be for use to ask organizations to disclose anything. This is really just providing a way to answer the questions around distance from conformance. It might be the case that regulators request that any such scoring be made public.
ac: including the proportional instances seems to help. It correlates more closely to the user experience,
<Zakim> Ben_Tillyer, you wanted to ask about the category weighted scores
ac: Given that this is just two sites, we would need to rerun this across more sites
… from the other sites we were working on, that most sites score above 50% by default, even pretty awful ones.
… You need to include a lot of different types of content in order to score worse than 50%.
BT: Amazing. The range that you've you've got for the per disability categories. And I was thinking that if I had those ratings of any product.
… Beyond, I would probably look to ignore the the higher percentage points. If I was using the disability category.
gregg: Yeah, the question was, you said it had poor experience. and the, what does poor experience meant?
… Is it for someone with a visual, or a cognitive, or a hearing, or physical, or with no disability?
<Zakim> Rachael, you wanted to speak to the categories use
GP: So essentially I have created in the other sheet in the website sheet a visual score because the audience of this user testing was primarily people with visual disabilities.
I guess my impression of the categories was that they weren't going to so much be weighted as they were setting minimums to ensure.
rm: My impression of the categories was that they weren't going to so much be weighted as they were setting minimums to ensure.
<Zakim> Jennie_Delisi, you wanted to ask about the weighting
gregg: it might be useful to have two scores if you're doing this. One of them is overall, including pass if you don't have this, and one which is.
gregg: don't count it if you don't have it, um, because that would help you to not have somebody think it's really good when, in fact, it really... nobody's even tried to make it accessible.
<Zakim> Jennie_Delisi, you wanted to ask about option to amplify the difference
ac: it's worth saying, so this website, the first website, is a big content site with a lot of stuff on each page, which is why you've got so many checks.
And why, you know, because you're having to count every instance that could have failed.
jd: If we are using the scale of 100. for the percentage-wise into limiting away. So if you're saying the biggest range is between 60 and 95%.
<Zakim> giacomo-petri, you wanted to mention the opposite of these "high" numbers
jd: If that number could then have a way of getting amplified in terms of its granularity, we might notice a greater difference between certain ones, and I'm not sure if this.
<Jennie_Delisi> This is fantastic information - really helpful
GP: If we look at the, not the average value, but the final success criteria that are passing, they are 19 out of 55, which in terms of conformance, it is a very low score.
… But this is not reflected by the user testing. So it's a tricky one because, yeah, probably with those with distance towards conformance, numbers are quite high.
… But on the other end, the current conformance score based on pass-fail success criteria, it is the opposite, so you are scoring very low, even though your website is not so bad.
ac: the way I'm thinking about that is because this site has really big pages with a few issues on it that are spread out, but there's quite a variety of issues across the pages as well.
… It sort of hits all these triggers, but somebody who's actually using the site is only going to be encountering a subset of those, because they don't explore the whole of every page.
ac: Andrew Hick had an unweighted score. We've got the average pass rate, and then the weighted score.
ac: the judgment call on severity was the hardest part of training. Although the results were fairly consistent.
ac: unless it was built without responsive design, by the time the team reached 4.8, all of the functional needs were supported except for blind and motor impaired people
alastairc: [Slide 29] When trying to align the score with user experience for pwd, the following factors: Severity, proportionality, provision level, and disability categories. Thinks we may need to add a 'safety' level.
<Detlev> is the link to the presentation in IRC?
<Detlev> thanks, Adam!
alastairc: Reads [slide 30-31] Adding complexity to the site, you are more likely to have barriers. There's a potential, depending on how the provisions are written to create reverse incentive for things. This would handle cases for not applicable.
GreggVan: One slide said "Except people who are blind and have physical disabilities" - does this mean one or the other, or people that have both.
alastairc: Good question, unsure, but can find out.
GreggVan: Idea of subtracting 50 from the score, thinks it might be a good idea because it seems that anything that scored less than 50 would be pretty much inaccessible.
<Zakim> wendyreid, you wanted to ask about positive gamification
wendyreid: Noticed that we put notes about gamification, and wonders if there's a positive side to gamification that could be considered. Knowing that gamification can positively influence motivate organizations to strive to achieve. One of the possible conclusions is around severity, and maybe give examples and guidance. for example, checkout flow
… has a checkout button that's not labeled is more severe than an unlabeled banner. People need to understand how to get the highest gain in both usability and in the score, this might be motivating. It's hard to determine where to put effort to get those numbers up.
<Zakim> Rachael, you wanted to say I think gamification is outside of conformance
<Zakim> GreggVan, you wanted to say that Gamification and Gaming are two completely different things. (the fact that they use the same root "game" is a quirk of English) Gamification is great as Wendy pointed out. Gaming the system is not.
Rachael: Chair hat off. I think gamification downside for strictly conformance hooks would be naturally there if we decided to go with a scoring conformance model. Gamification really does support organizational motivation, it's a little harder to build into law, but it would be a great place of a policy document also applies to orgs to talk about
… how to use the conformance model to gamify and promote accessibility improvements.
GreggVan: Wanted to point out that gamification and gaming the system are completely different concepts.
<Zakim> alastairc, you wanted to comment on gamification and positive efforts
GreggVan: and +1 to Bruce comment
<Zakim> bbailey, you wanted to ask if subgroup considered tiers? Namely, 50%=F 60%=D 70%=C 80%=B 90%=A and then S tier
<bbailey> Did subgroup considered tiers? Namely, 50%=F 60%=D 70%=C 80%=B 90%=A and then S tier for assertion of full conformance plus supplemental best practices.
alastairc: Chair hat off to reply to Wendy. Severity: we can take is the most useful thing in terms of helping align whether we can gamify that. The problem I see is that it's very hard to consistently say what things would be considered severe across the completely different kinds of interface that this gets applied to the different sizes.
… Suggestion would be to keep the conformance scoring as simple as possible, but to build on that. You may sit down as an organization and determine what are the most important tasks (within the product) for us and the users, and document if there's a legal case, or regulator knocking on your door. You need to take that into account severity and
… provide recommendations on what to do next.
bbailey: Love the distance from conformance, and is fascinated about how much work all the math is. People going to go full conformance claiming some of the supplemental best practices, and they could be put in X tier.
alastairc: Focus on the distance from conformance, and think about ranges within.
<kirkwood> in practice that ‘severity” can depend on the number of occurrences on a site. or the “importance” of the path b locked. even the constiuant group blocked etc.
<Zakim> wendyreid, you wanted to mention flows as part of this too
Ben_Tillyer: Thinking about how to widen this number of what range we're looking at. Wonders if there's something we can do in terms of looking at the number of defects on a site and the disabilities that may be impacted for the site. Perhaps a progressive exponential scoring for each defect, instead of them being weighed all the same?
<Zakim> Jennie_Delisi, you wanted to say have some standardized severity items and room for more
wendyreid: Thinks there are implications, and challenges is implying things like flows. There's "like" things, such as occurrences, location of occurrences. There may be repetitive elements across pages (such as a website header), and it may be frustrating to continually come across the same defect. How could we account for the types of things? Are
… they additive so they are weighted? Segment it? We want to make sure, especially in large organizations that have different teams split into different parts, that those are communicated and accounted for.
<Zakim> Rachael, you wanted to say keep the math simple and visible
Jennie_Delisi: Want to make a point on a phrase that was in the severity levels. There is the internal reporting around this, but then there's the view reporting from multiple products as a comparison method to see which product would be more accessible to their users. If we can at least have a standardized list of severity definitions that would
… reduce the requirement to have nondisclosure agreements gathered prior to being able to view this kind of information. I think we can better support people choosing accessible products will help the goal long-term.
<Zakim> alastairc, you wanted to comment on Ben's comment, and how you differentiate large interfaces from small and to also ask about severity definitions and how they apply across interfaces
Rachael: Chair hat off. The more complex the math, the higher the risk of non-adoption. Want to continually consider this as we figure out how to score.
<giacomo-petri> +1 to Alastair
alastairc: Chair hat off, on Ben's comment about increasing the weighting by the number of instances. It increases the complexity, but it could also distort things because of the size of the product. It also applies to apps, such as watch apps, which have a more constrained interfaces, or a govUK page, There's such a variation of results. If you
went down the weighting, then you might be penalizing products that have a much larger interface. Also about Jenny's point about severity defitnions, there's a question around how you would have a definition of severity. I think severity is one of those things that you need to sit down and define how that should work in your context.
<bbailey> I am also worried about the maths, but that the numerical results maps to ABCDF seems like a happy coincidence that makes scoring more approachable.
<Zakim> giacomo-petri, you wanted to talk about "counting" issues
<bbailey> +1 to theo's offer !
HaTheo: Understanding Rachael's point, if you increase the complexity of the way you calculate the score, you might influence adoption. Another question is hard is it for companies to calculate the score, and interpret the score? In the sense that looking at the score; offered to write down the comment (yes please do HaTheo)
<Zakim> alastairc, you wanted to comment on needing a buckload of data to work out a normalised score.
<kirkwood> +1 to Giacomo
giacomo-petri: Want to echo Alastair said about the number of issues is definitely a key factor. There is a risk of disproportionate penalizing larger pages. and also severity is important, but it is again subjective - there are just too many factors that affect the counting and the severity thing, and it's not easy.
<Rachael> +1 to context in severity, want to add disability to context list
<kirkwood> Also the severity can depend on many factors, including the business priorities of the content owner.
alastairc: Wanted to reply to Theo, chair hat off. Having a normalized average would require a bucket load of data. We'd have to have good test results across a wide range of sites. Would need a plan and somebody to pay for it. Getting a rant of text data of sites and working out what those ranges would be to give a more useful representation.
<Jennie_Delisi> severity examples could include: possible loss of life issues, severe weather information, etc.
<kirkwood> agree C passes
<Rachael> +1 to clearly defining what level passes/conforms fully
<bbailey> The regulator can say what grade is good enough.
GreggVan: +1 to Bruce's idea, and thought about using grades, but caution using grades after thinking through it.
JeroenH: We used the ABCDE scale in the Netherlands as well. We don't have the misinterpretation of people that C might be good enough. It depends on how you interpret the scale. We use an interpretation more like energy label on the scale.
alastairc: Pulled up an image showing ABCDE on an energy scale.
alastairc: Wrapping this up, we did briefly consider some potential reporting requirements, and to emphasize, this isn't conformance. This is separate from conformance and building on that to allow people to differentiate between not passing at all and being fully conformant. It might help to include a could have factors. As part of a conformance
… claim, you're going to state which pages are included and for this scoring, it might help to include things like media count, component count, or other things which show how much surface area you've got as part of your interface.
<kirkwood> Potential metrics. Also used in architecture: Percentage of Compliant Elements. Distribution Ratios. Accessible Route Coverage. Task Success Rate. Wayfinding Efficiency
alastairc: Invites others to contribute, especially those with testing data, and emphasized this is a very generic thing. This is for the next editor's draft. It could be its own section within WCAG 3 or a completely separate document.
<bbailey> GSA has quite a bit of data from 2024 and 2025 annual survey: https://
<bbailey> https://
kirkwood: Comment in IRC above. May want to consider paralleling some of the metrics around architectural accessibility. Percentage of compliant elements, distribution ratio of the compliance. Tests the success rate and way finding efficiency, and usability and the accessible path. Maybe somethings that we're overlooking that can be included here.
A little concerned that we maybe have not had quite enough feedback within the blind community for what we're doing.
bbailey: Has quite a bit of data with downloads with WCAG 2 criteria type. The federal gov settled on compiling.
Poll: Should we continue to test the Progress to Conformance score?
<Adam> +1
alastairc: Wants to take a general temperature of the group.
<Ben_Tillyer> +1
<JeroenH> +1
<ryomtoob> +1
<Glenda> +1
<AWK> +1
<bbailey> +1
<ShawnT> +1
alastairc: +1 if you think this has been useful and we should carry on testing it (getting a wider set of data) or minus one if you don't think we should.
<Rachael> 0 because based on previous conversations I am not sure the group is going to be OK with some core provisions not being "required"
<janina> +0 Interesting, but I have concerns that might be called concerns over category confusions
<Charles> +1 testing will reveal something not yet considered
<Detlev> if this builds on conformance and we do not knoe yet how to measure that, this might better be on the back burner?
<Detlev> +0
0, I don't know if more data would be beneficial to 'prove' anything with this approach.
<CClaire> +0
<Zakim> giacomo-petri, you wanted to answer
wendyreid: what about usability testing as part as a reference point?
<jtoles> +1
0 share Delev's concern.
<Stephanie> +0 too confusing or complex for general public
<HaTheo> +1 to this direction, it’s a really solid starting point. However, to actually apply this at the scale of the broader audience reading WCAG, we need to introduce more rigor I think. Without data normalization and tracking Inter-Rater Reliability (IRR), the current testing approach just isn't ready to handle that kind of volume and variance... I
<HaTheo> ... also would be happy to help think about this.
giacomo-petri: Essentially these are testing subdivided in tasks. Don't want to guide the user in a defined path. In one of the sheets, there is our internal score which is really a progress toward conformance which is not intended to measure usability or the effective accessibility of the products, since we count all the issues and the total of
… numbers of checks that we perform. It is a metric that helps you understand how many things you're doing right and how many things you're doing bad. The difference is the effort that you have to put essentially to reach conformance, but has nothing to do with the usability of the product. There's a lot of subjectivity aspects that are tricky to
… manage objectively.
<kirkwood> FYI regarding building accessibility metrics: Building accessibility conformance is typically measured using physical design standards, spatial metrics, and functional evaluations rather than a single numerical score. The specific metrics used depend on the regulatory framework being enforced.
giacomo-petri: The user testing is performed by PWD following specific tasks without too much guidance that would jeopardize the user testing itself.
<Zakim> alastairc, you wanted to comment on difficulties
Detlev: concern that with this score towards conformance. I feel we are dancing around measuring conformance for a number of years now, and if this builds on conformance it might be wiser to try to make difficult decisions regarding how we're going to measure conformance and then move into the things around this.
<Detlev> Heather, "unknown" was Detlev speaking
<Zakim> Rachael, you wanted to ask about required provisions
alastairc: There's a starting assumption that conformance is meeting conformance. There's a basic conformance and assumptions. Comments about more data 'proving' anything. We might be able further understand the barrier scores with numbers that are close. We had the correlation between a fairly simple score and the subjective, if we had the
… usability testing score, and had the correlation it could be useful.
<AWK> alastairc Would be interesting to see the barrier score correlation for each major disability group
AWL That would be possible for the 4 categories we use (keyboard / low-vision / screenreader / cog + deafness)
<GreggVan> s/ disability fails/provision fails/
<Zakim> wendyreid, you wanted to ask about the core goal of conformance
Rachael: Thinks this data is helpful and it shows some promise with scoring. We had a conversation about a year ago that basically said the group was not comfortable with having every provision up for grabs. Basically what the said is in the current version is what the group was fairly comfortable with as must pass. With that baseline, wants to
… clarify if there is a different way of looking at conformance where we focus on the core, and weight them higher, if there's another way to look at it from a reporting standpoint separate from conformance. (chair hat half on, half off).
<HaTheo> I feel like the conformance model is being pass fail today is not loved as it makes things like filling out ACRs really nebulous. I think the issue is that these are really hard to run a program on, as it's not motivating to compare across products and track progress if you don't know what is "normal" vs "exceptional"
<Rachael> +1 to exploring a strict "conformance model" and additional approaches as part of a policy/reporting document
wendyreid: Questioning how we frame this, or how we frame this for ourselves. We all want the web to be accessibility, but being completely realistic is that it's really hard to be 100% conformance. There are different pathways to get there. Framing this is critical. Like with any other spec, to conform to WCAG 3, you must achieve all the
… applicable criteria to your product. Knowing this is difficult, here are the ways you can approach that: leveling, grading, tiers of criteria, etc. To track progress, measure progress or measure achievement, here are the different grads and stages to get to, and the challenge to make them approachable and possible?
<Zakim> GreggVan, you wanted to say "if there is a good correlation between disabilty fails and conformance -- would there not also be the same correlation between passes and conformance? If not can you explain: (or is it too mathy)
<Ben_Tillyer> +1 to Wendy. Been working out how to put this into words. Give people different routes that we know have some benefits, ask them to pick one and to justify why?
<Stephanie> +1 to Wendy
<giacomo-petri> +1 to wendyreid
GreggVan: +1 to Wendy's comments. If we strive for perfection, anything short is a failure. If we can combine Rachael's, let's make sure we can get all the critical showstoppers and provide a path for people to keep moving up that would be good. Another question: You said there's a high correlation between fails and fonromance or something.
… Wouldn't there not be the same correlation with passes since fails and passes where you can submit one by subtract the other or am I missing something?
<Zakim> giacomo-petri, you wanted to talk about "stop when you reach a level"
alastairc: Comparing two things, one was WCAG fails, and to reverse that. The barrier score is subjective based on impact of those fails. So to me this is saiying the more WCAG files you have, the more likely there are to be more impactful issues. Subjective by the auditor who looks at each issues and determines that it's going to block you, or its
… not that important.
<wendyreid> +1 to Giacomo
giacomo-petri: Agrees with Wendy's point, understands that you need to ensure that companies are making progress and they don't stop when they reach the phantomatic legal requirement. This can be addressed by policies but especially in the context of large companies, we need to acknowledge that maintaining a consistent level is not as
… straightforward as it might seem due to frequent updates.
<Rachael> +1 to taking the training aspect and novice perspective in mind
<Zakim> alastairc, you wanted to agree with Wendy and dealing with flashing
Stephanie: Agrees with Giacomo and Wendy. From my perspective, we know we're talking to companies who provide products for Sydney government who have non accessibility specialists, people need to be told what to do. 250 departments and 600k staff is a huge scale. The more rigid conformance model the better. Not everyone is reading the WCAG spec.
<giacomo-petri> +1 on supporting, sorry Adam
<Zakim> bbailey, you wanted to mention federal expectation is full conformance
alastairc: Anytime we try to define severity, or which parts of the product should be included, we're going to run into the problem of the infinite variety. What this means is the variation of digital interfaces. It's hard to define that kind of thing, and is better left of to people to make those judgement calls. The case of judgement doesn't
… really sit in the conformance model, so I agree with that. The questions are really: Do these numbers actually help? Do the scores actually make sense. Not sure at the beginning if they did, but then considering the granularity. Thinks there should be immediate non-conformant failures (like flashing).
bbailey: Want to plug the Section 508 data again. It's a question of quality. Self-reported versus audited.
<Zakim> Rachael, you wanted to react to bbailey to answer alastair
alastairc: We don't have the audit results, we could create scoring for those, but we've got some usability measure, and it's tricky to compare the data.
Rachael: Side conversation with Bruce and I was having us get the usability side of ti to be able to compare it. thinks we can get some of it.
bbailey: There are some cover pages, summary sheets for each, I think it's there.
bbailey: There are some cover pages, summary sheets for each agency.
alastairc End of question, end of presentation, from the poll, it looks like we're supporting continuing. Asks for other members to contribute to the conformance subgroup.
Adam: Thank you to everyone for participating. see you next week.