w3c/wbs-design
or
by mail to sysreq
.
The results of this questionnaire are available to anybody.
This questionnaire was open from 2021-04-29 to 2021-05-04.
17 answers have been received.
Jump to results for question:
Based on the following discussion points from the meeting, can we agree to use a rating scale (adjectival rating) at the outcome (testable statement) level?
Choice | All responders |
---|---|
Results | |
Yes, use a rating scale at the outcome level and test how many levels works well | 8 |
Something else (add to comments) | 8 |
(1 response didn't contain an answer to this question)
Responder | One-line description of question | Comments |
---|---|---|
Jennifer Delisi | Something else (add to comments) | Regarding list item 4 - I'm unclear of why "but not the final scoring approach" is listed here, because I believe this is about the outcome (testable statement) level. Apologies if I am misunderstanding the question. Regarding list item 6 - I do not feel qualified without reviewing research about rating scales and levels, to answer this question. My assumption is that there is research that has tested different scoring models, in a variety of settings. If levels were selected based on this research, along with a researched-method for writing the scoping for levels (helps testers decide where their score falls when it is more ambiguous), then I would find it easier to vote on this question. If there is research the group has reviewed and is considering while answering this response, it would be helpful if a link could be shared. |
Sarah Horton | Something else (add to comments) | Continue to research, discuss, and prototype approaches. |
Stefan Schnabel | Yes, use a rating scale at the outcome level and test how many levels works well | |
Oliver Keim | Yes, use a rating scale at the outcome level and test how many levels works well | |
Jake Abma | Something else (add to comments) | Continue to research, discuss, and prototype approaches. |
Justine Pascalides | Yes, use a rating scale at the outcome level and test how many levels works well | At a conceptual level, yes. |
Makoto Ueki | Something else (add to comments) | Re: 4 - Percentages can be more repeatable and reliable than any other things to present a more nuanced way of showing how accessible their website, app or product is. There will be a number of guidelines (used to be "SC" in WCAG 2) where we can calculate percentages. If it is not the "final" scoring approach, I'd like to confirm how the final approach in this context would be, before I say "Yes". And +1 to John's "concerns over subjective determinations". We should check if multiple testers will get the same results/scores at the end. It maybe at later timing, not now. But the results of multiple assessments should be repeatable. |
Michael Gower | Something else (add to comments) | Maybe you don't want any feedback on your list, but here are a few comments: "Simplicity will most likely come from consistency in how we handle scoring." I'd rather you used "Clarity" than 'simplicity', in almost all cases where you've used the word. "Transparency adds granularity so increases the number of checkpoints" I'm not sure I understand this, or agree with it :) "We resolved that for WCAG 3, testing will aim to improve reliability between testers (from WCAG 2.x) and that we will work on testing to measure this." I'd say "consistent results" rather than "reliability" This is kind of a radical departure from this, and I apologize for just stating it in a survey question, but I wonder if allowing teams to report on design and dev process (its existence and adherence to) isn't at least as important a metric to capture in regard to pursuing accessibility. The ISO approach, if you will. |
Marc Johlic | ||
Jeanne F Spellman | Yes, use a rating scale at the outcome level and test how many levels works well | We have a group working on testing WCAG3 that can test this. We need more people with testing experience to help so we can get better data. |
Alastair Campbell | Yes, use a rating scale at the outcome level and test how many levels works well | |
Andrew Kirkpatrick | Something else (add to comments) | Without more specific examples I can't answer this concretely. Am concerned about subjectivity (e.g., "process"). |
Rachael Bradley Montgomery | Yes, use a rating scale at the outcome level and test how many levels works well | |
David MacDonald | Yes, use a rating scale at the outcome level and test how many levels works well | |
Bruce Bailey | Yes, use a rating scale at the outcome level and test how many levels works well | |
Laura Carlson | Something else (add to comments) | Continue to research, discuss, and prototype approaches. |
John Foliot | Something else (add to comments) | Re: Bullet 4 - I reject the assertion that "percentages do not work across all outcomes so shouldn't be our choice" as pretty much anything can be measured as a percentage. (xx/100) Re: Bullet 6 - "The question is how many levels should be on a rating scale." - Again, this can be handled by using percentages (for granularity), with minimum percentage values then mapping to Bronze, Silver and Gold. I also continue to have grave concerns over subjective determinations, as no two experts will always agree, and we're setting up a point of disagreement between evaluators which will be problematic for regulators. For Question 3 (Scope) the term "Process" requires significantly more clarity: clicking on a link or button will always initiate a process, for example, a flyout menu with 35 menu items = 35 potential process 'starts' (i.e.the start of a process) which is "The user wants to go to a new page". |
WCAG 2 defines conformance against web pages. The first public working draft of WCAG 3 defines scope in terms of views and processes. There have been comments about the need for a better definition of process.
How do you think we should define scope within conformance?
Choice | All responders |
---|---|
Results | |
Use WCAG 2 scope of web page adjusted for non-html. | |
Use the WCAG 3 FPWD scope to views and processes and better define both. | 15 |
Something else (add to comments) | 2 |
Responder | How should we handle scope in 3.0? | |
---|---|---|
Jennifer Delisi | Use the WCAG 3 FPWD scope to views and processes and better define both. | |
Sarah Horton | Use the WCAG 3 FPWD scope to views and processes and better define both. | |
Stefan Schnabel | Use the WCAG 3 FPWD scope to views and processes and better define both. | |
Oliver Keim | Something else (add to comments) | |
Jake Abma | Use the WCAG 3 FPWD scope to views and processes and better define both. | |
Justine Pascalides | Use the WCAG 3 FPWD scope to views and processes and better define both. | |
Makoto Ueki | Use the WCAG 3 FPWD scope to views and processes and better define both. | |
Michael Gower | Use the WCAG 3 FPWD scope to views and processes and better define both. | |
Marc Johlic | Use the WCAG 3 FPWD scope to views and processes and better define both. | |
Jeanne F Spellman | Use the WCAG 3 FPWD scope to views and processes and better define both. | |
Alastair Campbell | Use the WCAG 3 FPWD scope to views and processes and better define both. | I think a key thing to also agree is that (like WCAG 2.x) it is up to the person claiming conformance that determines which views/processes to include. And then it's up to regulators (etc) to determine for certain industries/sectors what type of views/processes they must include. |
Andrew Kirkpatrick | Use the WCAG 3 FPWD scope to views and processes and better define both. | |
Rachael Bradley Montgomery | Use the WCAG 3 FPWD scope to views and processes and better define both. | |
David MacDonald | Something else (add to comments) | It seems a little premature, because I think of this question like an exective summary. Lets get further into the actual methods before we decide what will be most appropriate |
Bruce Bailey | Use the WCAG 3 FPWD scope to views and processes and better define both. | |
Laura Carlson | Use the WCAG 3 FPWD scope to views and processes and better define both. | |
John Foliot | Use the WCAG 3 FPWD scope to views and processes and better define both. | the term "Process" requires significantly more clarity: clicking on a link or button will always initiate a process, for example, a flyout menu with 35 menu items = 35 potential process 'starts' (i.e.the start of a process) which is "The user wants to go to a new page". |
The following persons have not answered the questionnaire:
Send an email to all the non-responders.
Compact view of the results / list of email addresses of the responders
WBS home / Questionnaires / WG questionnaires / Answer this questionnaire
w3c/wbs-design
or
by mail to sysreq
.