W3C

Site Navigation

Nearby

Results of Questionnaire Updating Editor's Draft with Testing

The results of this questionnaire are available to anybody.

This questionnaire was open from 2022-04-21 to 2022-05-24.

18 answers have been received.

Jump to results for question:

  1. Approval to add testing section to editor's draft as exploratory
  2. Approved: Agree with removing ratings, scorings, and critical errors
  3. DEFUNCT: What are outstanding questions on the new approach?
  4. DEFUNCT: Agree with adding in test type content as exploratory
  5. Terminology

1. Approval to add testing section to editor's draft as exploratory

The test section has been updated to reflect survey comments and discussion from the April 26th meeting.

Do you:

Summary

ChoiceAll responders
Results
Approve the testing section content be added as exploratory 10
Approve the testing section content be added as exploratory with additions to the editor's note (see comments) 2
Approve the testing section content be added as exploratory with adjustments (see comments) 2
Something else

(4 responses didn't contain an answer to this question)

Details

Responder Approval to add testing section to editor's draft as exploratoryComments
Alastair Campbell
Rachael Bradley Montgomery
Michael Gower
Gregg Vanderheiden
Jennifer Strickland Approve the testing section content be added as exploratory
John Foliot Approve the testing section content be added as exploratory I am approving the addition of this content so that the WG can now start the "exploration" part of the Process. By doing so it is my expectation that the exploration and vigorous discussion around the 4 types of proposed test types can happen.

"Exploratory: The working group is exploring what direction to take with this section. This content is not refined, details and definitions may be missing. Feedback should be about the proposed direction. It is hidden by default."

My approval here IN NO WAY is an approval of all 4 test types, and in fact I plan to object quite strenuously to at least 1, and possibly 2 of the proposed test types when that discussion commences.

It is also my expectation and desire that the discussion on Test Types commence soon.
Andrew Somers Approve the testing section content be added as exploratory with adjustments (see comments) 1) "Unconditional" is not a good term for what should be called "objective" or "definitive" or more plainly "Pass/Fail". This is also the sole test type that should give rise to defining legislation or legal statute, with the defining "MUST/MUST NOT" result.

Related: because WCAG has been promoted to legal statute in some instances, it is important to consider consequences and ramifications of any SC that defines a MUST/MUST NOT property. As such, it is important for the document to clearly demarcate or categorize those criteria that have adequate support for such legal standing versus criteria that can only support a softer level of "guideline" with a "SHOULD/SHOULD NOT", which should not be ascribed or promoted into a legally binding context.

2) "Conditional" seems to be describing a criteria that has a subjective attribute, and is potentially a criteria where a range-based score would still be appropriate. I think a better term here might be "qualified" and the strongest binary result being "SHOULD/SHOULD NOT" but in this case, a score or range result should be permissible in some cases as a function of design guidance—but at worst to a level of "warning" and not a level of "fail."

3) "Conventional" this section is very confusing, as it spends most of its time discussing "conditional" ???? Instead, let's discuss use-cases.

In the currently developing APCA Readability Guidelines, we have a set of use cases. I discuss the use case concept in more detail here:

https://github.com/Myndex/SAPC-APCA/discussions/39

This is based around the fact that the use-case for a column of body text has very different design requirements than that of a large bold headline, or say a small copyright note in the corner, etc.

IMO "Use Case" or "Contextual" is a better term here than "conventional" which is ambiguous.

4) "Procedural" I think "Process Evaluation" is a more descriptive terminology.


Gundula Niemann Approve the testing section content be added as exploratory I'm not happy with the naming of the test types. Therefore I appreciate the editor's note which asks the public for suggestions.
Laura Carlson Approve the testing section content be added as exploratory Agree that we need to explore what direction to take with this section.


Re-defining some of these tests to be subjective seems problematic to me. This is especially true if WCAG 3.0 will be adopted into regulations.
Suzanne Taylor Approve the testing section content be added as exploratory I think the plain language section should be simpler. Using terminology suggested in question 7, maybe something like:

Basic Checks: Testers will do simple tests. Testers' opinions or judgements are not part of these tests at all.
Basic Evaluations: Testers will do simple tests and they will use opinions or judgements. But, most testers will agree most of the time.
Conditional Checks: Different sites or parts of sites will be tested with different groups of tests. First, testers will decide which group of tests is best for what is being tested. Then, testers will do the tests from that group of tests. The tests in the groups can be basic checks, basic evaluations, or both.
Process Verification: Testers will write down that a process has been followed. The processes improve accessibility.
Shawn Lauriat Approve the testing section content be added as exploratory
Charles Adams Approve the testing section content be added as exploratory
Todd Libby Approve the testing section content be added as exploratory
Sarah Horton Approve the testing section content be added as exploratory
Wilco Fiers Approve the testing section content be added as exploratory with adjustments (see comments) I don't feel great about explicitly saying that procedural tests have reduced inter-rater reliability. I don't think that's necessarily true, and feel it's too early for us to give up on inter-rater reliability for process testing. Lets first figure out if we can make it work with a high degree of inter-rater reliability. To address this, remove the following phrase: "which higher than that of a procedural test".

I don't think "Text has minimum contrast" is an unconditional test. It's certainly not a great example of it. It relies on the "purely decorative" and "human language" definitions, which have a good deal of grey area. In ACT lingo this isn't an objective rule. Very few rules actually are. I think this needs to be replaced with the "HTML page has non-empty title" rule.


I'd like a question added to convention tests. I think an open question is how organisations can declare which convention they are following in a way that is transparent, both to their users and for the purpose of verification in audits.
Andrew Kirkpatrick Approve the testing section content be added as exploratory with additions to the editor's note (see comments) All in favor of exploration to validate whether this is a viable approach.
Bruce Bailey Approve the testing section content be added as exploratory with additions to the editor's note (see comments) I am comfortable as-is (for exploratory) but +1 to concerns (in this survey) for the labels of the test type categories.

I suggest non-conditioned or less-subjective over unconditional.

We might even consider resorting to generic categories (Type I tests, Type II tests, etc.) while exploring better labels for the four types of tests.
Jeanne F Spellman Approve the testing section content be added as exploratory

2. Approved: Agree with removing ratings, scorings, and critical errors

Do you agree with removing the content about ratings, scoring, critical errors and conformance related to these from the draft?

Summary

ChoiceAll responders
Results
Agree 10
Agree with some adjustments (comments) 1
Disagree (reasoning in comments)

(7 responses didn't contain an answer to this question)

Details

Responder Approved: Agree with removing ratings, scorings, and critical errorsComments
Alastair Campbell Agree
Rachael Bradley Montgomery Agree
Michael Gower Agree I think it caused a lot of churn during the current charter. My conclusion is that until we have more things to actually assess using defined methods and outcomes, it's difficult to assess and apply. At the end of the next charter period, I'm hoping we may have advanced far enough to fully understand approaches and ramifications for measuring and rewarding accessible content of different maturities.
Gregg Vanderheiden Agree
Jennifer Strickland Agree
John Foliot Agree By "removing the content" I am interpreting this as "the content will be physically removed (deleted) from the current draft" - which is what I am noting agreement to.
Andrew Somers
Gundula Niemann Agree
Laura Carlson
Suzanne Taylor
Shawn Lauriat
Charles Adams
Todd Libby Agree
Sarah Horton Agree with some adjustments (comments) Would it be possible to leave the errors section and remove the rating section? Critical errors are about severity of issues as experienced by users rather than about tests, ratings, scoring and conformance.
Wilco Fiers
Andrew Kirkpatrick
Bruce Bailey Agree
Jeanne F Spellman Agree I don't think we are done with these items forever -- they may turn out to be the best approach with modifications. However, it doesn't make sense to have them both in the same draft -- too confusing.

3. DEFUNCT: What are outstanding questions on the new approach?

Part of the process we outlined includes an editor's note that captures outstanding questions about the approach we are currently exploring. The current editor's note includes:

  • How to document procedure tests and whether the results of the procedure should be part of the evaluation,
  • How well this approach to testing supports additional requirements not addressed in 2.1,
  • Testing the accuracy, reliability, repeatabilty, etc. of this approach,
  • How well this approach will support regulatory needs, and
  • Developing examples of each test type.

What changes or additions would be needed to this list before this section could be added?

Details

Responder DEFUNCT: What are outstanding questions on the new approach?
Alastair Campbell
Rachael Bradley Montgomery Full list after Friday's Silver Meeting:
- Developing detailed examples of each test type,
- What to document for procedure tests,
- Whether protocols is met by procedure tests or, if not, how it will be incorporated
- How well this approach to testing supports additional requirements not addressed in 2.1,
- Testing the accuracy, reliability, repeatabilty, etc. of this approach,
- How well this approach will support regulatory needs,
- How a process defined and documented,
- How these tests will be integrated into a conformance model (including levels or scores), and
- Get feedback from designers, developers and other communities on word choice.
Michael Gower My sense is that as we continue some of our current exercises, we should get clarity on this. I'm aware we are against a looming deadline for recharter, but some things just take time. My sense is the chairs are doing what they can. Getting WCAG 2.2 to the next stage would help us all focus on this
Gregg Vanderheiden
Jennifer Strickland
John Foliot
Andrew Somers
Gundula Niemann further questions:
Is the naming of the various test approaches appropriate. are there better options?
Are there further test types that should be added?
Laura Carlson
Suzanne Taylor
Shawn Lauriat
Charles Adams
Todd Libby I think there is a typo here, "include the criteria being testest and guidance", shouldn't that be "...being tested..."?

Also, "guidance on evaluating how well the content meets the test criteria for conditional test.", I think could be re-worded "guidance on evaluating how well the content meets the criteria of the conditional test."

"Procedure tests include assistive technology testing" I think could also be re-worded to, "Procedure tests could include assistive technology testing".
Sarah Horton
Wilco Fiers
Andrew Kirkpatrick
Bruce Bailey
Jeanne F Spellman I don't think we need to think of everything. That's part of the purpose of public review. It's a good start and communicates we don't think the work is done.

4. DEFUNCT: Agree with adding in test type content as exploratory

Assuming all the outstanding work discussed in question 2 is incorporated, what changes are needed before we can add this exploratory work on test types and scope to the editor's draft?

Summary

ChoiceAll responders
Results
The content can go in as is 6
The content needs the adjustments noted in the comments before it goes in 2
Something else 2

(8 responses didn't contain an answer to this question)

Details

Responder DEFUNCT: Agree with adding in test type content as exploratoryComments
Alastair Campbell The content can go in as is A couple of wording things I wasn't sure about:
Under Objective tests, "Methods using objective tests include the critera being tested against." Perhaps I'm confused by the word criteria here (which is used in a different way in WCAG2), but I'm not sure what this paragraph is trying to say.

"include the criteria being testest and guidance", should that be "tested"?

"guidance on evaluating how well the content meets the test criteria for conditional test.", should that be "guidance on evaluating how well the content meets the criteria of the conditional test."

"Procedure tests include assistive technology testing", I think that should be "could include", we might find reasons not to include some of these in future.

Rachael Bradley Montgomery The content can go in as is
Michael Gower The content needs the adjustments noted in the comments before it goes in An attempt to strengthen the terminology in order to get us all on the same relative page, to reduce confusion and align different sub-groups' goals, is likely in order.
Gregg Vanderheiden Something else my concern is that these are not exclusive categories. For example a conditional test should still be objective. In fact all tests need to be objective to be a test. Even a procedure test should be objective -- everyone should agree whether a procedure has been carried out or not -- or else it is not a test -- or at least cannot be a requirement.

In school we had tests that were arbitrary -- the judgement of the teacher. But to require something - requires that it be something that someone can determine if they succeeded or not. it needs to be objective to be required.

MAYBE if we changed the first category to Unconditional, conditional, conventional (not sure about this one), procedure

So this needs to be rethought a bit before it goes in. Intersested to hear the discussion about this at the meeting.


Jennifer Strickland The content can go in as is
John Foliot
Andrew Somers
Gundula Niemann Something else I assume that working on the comments on question 2 might initiate further changes (like adding test types, renaming test types, fixing explanations, adding examples, ...) which in turn need agreement and maybe adjustments.
I feel all this needs to be completed.
Laura Carlson
Suzanne Taylor
Shawn Lauriat
Charles Adams
Todd Libby The content can go in as is
Sarah Horton The content needs the adjustments noted in the comments before it goes in The concept of convention tests is pretty new and needs more discussion in the group. I suggest removing it for this version.

In defining what is being tested, is a “subset of a view” really “a subset of items in a view”?
Wilco Fiers
Andrew Kirkpatrick
Bruce Bailey The content can go in as is
Jeanne F Spellman The content can go in as is

5. Terminology

The test types are currently called objective, conditional, convention and procedure. The scopes are currently called item, user process, view, and aggregate. Please include alternatives for any terminology you do not support within the comments field.

We will not discuss this question at the upcoming call but will use the results to craft a later survey.

Summary

ChoiceAll responders
Results

Details

Responder Terminology
Alastair Campbell
Rachael Bradley Montgomery "constant" instead of "objective"
Michael Gower 'conditional' and 'convention' are the toughest to understand without any context.
Gregg Vanderheiden my concern is that these are not exclusive categories. For example a conditional test should still be objective. In fact all tests need to be objective to be a test. Even a procedure test should be objective -- everyone should agree whether a procedure has been carried out or not -- or else it is not a test -- or at least cannot be a requirement.

In school we had tests that were arbitrary -- the judgement of the teacher. But to require something - requires that it be something that someone can determine if they succeeded or not. it needs to be objective to be required.

MAYBE if we changed the first category to Unconditional, conditional, conventional (not sure about this one), procedure
Jennifer Strickland
John Foliot
Andrew Somers Terminology Suggestions:

None of the 4 test terms in the draft come across to me as clearly definitive or meaningful in view of their explanation.

Unconditional tests, —— suggest "Pass/Fail" or "Definitive" pr "Objective"

Conditional tests, —— suggest "Qualified" or "Subjective Evaluation"

Conventional tests —— suggest "Use Case" or "Contextual" or perhaps "Relative Matrix"

Procedural tests —— suggest "Process Evaluation"


## _Minimum Scope Categories_

In considering a breakdown of scope categories, I see adding at least "use case", "mode" and "device" and I discuss more at: https://git.myndex.com/scopeconsiderations.html

- Item
- Use-case
- Mode
- View
- Aggregate
- Device
- Process
Gundula Niemann I not agree with the naming 'item', as in fact it refers to user interface component. It should be named so, that us, the therm 'user interface component' should be used.
I do not agree to the name 'conditional', as conditional suggests that a precondition shall be fulfilled, and only then something applies. I suggest to use 'qualitative' instead.
Laura Carlson
Suzanne Taylor Maybe something like:

4. Evaluating
4.1 Types of Evaluations
4.1.1 Fixed Objective Tests
4.1.2 Fixed Subjective Evaluations
4.1.3 Convention-based Objective Tests
4.1.4 Process-based Subjective Evaluations

Or maybe:

Basic Checks
Basic Evaluations
Conditional Basic Checks and Evaluations
Process Verification



Shawn Lauriat
Charles Adams
Todd Libby
Sarah Horton
Wilco Fiers
Andrew Kirkpatrick
Bruce Bailey I am not sure I am grokking the difference between convention and procedure.
Jeanne F Spellman

More details on responses

  • Gregg Vanderheiden: last responded on 26, April 2022 at 15:30 (UTC)
  • Jennifer Strickland: last responded on 20, May 2022 at 19:07 (UTC)
  • John Foliot: last responded on 20, May 2022 at 19:54 (UTC)
  • Andrew Somers: last responded on 22, May 2022 at 01:23 (UTC)
  • Gundula Niemann: last responded on 23, May 2022 at 12:14 (UTC)
  • Laura Carlson: last responded on 23, May 2022 at 15:35 (UTC)
  • Suzanne Taylor: last responded on 23, May 2022 at 16:39 (UTC)
  • Shawn Lauriat: last responded on 23, May 2022 at 17:21 (UTC)
  • Charles Adams: last responded on 23, May 2022 at 19:07 (UTC)
  • Todd Libby: last responded on 23, May 2022 at 23:23 (UTC)
  • Sarah Horton: last responded on 24, May 2022 at 08:24 (UTC)
  • Wilco Fiers: last responded on 24, May 2022 at 11:19 (UTC)
  • Andrew Kirkpatrick: last responded on 24, May 2022 at 11:35 (UTC)
  • Bruce Bailey: last responded on 24, May 2022 at 12:41 (UTC)
  • Jeanne F Spellman: last responded on 24, May 2022 at 13:39 (UTC)

Non-responders

The following persons have not answered the questionnaire:

  1. Chris Wilson
  2. Lisa Seeman-Horwitz
  3. Janina Sajka
  4. Shawn Lawton Henry
  5. Katie Haritos-Shea
  6. Shadi Abou-Zahra
  7. Chus Garcia
  8. Steve Faulkner
  9. Patrick Lauke
  10. David MacDonald
  11. Gez Lemon
  12. Makoto Ueki
  13. Peter Korn
  14. Preety Kumar
  15. Georgios Grigoriadis
  16. Stefan Schnabel
  17. Romain Deltour
  18. Chris Blouch
  19. Jedi Lin
  20. Kimberly Patch
  21. Glenda Sims
  22. Ian Pouncey
  23. Léonie Watson
  24. David Sloan
  25. Mary Jo Mueller
  26. John Kirkwood
  27. Detlev Fischer
  28. Reinaldo Ferraz
  29. Matt Garrish
  30. Mike Gifford
  31. Loïc Martínez Normand
  32. Mike Pluke
  33. Justine Pascalides
  34. Chris Loiselle
  35. Tzviya Siegman
  36. Jan McSorley
  37. Sailesh Panchang
  38. Cristina Mussinelli
  39. Jonathan Avila
  40. John Rochford
  41. Sujasree Kurapati
  42. Jatin Vaishnav
  43. Sam Ogami
  44. Kevin White
  45. E.A. Draffan
  46. Paul Bohman
  47. JaEun Jemma Ku
  48. 骅 杨
  49. Victoria Clark
  50. Avneesh Singh
  51. Mitchell Evan
  52. biao liu
  53. Scott McCormack
  54. Denis Boudreau
  55. Francis Storr
  56. Rick Johnson
  57. David Swallow
  58. Aparna Pasi
  59. Gregorio Pellegrino
  60. Melanie Philipp
  61. Jake Abma
  62. Nicole Windmann
  63. Oliver Keim
  64. Ruoxi Ran
  65. Wendy Reid
  66. Scott O'Hara
  67. Muhammad Saleem
  68. Amani Ali
  69. Trevor Bostic
  70. Jamie Herrera
  71. Shinya Takami
  72. Karen Herr
  73. Kathy Eng
  74. Cybele Sack
  75. Audrey Maniez
  76. Jennifer Delisi
  77. Arthur Soroken
  78. Daniel Bjorge
  79. Kai Recke
  80. David Fazio
  81. Daniel Montalvo
  82. Mario Chacón-Rivas
  83. Michael Gilbert
  84. Caryn Pagel
  85. Achraf Othman
  86. Fernanda Bonnin
  87. Jared Batterman
  88. Raja Kushalnagar
  89. Jan Williams
  90. Isabel Holdsworth
  91. Julia Chen
  92. Marcos Franco Murillo
  93. Yutaka Suzuki
  94. Azlan Cuttilan
  95. Joe Humbert
  96. Ben Tillyer
  97. Charu Pandhi
  98. Poornima Badhan Subramanian
  99. Alain Vagner
  100. Roberto Scano
  101. Rain Breaw Michaels
  102. Kun Zhang
  103. Jaunita George
  104. Regina Sanchez
  105. Shawn Thompson
  106. Thomas Brunet
  107. Kenny Dunsin
  108. Jen Goulden
  109. Mike Beganyi
  110. Ronny Hendriks
  111. Breixo Pastoriza Barcia
  112. Olivia Hogan-Stark
  113. Rashmi Katakwar
  114. Julie Rawe
  115. Duff Johnson
  116. Laura Miller
  117. Will Creedle
  118. Shikha Nikhil Dwivedi
  119. Marie Csanady
  120. Meenakshi Das
  121. Perrin Anto
  122. Stephanie Louraine
  123. Rachele DiTullio
  124. Jan Jaap de Groot
  125. Rebecca Monteleone
  126. Ian Kersey
  127. Peter Bossley
  128. Anastasia Lanz
  129. Michael Keane
  130. Chiara De Martin
  131. Giacomo Petri
  132. Andrew Barakat
  133. Devanshu Chandra
  134. Helen Zhou
  135. Bryan Trogdon
  136. Mary Ann (MJ) Jawili
  137. 禹佳 陶
  138. 锦澄 王
  139. Stephen James
  140. Jay Mullen
  141. Thorsten Katzmann
  142. Tony Holland
  143. Kent Boucher
  144. Abbey Davis
  145. Phil Day
  146. Julia Kim
  147. Michelle Lana
  148. David Williams
  149. Mikayla Thompson
  150. Catherine Droege
  151. James Edwards
  152. Eric Hind
  153. Quintin Balsdon
  154. Mario Batušić
  155. David Cox
  156. Sazzad Mahamud
  157. Katy Brickley
  158. Kimberly Sarabia
  159. Corey Hinshaw
  160. Ashley Firth
  161. Daniel Harper-Wain
  162. Kiara Stewart
  163. DJ Chase
  164. Suji Sreerama
  165. Lori Oakley
  166. David Middleton
  167. Alyssa Priddy
  168. Young Choi
  169. Nichole Bui
  170. Julie Romanowski
  171. Eloisa Guerrero
  172. Daniel Henderson-Ede
  173. George Kuan
  174. YAPING LIN
  175. Justin Wilson
  176. Tiffany Burtin
  177. Shane Dittmar
  178. Nayan Padrai
  179. Niamh Kelly
  180. Matt Argomaniz Matthew Argomaniz
  181. Frankie Wolf
  182. Kimberly McGee
  183. Ahson Rana
  184. Carolina Crespo
  185. humor927 humor927
  186. Samantha McDaniel
  187. Matthäus Rojek
  188. Phong Tony Le
  189. Bram Janssens
  190. Graham Ritchie
  191. Aleksandar Cindrikj
  192. Jeroen Hulscher
  193. Alina Vayntrub
  194. Marco Sabidussi
  195. John Toles
  196. Jeanne Erickson Cooley
  197. Theo Hale
  198. Gert-Jan Vercauteren
  199. Karla Rubiano
  200. Aashutosh K
  201. Hidde de Vries
  202. Julian Kittelson-Aldred
  203. Roland Buss
  204. Aditya Surendranath
  205. Avon Kuo
  206. Elizabeth Patrick
  207. Nat Tarnoff
  208. Filippo Zorzi
  209. Mike Pedersen
  210. Rachael Yomtoob

Send an email to all the non-responders.

Compact view of the results / list of email addresses of the responders

WBS home / Questionnaires / WG questionnaires / Answer this questionnaire

Report issues on GitHub project w3c/wbs-design or by mail to sysreq.