W3C

Results of Questionnaire Preparation for Joint ACT - Silver Meeting of 14 May 2021

The results of this questionnaire are available to anybody.

This questionnaire was open from 2021-05-06 to 2021-05-14.

14 answers have been received.

Jump to results for question:

  1. Your preparation for the joint meeting
  2. Outcome & its definitions
  3. Critical Errors & Rating
  4. Relevant Headings Method
  5. Other comments

1. Your preparation for the joint meeting

Please let us know how extensively you have reviewed these documents in preparation for the meeting:

Summary

ChoiceAll responders
Results
I have closely reviewed the documents 10
I have skimmed the documents 3
I have not reviewed the documents 1

Details

Responder Your preparation for the joint meetingComments 1. Your preparation
Sheri Byrne-Haber I have closely reviewed the documents I helped write these guidelines
David MacDonald I have closely reviewed the documents Some comments:

A) The requirement that headings need to be visually distinct requires some testable statements. Headings are generally made visually distinct in several ways.
(1) bold (2) Large font (3) spacing (4) positioning above the content (5) occasionally the length of the string of text in the heading can be an indicator (6) Occasionally the use of Capitalization indicates visually that it is a heading

How many of these would need to be present for it to pass. How are we sure something is a heading? Is it because it is introducing content below it? I think we would need to say what we mean by a heading.


Wilco Fiers I have closely reviewed the documents
Aron Janecki I have closely reviewed the documents
Jennifer Chadwick I have not reviewed the documents
Janina Sajka I have skimmed the documents Skimmed is probably understating things, but I'm not confident
enough to say "thoroughly reviewed" either.
Anne Thyme Nørregaard I have closely reviewed the documents I have looked closely at the documents linked from this questionnaire, but it has been a a few months since I last read through all of the WCAG 3.0 documents, and I have never worked actively with those documents, so my review of the documents in this questionnaire is done a bit out of context.
Trevor Bostic I have skimmed the documents
Jeanne F Spellman I have closely reviewed the documents
Danielli Franquim I have closely reviewed the documents
Melanie Philipp I have closely reviewed the documents
Bruce Bailey I have skimmed the documents
Todd Libby I have closely reviewed the documents
Andrew Kirkpatrick I have closely reviewed the documents

2. Outcome & its definitions

Carefully review Outcome: Headings organize content . Focus on the following areas:

  • Is it easy to understand and teach?
  • Is the requirement clear, or does it need further definitions, if so, where?
  • Are there errors, either in leaving gaps, or requiring more then is intended?

Summary

ChoiceAll responders
Results
I am in full support of this, and believe I could effectively test it without further additions. 3
I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors. 8
I am unsure if this is on the right track, and have a the following suggestions that would need to be addressed so that I could test using this. 1
I believe the outcome needs to be rethought for it to be testable, and suggest the following approach to doing so: 2

Details

Responder Outcome & its definitionsComments on 2. Outcome
Sheri Byrne-Haber I am in full support of this, and believe I could effectively test it without further additions.
David MacDonald I believe the outcome needs to be rethought for it to be testable, and suggest the following approach to doing so: The critical error says "One or more headings necessary to locate the content needed to complete a process are not visually distinct."

Does that mean that there is plain text there that says something like "Mailing address" but is not visually distinct? Where the evaluator would simply have to *identify* headings that are visually present but not visually distinct.

I'm assuming it doesn't mean the evaluator needs to examine the process and decide whether there SHOULD be a heading when there is nothing? Is that a right assumption?

This failure seems to trigger a critical failure, which gives the author 0% (0 points) for the SC in the process. Seems like a high cost and yet the language seems a bit ambiguous for such a consequence.

Wilco Fiers I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors. Lots of questions come to mind, that I believe will need to be answered:
- What exactly is a "logical block"? How does someone decide if content is organised in logical blocks? Is there some maximum size a block needs to be? Can a "logical block" contain other blocks? I think "logical block" needs a definition, and it needs one that can distinguish between something like a page with lots of little topics, and a chapter in a book which can be dozens of pages long without needing a heading.

- What is the relationship between "logical blocks" and "subsequent content". Is the expectation here that each "logical block" starts with a heading, followed by its "subsequent content"? I would guess so. I think this can be clarified.

- Why are headings the only allowed way to organising content? For example using section elements, modals, clear use of borders and whitespace, etc. I think this outcome needs to allow for other ways to organize. It should also ensure that the visual organisation is consistent with the semantics of the content.

- The phrase "This makes locating and navigating information easier and faster", does it have a purpose beyond explaining the goal of the outcome. Should it be taken as a normative requirement that all headings must make navigating information both easier, and faster? If so, how would someone test if a heading makes navigating easier and faster? If this isn't a requirement, it doesn't seem like it should be part of the normative text. I suggest to put these phrases in an informative section of WCAG 3.
Aron Janecki I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors. It would be a bit more clear to me if I only had the rating scale. Certain phrases need to be well defined, like "logical blocks" or "expected headings".
Jennifer Chadwick I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors. Also - "Functional Categories" - I wasn't sure if this was the same as what is commonly understood in Europe, i.e. EN 301 549 - "Functional Needs". If they are the same, can the same term be used?
Janina Sajka I am unsure if this is on the right track, and have a the following suggestions that would need to be addressed so that I could test using this. Headings are be very important; but other semantics may serve as well
or even better in some situations, e.g. is push advertising in the midst of a ecommerce process
equivalent to the 3 or 4 results of the search? What about pushing related products?
Anne Thyme Nørregaard I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors. From reading the outcome, I have no idea if it is supposed to address visual headings, semantic headings or both. And what if there are differences between the visual and the semantic headings? Which ones are then considered here? And is it a failure that there are differences?
In an ideal world, I would like it to be unambiguously defined in the normative parts of the standard what the requirement applies to, so that there can be no doubt about what the right interpretation of the success criteria is. Especially when it is used for rating, since the rating will depend on what content is considered applicable for the requirement, and in the light of this statement from WCAG 3.0, 5.4 Technology specific testing: "The outcome is written so that testers can test the accessibility of new and emerging technologies that do not have related methods based solely on the outcome."

I am also unsure if a visual headings hierarchy that does not make sense (e.g. <h2>Dog food</h2><h2>Cat food</h2><h3>Bird food</h3><h1>Hamster food</h1>) would be a failure for either the "logical blocks" or "relevant headings" part.
Trevor Bostic I am in full support of this, and believe I could effectively test it without further additions.
Jeanne F Spellman I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors. Specific rules as part of the method would be helpful for testing. I think at the Outcome level (technology agnostic) this is the correct amount of information, especially for beginners.
Danielli Franquim I am in full support of this, and believe I could effectively test it without further additions.
Melanie Philipp I believe the outcome needs to be rethought for it to be testable, and suggest the following approach to doing so: See "5. Other comments".
Bruce Bailey I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors.
Todd Libby I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors.
Andrew Kirkpatrick I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors.

3. Critical Errors & Rating

Carefully review the Critical Error

strong> and Rating under Outcome: Headings organize content . Focus on the following areas:

  • Are the critical errors testable, considering the topics under question 2?
  • Is the rating system clear?
  • Will the ratings be reproducible (inter-rater reliability)?
  • Will the critical errors be reproducible (inter-rater reliability)?

Summary

ChoiceAll responders
Results
I am in full support of this, and believe I could effectively test it without further additions. 2
I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors. 5
I am unsure if this is on the right track, and have a the following suggestions that would need to be addressed so that I could test using this. 1
I believe this needs to be rethought for it to be testable, and suggest the following approach to doing so: 6

Details

Responder Critical Errors & RatingComments on 3. Critical Errors & Rating
Sheri Byrne-Haber I am in full support of this, and believe I could effectively test it without further additions.
David MacDonald I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors. RATING SCALE - these comments are both for this Guideline and rating scales in general:

It seems that the rating scale requires the tester to:
(1) count up all text that "should" be presented visually as headings but are not visually distinct
(2) Count up all the instances of visually distinct headings that are not marked up as headings
(3) Count all instances of properly marked up headings that are not visually distinct
(4) count up all passing instances of text that "should" be presented visually as headings and are properly shown as visually distinct
(5) count up all passing instances of visual headings that "should" be coded as headings and are properly shown as visually distinct
(6) calculate the percentage of passes

Scanning a page for errors is very quick but stopping for every heading that passes is going to take much longer. It should be noted that counting all the passing instances of a guideline is time used by an evaluator (and expense by a website owner) that does translate into improved accessibility. This increases the cost of evaluation. So if the company doesn't increase the budget, they may end up with less accessibility for the equivalent budget under WCAG 3 than 2.x.





Wilco Fiers I believe this needs to be rethought for it to be testable, and suggest the following approach to doing so: I'm not sure headings are ever necessary to locate content. They make it easier to find content, but necessary seems like too strong a word. I'm also not clear yet on "process". It sounds like anything can be a process. If that's the case, then it follows that every missing heading is a critical error, because every piece of content can be part of some process. Or perhaps, if you aren't testing processes, then maybe nothing is ever critical here?

This phrase "necessary to complete a process" can be understood to refer to just the last step in the process. That doesn't seem correct, so perhaps just "necessary for the process"?

I'm not entirely sure a missing heading can ever be a critical issue. It seems to that content missing lots of headings could be a significant problem, but the scoring kind of has that covered. I would suggest not including critical errors for this outcome.

Testing this outcome feels like it's going to be highly time consuming. Having to read each page and understand it well enough for a tester to work out what heading are necessary is going to take a lot of effort. Then there's the issue that testers often aren't domain experts of what they are testing, so many testers wouldn't be a good judge of what would even need a heading. I'm not sure what solution to suggest on this. I'm not convinced this is something that can be tested reliably.

The percentage scoring here feel a little flawed when it comes to lower numbers. A page that has 1 missing heading can only ever reach a score of 3 or 4 if there are 8 correct headings on the page. I suspect most pages don't have 9 headings on average, so if a tester thinks even one heading is missing, and they very well might, the page most likely gets a 2. I think this incentives adding useless headings to the page, just to be safe with the numbers.
Aron Janecki I believe this needs to be rethought for it to be testable, and suggest the following approach to doing so: I'm not sure what "Expected headings" means and how to test for it. This is a very subjective definition.
Jennifer Chadwick I believe this needs to be rethought for it to be testable, and suggest the following approach to doing so: Though non-normative, it feels like incomplete guidance for a critical error without referencing the way it needs to be coded: HTML5 (H1, H2, H3) or ARIA role="heading" aria-level="1".
Janina Sajka I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors. Worried that we need better definitions of "process" per the above comment
on whether all content in a view can be presumed related.
Anne Thyme Nørregaard I believe this needs to be rethought for it to be testable, and suggest the following approach to doing so: I actually don't know what would be a critical error here.
A long form that is not using headings to divide it into more digestible sections, is that a critical error?
What about a check out flow, where one of the screens in the process don't have any headings, but where the purpose of the page is otherwise clear and there is a clear call to action, would that be a critical error?
What if I open up a public sector self service solution, where on the first screen I am presented with a short "info + disclaimer" screen, and a big green "Start" button, but only one level 1 heading at the top of the page, is that a critical error?

For the Rating, I am curious how to rate pages that require only 1 or 2 headings.
And also to what extent textual analysis is required. I do have a BA in language, but not all accessibility testers do.
And how any kind of inter-rater reliability is expected in this. At least it must be very clear what the accessibility requirements are, so that it will not come down to a matter of taste, e.g. "If I had written this news article, I would have squeezed in a few more headings", or "this section could potentially have been split up in two with each their heading".
Trevor Bostic I am unsure if this is on the right track, and have a the following suggestions that would need to be addressed so that I could test using this. For the critical error, how would you test to determine if a heading is necessary to locate content? Would possible alternatives like regions be allowed? What other exceptions might there be?
For the rating, what is the definition of expected heading? Could it be an improperly implemented heading or would a missing heading (or what the tester believes is a missing heading) also be considered?
I have some concerns for inter-rater reliability unless some of the definitions become more well-defined.
Jeanne F Spellman I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors. This needs more specific guidance the bands that would define the quality of what is needed to test the process. It possibly could be defined at the outcome level, but to be consistent with other Outcomes, this guidance should be at the Method level.
Danielli Franquim I am in full support of this, and believe I could effectively test it without further additions.
Melanie Philipp I believe this needs to be rethought for it to be testable, and suggest the following approach to doing so: Critical errors: Do we have research that shows that the lack of heading text, lack of visual distinction of heading text, lack of heading markup, or lack of perfect hierarchy of heading levels would ever actually critically block people from completing a task? If that research exists, that could help inform if/when an assessor would flag a critical error regarding headings.

Inter-rater reliability: This outcome and its supporting docs feel like a collection of best practices for content authors to consider when creating their content - not testable, ratable statements that an assessor can realistically apply.

Rating: I am concerned that the use of percentages to score these Outcomes is flawed will not reflect a "Valid" result (see Wilco's comments). The Silver docs describe Validity as "This attribute is related to the extent to which the measurements obtained by a metric reflect the accessibility of the website to which it is applied. This is probably the most important metric to study in any conformance proposal. Does the score that a test web site or digital product achieve in any proposed scoring actually reflect the score that the digital product should get?"
Bruce Bailey I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors.
Todd Libby I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors.
Andrew Kirkpatrick I believe this needs to be rethought for it to be testable, and suggest the following approach to doing so: Missing one or more headings is a high penalty, I would like to know what criteria we have for something to rise to "critical issue". The subjectivity is that people will evaluate what headings are needed differently. In addition, the percentages in the ratings make it so that very few pages will be able to miss even 1 heading and get a rating of 4.

4. Relevant Headings Method

Carefully review Method: Relevant Headings . Focus on the following areas:

  • Is the relationship to the outcome clear?
  • Will the method be reproducible (inter-rater reliability)?
  • Can the method be tested efficiently, or will testing effort significantly increase?
  • Does the method constrain what tools & test methodologies can use it?

Summary

ChoiceAll responders
Results
I am in full support of this, and believe I could effectively test it without further additions. 3
I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors. 5
I am unsure if this is on the right track, and have a the following suggestions that would need to be addressed so that I could test using this. 2
I believe this needs to be rethought for it to be testable, and suggest the following approach to doing so: 3

Details

Responder Relevant Headings MethodComments on 4. Method
Sheri Byrne-Haber I am in full support of this, and believe I could effectively test it without further additions.
David MacDonald The summary says:

> Headings are used as navigation features by any user who has a disability that impacts their ability to read. If the heading is not relevant to the subsequent content, it cannot successfully provide navigation support.

This sounds like a new kind of failure introduced in the (non normative?) informative section which doesn't appear to be mentioned in the guideline. The failure here is a that a properly marked up heading that is also visually distinct, has a text string in the heading doesn't correctly identify the subsequent content.

Is this a morphing of 2.4.6 into WCAG 3? If so, then I think the normative text would have to reflect that requirement.

Wilco Fiers I am unsure if this is on the right track, and have a the following suggestions that would need to be addressed so that I could test using this. It's not clear what is and what isn't a heading. Is that determined based on its content and place in a page, is it based on its semantics, its role in the accessibility tree, it's style, all of the above? I suggest we create a definition for heading, that defines it based on its function, and have other outcomes ensure the function is correctly communicated using styles and semantics.

This outcome and method is essentially assuming that someone will intuit what a section is. But when we worked on this in ACT we found lots of examples where this wouldn't be so clear cut. The ACT-R CG has had a heck of a time trying to figure out how to define section, and we didn't succeed at it. I'm not sure this outcome / method works without a solid definition of section.

What is the relationship between "Measurement", "Test for sections with headings", and "Test for relevance of headings"? How do you get a percentage using those two tests? This should probably just be written out explicitly.

Then there are lots of nit-picky questions I have about this too:
- In Summary; "Headings are used as navigation features by any user who has a disability", is that true?
- In Summary; "People who are trying to locate controls in an application rely on headings." I doubt that, it's usually more about putting the controls in a logical place in the UI.
- In Description; the outcome has a different name, seems outdated.
- In Unit Tested: What is the relationship between a "segment or group of content" and "logical group"?
- In Procedure 1; How is content divided into sections? Is it by heading, because if so, then step 2 in the procedure is always true by definition.
- In Procedure 2; What does it mean for a heading to "relate" to content?
Aron Janecki I am unsure if this is on the right track, and have a the following suggestions that would need to be addressed so that I could test using this. How do you define a logical block? Different people may section content in a different way. If a section does not have a heading for one person it does not mean the other person will consider that as an issue.
Jennifer Chadwick I believe this needs to be rethought for it to be testable, and suggest the following approach to doing so: At first glance I'm confused by what is a 'relevant' heading versus any other heading.
Janina Sajka I am in full support of this, and believe I could effectively test it without further additions. Does appear to flow from provided definitions ...
Anne Thyme Nørregaard I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors. The Detailed description seems to contain information that is not quite covered in the Tests tab.

In the Tests tab, it is a bit unclear from the page itself what the relationship between the different tests are. Is it an AND or an OR relationship between them? And how does this relate to the Rating?
And what is the relationship between the "Measurement" and the tests?

The text in Measurement is not coherent: "Measurement: Create a percentage of the expected headings within a view that have appropriate text alternatives percentage of the expected headings that are present is provided."

For the "Test for sections with headings":
- Would this mean that <header>, <main>, <footer> also need headings?
- How do we ensure no one interprets this as "there cannot be a paragraph that does not have its own heading?"
- Are we talking visual headings or semantic headings or both? And how do you rate it, if something has only of the two for some of the content?
-
Trevor Bostic I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors. In the test section, I find the phrasing of the measurement to be difficult to understand. The examples given in the method seem more relevant to the outcome "Conveys hierarchy with semantic structure" than "Headings organize content".
Jeanne F Spellman I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors. I think we need a section on each Method with specific ACT rules or links to the ACT test - but it's more likely to be used if it is directly in the Method (IMO).
Danielli Franquim I am in full support of this, and believe I could effectively test it without further additions.
Melanie Philipp I believe this needs to be rethought for it to be testable, and suggest the following approach to doing so: In summary it says "People who are trying to locate controls in an application rely on headings." This assertion seems like a stretch.

Regarding "Unit Tested: Segments or groups of content" - “segments or groups of content” needs to better defined. From assessor to assessor, the choice of unit(s) can vary greatly AND can be used to influence (game) the scoring.

Regarding "Measurement: Create a percentage of the expected headings within a view that have appropriate text alternatives percentage of the expected headings that are present is provided." This sentence doesn't make sense.

Regarding "Test for sections with headings" - What is a “section”? Is a paragraph a section? How much content makes a “section”? How does one decide when a section needs a heading? Every paragraph? Every XXX words?

This test:
- Requires subject matter expertise and language skills such that the assessor can read and understand all the content and judge if the breakdown of “sections” is “good enough” and then that headings are used in the “right” places.
- Is extremely subjective. I am concerned that it cannot possibly be reliable from assessor to assessor.
- Requires a large investment in time to do the above.
Bruce Bailey I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors.
Todd Libby I believe this is on the right track, but to improve testability would propose the following suggestion at the discretion of the editors.
Andrew Kirkpatrick I believe this needs to be rethought for it to be testable, and suggest the following approach to doing so: Being able to clearly articulate whether the test is speaking to visual sections or code sections is needed.

5. Other comments

Do you have any other comments about the Headings organize content outcome, and its related documents?

Details

Responder Other comments
Sheri Byrne-Haber There is some question bout whether we want to get more explicit about best practices. For example, a "don't skip levels when you are getting deeper" check, or a "only 1 H1" check
David MacDonald I think we need a fair amount of work on this Guideline
Wilco Fiers I would be very interested to know how holistic tests may fit in with this method. Since the atomic test generates the percentage for the outcome, it doesn't seem like there is room for any other method of testing for this outcome.
Aron Janecki
Jennifer Chadwick
Janina Sajka
Anne Thyme Nørregaard I think the relationships between all of the different "views" of a requirement has to be very explicitly communicated right were the content is for the standard to be easy to use. As it is right now there are several different truths that seem competing, or at least presenting different perspectives on what is required, and I would find it difficult to "get everyone on the same page" when working with external testers, as well as teaching people how to use the standard.

In general from at testing perspective, I am missing the "applicability" perspective throughout the documentation, and also more objectivity in the pass/fail criteria.
Trevor Bostic
Jeanne F Spellman It would be interesting if the rules can be adapted to apply at the Outcome level, or if they are sufficiently technology specific that they would have to be positioned at the Method level.
Danielli Franquim
Melanie Philipp I did a deep dive into the two other headings Outcome as well and have similar doubts about the testability, inter-rater reliability, validity, and time to complete.

I realize I haven't answered the request to "suggest the following approach to do so". However, these Outcomes feel like a collection of best practices for content authors to consider when creating their content - not testable, ratable statements that an assessor can realistically apply - so other than a complete rethinking of what heading guidance can meet the goals of Reliability, Validity, etc. I, unfortunately, don't have specific suggestions.
Bruce Bailey
Todd Libby
Andrew Kirkpatrick

More details on responses

  • Sheri Byrne-Haber: last responded on 12, May 2021 at 19:01 (UTC)
  • David MacDonald: last responded on 13, May 2021 at 01:25 (UTC)
  • Wilco Fiers: last responded on 13, May 2021 at 12:39 (UTC)
  • Aron Janecki: last responded on 13, May 2021 at 15:42 (UTC)
  • Jennifer Chadwick: last responded on 13, May 2021 at 23:47 (UTC)
  • Janina Sajka: last responded on 14, May 2021 at 00:12 (UTC)
  • Anne Thyme Nørregaard: last responded on 14, May 2021 at 10:47 (UTC)
  • Trevor Bostic: last responded on 14, May 2021 at 12:40 (UTC)
  • Jeanne F Spellman: last responded on 14, May 2021 at 12:51 (UTC)
  • Danielli Franquim: last responded on 14, May 2021 at 12:56 (UTC)
  • Melanie Philipp: last responded on 14, May 2021 at 13:59 (UTC)
  • Bruce Bailey: last responded on 14, May 2021 at 14:08 (UTC)
  • Todd Libby: last responded on 14, May 2021 at 14:39 (UTC)
  • Andrew Kirkpatrick: last responded on 14, May 2021 at 14:54 (UTC)

Non-responders

The following persons have not answered the questionnaire:

  1. Gregg Vanderheiden
  2. Janina Sajka
  3. Shadi Abou-Zahra
  4. Makoto Ueki
  5. Peter Korn
  6. Alastair Campbell
  7. Léonie Watson
  8. David Sloan
  9. Mary Jo Mueller
  10. John Kirkwood
  11. Detlev Fischer
  12. Matt Garrish
  13. Chris Loiselle
  14. John Rochford
  15. Sarah Horton
  16. JaEun Jemma Ku
  17. Denis Boudreau
  18. Rachael Bradley Montgomery
  19. Francis Storr
  20. Aparna Pasi
  21. Ruoxi Ran
  22. Charles Adams
  23. Arthur Soroken
  24. David Fazio
  25. Daniel Montalvo
  26. Caryn Pagel
  27. Julia Chen
  28. Rain Breaw Michaels
  29. Jaunita George
  30. Suji Sreerama

Send an email to all the non-responders.


Compact view of the results / list of email addresses of the responders

WBS home / Questionnaires / WG questionnaires / Answer this questionnaire