Accessibility Conformance Testing Teleconference

TPAC & Auto-WCAG F2F

wilco: Shadi & I were talking and wondering if we should set up a meeting for auto-wcag to work on rules
... thinking of doing this part of the WAI-Tools project
... wondering if we should combine this with TPAC
... it makes sense for us to do sth around TPAC, lot of work ahead of us (getting rules into the repo)
... TPAC is in October
... we should be in CR at that point
... we're going to put effort into proof of implementations
... but also focus on getting rules
... since auto-wcag is considering meeting, combining with TPAC is an opportunity
... a 2/3 days either before or after TPAC
... the question to the group is are you planning to join TPAC, and would you be interested in an auto-wcag meeting?

shadi: you're implying we're having a f2f meeting at TPAC, w/b good to make a decision on that first
... auto-wcag is not part of this group but directly related
... meeting would be about bashing out rules
... and the question is whether it w/b helpful to have that in close proximity
... or make that a completely separate event and date
... the ACT f2f is part of this group's work (not a strong requirement, but strongly suggested)
... the auto-wcag w/b optional and only for those involved in writing rules

rdeltour: I'll be at TPAC, but for the whole week so adding another meeting make it difficult to attend all

cpandhi: unfortunately I'm not going to TPAC

anne_thyme: we'll also be at TPAC, and we're also concerned about the length of combined meetings

shadi: the reason we thought about co-locating was to get more international participants
... question to Romain, do you think we should have any overlap with the Publishing WG?
... maybe a couple hours are enough?

rdeltour: yes, I think that would be a great idea

shadi: I was also thinking about auto-wcag meetings
... having rule writing for technical people writing rules in both groups

rdeltour: yes, it's a great idea

shadi: can you think of people? please share the idea with colleagues

rdeltour: not a lot of people are actively writing formal test rules in the Digital Publishing community, but yes it's definitely interesting
... I can think of various people doing a11y testing in digi pub that would be interested in such a meeting

shadi: to be clear, we're talking about 3 different meetings:
... 1. the ACT meeting
... 2. the Auto-WCAG meeting, in the context of TPAC
... 3. a third thing like a work meeting related to the Auto-WCAG group, but separate and organized through the WAI-Tools project

Define a format for rules and test cases #195

<Wilco> https://github.com/w3c/wcag-act/issues/195

wilco: in thinking about how we want to start collecting rules, I figured auto-wcag is a good model to look at
... auto-wcag has done a number of things, starting out using WikiText, then move on to MarkDown in github, then files transposed to HTML
... the idea is that we don't just allow Markdown, that might be too constrained
... the idea is that when someone is ready with some rule, they can provide it as a Markdown file or HTML file, and we would accept either
... it would require more work on the templates, but it's doable.
... it also means we're not stuck on MarkDown
... I was thinking of a similar approach for the other files
... implementation manifest (YAML or JSON), example pages, test cases

tbostic: can someone share a link of an implementation manifest?

wilco: no one can, we haven't produced one yet
... the idea is that if someone writes a rule and want to share it, they can provide a manifest that shows which rules are used, in which way
... so that organizations can tell us what they're using and we can keep track of which rules are used
... to know if we have enough implementations for rules to be public
... reminder: we need at least 2 implementations of a rule for it to be published in the repository
... manifests allow us to have a dynamic way of tracking that

<Wilco> https://w3c.github.io/wcag-act-rules/rule-tests/ACT-R1-tests.html

wilco: if you look at the ACT-R1 page
... it has examples, but also test cases
... the test cases are in the iframes
... the way I though of making this available is for the example page to be in MarkDown or HTML, and for test cases to be in HTML
... that was my idea, but don't know what other thinks.
... if you're writing rules, would you be able to provide these files in these formats?

rdeltour: I agree, +1

<MoeKraft> +1

cpandhi: yes, I think it pretty much covers what we need
... +1

<cpandhi> +1

wilco: if we go with these languages and formats, we're gonna be stuck with that
... do we need to make it a wider question with a survey?

shadi: yes I would say so

wilco: or does this need to be bubbled-up to AG?

shadi: I don't think so, this would be either defined in the spec or the review process
... so w/b defined somewhere and would then be reviewed by AG
... for this group we could do an email CfC or survey
... I do have one comment on the format itself
... regarding the test cases, I really suggest, along with HTML, some form of metadata describing the tests themselves
... it could be either an associate file or microdata within the HTML somehow
... my suggestion would be a separate file with basic metadata descriptions
... sometimes you have external additional resources (video, CSS)
... we should have a testcase description file to say how the rule is carried out on the test case

wilco: yes, it was definitely a thing I omitted from this list
... I'm not sure
... it doesn't seem we have a format for this. would we write this in EARL, in TCDL?

shadi: not EARL, which is only about the results
... it's more a counterpart of EARL

wilco: to be blunt, does anybody wants this?

shadi: yes, I've heard this request from people (e.g. LevelAccess)
... it's important to create a library
... if somebody just gives you a test case, how do you know how to perform the test?

rdeltour: isn't it what the rule says?

shadi: you might have parameters, to setup the test case, so that you know how to run the rule on the test case

wilco: I would probably look at the example page
... which describes the test, what to expect, etc

shadi: how do you maintain this page overtime?
... let's say you have 3 test cases, I provide you one more
... do you have to manually add that?
... maybe we can say that the headings are the metadata
... we need to say whether the test passes or fails, then how to setup the test case
... it's minimal metadata, but it needs to be defined
... we need keywords, so that a script can scrape that

wilco: if you can't get that from the example page, then it means the example page is incomplete

<cpandhi> Here is an example from the WAI-ARIA authoring practice - https://www.w3.org/TR/wai-aria-practices-1.1/examples/alert/index.html

shadi: I would prefere more automation, but can leave that to the people doing the testing

wilco: my concern is that I don't want to put more work
... I'm not necessarily inclined to put more work into that if people are not committed to do this or if we don't have people clearly interested

cpandhi: I just put a link to the authoring practice example
... was shadi talking about something like that? metadata about resources of the example?
... I think it's a good idea to have that kind of explanation of what the test case is doing, and the resources that go with the test files
... this is a little different from what we're doing, but there are similarities

shadi: yes. but I also agree with Wilco, let's take it step by step, see what we need

rdeltour: I can see the use of having some lightweight metadata to the test cases themselves, so that they can be used independently from the test example page

wilco: the problem is that I don't think there's something lightweight enough

rdeltour: I also agree to take it step by step, and see later if something obvious comes to mind

Add implementation manifest #194

<Wilco> https://github.com/w3c/wcag-act/issues/194

wilco: I have put some thoughts into what we may want from the implementation manifest
... [describes what's in the issue's opening comment]
... it's fairly lightweight, but I don't have an example ready
... any comments so far?

rdeltour: does it leave in the repository? how to ensure is kept up to date? what if it changes frequently?

wilco: the implementers themselves would be responsible

<Wilco> https://github.com/w3c/wcag-act/issues/194

anne_thyme: is it intentional there's nothing about support for test cases?

wilco: if you do not implement every test cases, you haven't actually implemented the rule
... my thinking was: what does it mean to have a rule automated?
... when you're running the rule, you see no human input and see a pass/fail result in every test case
... if for some test cases you don't see any firm results, then you have a semi-automated implementation
... if you have nothing, then it's manual
... I didn't specify any test case, since I don't think you can say you've implemented a rule if you have disagreement with test cases

anne_thyme: yes, I think it makes sense
... I just like the idea how knowing if we have implementations for individual test cases

cpandhi: when I first read the issue I was wondering whether the implementer was an organization or a piece of an implementation
... an implementor "contributes" a test file

wilco: an implementation is a tool or a methodology
... an implementor is whoever provides this tool or methodology

cpandhi: I think we should clarify the actions

wilco: right, I updated it

shadi: to the earlier question from Romain, yes there's a little maintenance to be done
... there will be broken links, etc

wilco: my thinking would be that this leaves inside the rules repository
... and can be updated with PR

shadi: yes, but it does require maintenance, possibly minimal

rdeltour: I'm not concerned about maintenance on our side (ACT) but more whether the implementor will actually commit itself to maintain this information

shadi: what has helped on the tools list is to have a "last updated" date
... moving on to the other question about results for each test cases
... I think it would be very interesting, for several reasons
... one reasons is that the more aggregation you have, the more likely you have to get false claims of support
... the other reason is that it's an opportunity for users to identify how the tools/implementations really work, and see which matches their needs

wilco: it's gonna make things more complicated
... let's say we want to track which test cases were passed, then we actually need to give them identifiers, make some sort of comparison
... I'm not sure we're doing ourselves a favor to get things increasingly complicated
... let's take that up next week
... thanks everyone, let's continue this discussion next week