https://github.com/w3c/wcag-act/pull/255
we started by briefly talking about the PR review tool; it's setup, but doesn't run on old PR
<maryjom> https://github.com/w3c/wcag-act/issues/236
<maryjom> https://github.com/w3c/wcag-act/issues/239
maryjom: these issues are related
... what are we doing with accuracy benchmarking,etc. definitions of false negatives, false positives
... I think we were wanting to replace the section with something more generic
... Wilco was thinking it could be a note external to the document
... I was wondering if it's possible to do something like that in an appendix?
romain: I think if it's in the spec document it needs to follow the rec track process even for informative sections, so it needs to be in a separate note document
anne_thyme: the idea was also to make that part of the review process
maryjom: yes, it makes sense, as it relates to how we approve rules, etc. we don't have to ahve yet another document
... we can try to keep our resources to a minimal number
... we would have a generic section in the format spec
... and then draft something and figure out where to put it in the process document
[scribe lost the audio for a while]
cpandhi: ???
... the definition of accuracy isn't aligned with the general meaning
maryjom: we had defined some methods of proving accuracy and when we last discussed this people mentioned they didn't use that method
... so the idea of putting that in the spec, as a normative section, wasn't a good idea
... we just need generic statements, and point to the process document that Shadi wrote
... we need to pass to process this
... 1. go through the section about accuracy and benchmarking and make it more generic
... 2. update the process document so that it can be referenced from the spec
... I will try to write something to address this
... we need to resolve all the issues we have currently open and get the spec out for its next release in a few weeks
... Wilco was hoping to resolve at least all the editorial stuff so that we can move on when he gets back
[summarizing the topic with Shadi]
shadi: I can help you with the editing, let's schedule a call later today
<maryjom> https://github.com/w3c/wcag-act/issues/229
maryjom: do we want to put together a note describing how to use EARL with the ACT Format?
romain: I think we may need to first clarify some issues with the output format, as raised in #162
... the basic idea is that the outcome of the rule may include outcomes from each individual test targets, and this is not well specified in the "Outcome" section of our spec
maryjom: does the PR about test aggregation help?
anne_thyme: I don't think so, it's a disconnected topic
... Casper is also implementing the output format in our system, and we're also missing some clarity about compositions, so I agree there's a bit of work
maryjom: I try to identify parts of the spec that need to be worked on
romain: the "outcome" section
shadi: 1st editorial nit is that the statement "MUST always result in one of the following outcomes" is about implementations, not about the rules
... the idea is that implementations must have an output format that has test target, test subject, and outcome
... but there's no real requirements how to do so
... for instance, does it need to be in JSON? can implementations chose whatever notation or use a specific serialization?
... we have a notation as an example, but for now it's not required
anne_thyme: what if a manual tester writes a report in plain text, will that be an acceptable format for the spec?
shadi: my understanding is yes
anne_thyme: if we don't have any requirement for the notation of the output, then we could mention it?
shadi: right now we actually say nothing
... we could also require some kind of structured format
... the next level is to require a specific syntax (CSV, JSON, etc)
... the next level is to define the schema
... we can chose how detailed we want to be
anne_thyme: if we want to see uptake of ACT Rules in manual testing, requiring them to output reports in JSON may not be the way to go
shadi: yeah, however most manual evaluators want to have some kind of structured output
... maybe this is just what we need
... and have a soft suggestion for a specific syntax/schema
what are people here assuming about the requirements?
<shadi> https://www.w3.org/WAI/eval/report-tool/
Jey: Wilco has mentioned to me in the past that we have the requirement to produce EARL+JSON-LD for reuse in some other tool
shadi: the link above is the reporter tool, and we'd love to have EARL/JSON-LD import there
... but do we want to require manual evaluators to require EARL or is it sufficient to have something else like a CSV, etc
... or we can say "if you're using something else, then you have to tell how it translates to EARL"
Jey: the reporter looks like a good tool for manual evaluators
shadi: yes, but we need to be open, this tool is just a proof of concept
cpandhi: just thinking aloud: I don't know if we want to restrict people to a certain format
... we've used HTML, CSV, and recently started using JSON
... three are converters, maybe we can just mandate a structured format and just recommend EARL but not require it
<cpandhi> +1
shadi: anybody has an objection to require a "structured format"?
+0.1
romain: what's a "structured format"?
<shadi> https://en.wikipedia.org/wiki/Formal_grammar
maryjom: "structured data format"?
romain: I think what we're trying to say is something that is "machine readable"
<shadi> "formalized grammar intended for automated processing"
romain: do we want that as an informative note? it's hard to test and specify?
shadi: yes, which brings us to the first point that we're specifying a format, not requirements on implementations
... we're tareting several audience: people writing tools, people writing rules, people evaluating rules
... regardless if we're writing it as a note or not, this issue is about rule implementers. we need an editorial pass
... let's say we have a definition for some kind of "structured format", anybody objects to recommending the use of JSON?
cpandhi: we can say "structured format such as JSON, CSV, …"
<Jey> +1 for JSON
maryjom: right, a list with those two at the top of the list
shadi: anybody wants to advocate for or favor any particular format?
... anybody objects to recommending EARL notation?
romain: we said that EARL needed a cleanup pass at some point?
... can we recommend it before doing this cleanup work?
... defining "structure format" and then the abstract structure will be difficult
... can we take the other way around: define one particular format (say in JSON) and require that implementers produce something the can be *converted* into that format
... does that make sense?
shadi: if you look at example 14 you have the 4 fields from the spec, but also other fields coming from EARL
... or either we need to refer to EARL or define these other things
... I like the idea to keep it simple
... the minimal requirement is to have a structured format, and we can name a few, and mention EARL
<cpandhi> +1
do people agree with that?
<maryjom> +1
romain: I agree with the principle, but I think our definition of what to put in this format, how it is structured isn't precise enough
maryjom: who drafts the changes?
shadi: usually this is the editors, but we have a lot of comments and a lot of things
... let's talk about that, if we want to add more people to the editor's team, etc
maryjom: yes, it w/b great to do that, as we have a lot of open issues
MoeKraft: some of the issues are purely editorial, a couple of them are calling for examples
... the straight up editorial I can work on, but I could ask for some help with examples
shadi: somebody who's writing rules might be a good candidate?
<maryjom> https://github.com/w3c/wcag-act/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22
shadi: my review of the doc is that I think we have all the pieces, but some things not clear or not explicit
... that's the kind of work that still needs to be done