Wilco: We have a face-to-face meeting in Copenhagen in September
Wilco: We've had these surveys out for two weeks now, quite a bit of feedback. What do we do from here?
Anne: So the question is what we do with the feedback from the specific rules?
Wilco: Yeah, what do we do with this and what should the process in general look like?
Anne: This seems like issues that should have been brought up in GitHub.
Wilco: Who should be providing this feedback though? Individual reviewers opening issues in the GitHub repo or should the chairs facilitate?
Anne: What about duplicates? If multiple reviewers find the same issues, there might be duplicates.
Wilco: So maybe collect the issues and have one person communicate it.
... This seems like something the chairs should be doing. Does that also suggest that we should be including suggestions? We can even create pull requests based on the feedback here.
Anne: If we're the ones creating the pull requests, we also own them. Ownership will be in the task force, not the community group.
Wilco: Also, how do we avoid having too many round trips between reviewers and the community group?
Anne: The community group does have a final call phase. The task force could use this grace period for feedback.
Wilco: Could this be timeboxed?
Anne: What part?
Wilco: In the W3C, there's usually a review period. I can imagine that we try to limit this to one or two round trips. Could we do it such that a rule is submitted, we'll review, and if you resolve the feedback, we'll be okay with publishing the rule.
Anne: What if other changes are also made?
Wilco: That's what I'm trying to avoid.
Anne: We cannot limit the community group to making changes that they think are good.
Wilco: If they make other changes, that deserves another review. At least for the new part.
... Does my previous suggestion seem acceptable?
Kasper: From experience, substantial changes are quite often made after a rule has gone into wider review.
Wilco: How else do we solve this then?
Anne: The final call phase is already hard to get out of, so are we potentially adding another difficult review phase?
... It's also important making sure that reviews are properly resolved by involing the person who made the review.
Alistair: If we do tie everything together with the community group by involving the person making the review, how would that work in practice? Do we have to invite everyone all the time?
*scribe has lost track of the discussion*
Alistair: The very end goal is to say yes or no to whether the W3C should publish a rule and that should be 99,9% decided already. Someone else will need to work out whether or not the rule is good.
Wilco: How do we figure out whether we should say "no"?
... If we have to review a rule 4 or 5 times, if we review it once a week that's already 5 weeks of work.
... So maybe this happens in a separate pull request where we can have a discussion on it.
Alistair: This could become a day job, which is the last thing anybody wants.
Wilco: Ideally, by the time the work gets to the task force, it's already done.
... Going by how much feedback we had in this initial round makes me wonder how this is going to work.
Anne: Many of the things we found were general quality issues that I think have been known in the community group for some time. We need to create incentive to work on and fix these quality issues.
Wilco: We really will expect whoever is submitting a rule to have these things sorted out.
Alistair: I would suggest we have some reviewers guidance. That will iron out a lot of the cases.
Wilco: There's additional requirements we should be looking for.
Alistair: We could even automate some of it.
Wilco: So we will just hand back rules if they're not good enough and the chairs will handle the communication. Should we then just hand back these rules that have been submitted?
... Anything else process wise worth discussing?
<Wilco> https://www.w3.org/2002/09/wbs/93339/ButtonName/
Alistair: Setting up some sort of survey for people who are not on the task force might also be an idea.
Wilco: The ACT-R has a requirement that there be at least 3 implementations of a rule before it's sent to the task force.
... I think that's very useful to do, but we haven't decided if the task force thinks that that's useful. Do we want implementations to go with rules?
Mary Jo: Documented in the rule? I think that it's important that there's at least 1 implementation to prove that the rule can exist.
Alistair: It's difficult to set a number, but at the least we're probably looking for 2 organisations.
Wilco: So, what are the reasons to look for implementations?
Alistair: First, to show that a rule can actually be done. What you don't want to do though is hold up innovation. If you require multiple implementations, maybe not everyone can implement it.
Wilco: It definitely sounds like we would want implementations. Does anyone think otherwise?
... So at least 1.
Anne: When we decided to skip Shadis proposal on process, I think we talked about the community group acting as the gatekeeper on implementations.
... Do we want to have additional requirements on top of what the community groups have?
Alistair: Let's maybe say you have one amazing test and only one implementation. If you push it through, you end up with W3C having a bunch of tests that only one tools can implement.
Wilco: I'd also argue that that is not harmonized.
Alistair: That can limit innovation.
... If it's a W3C standard, I do agree that everyone should be able to have a crack at it.
Wilco: So it could be on a case-by-case basis, but at least 1 implementation is needed.
Alistair: The more implementations, the higher confidence.
*scribe lost audio*
*audio is back*
Wilco: So I will take these rules back to the community group.