What is Wide Review and How do we achieve it -- 29 Oct 2014

<timeless> scribe: timeless

<Ralph> +RalphS

SteveZ: this session is because there were a lot of concerns about what wide-review means in the Process 2014 document
... so I wanted to describe the rationale between Process 2005 and Process 2014
... and b) suggest things that could be done to achieve Wide Review
... and c) answer questions / issues
... no slides for this
... in decisions that went into Process 2014 (revision of Chapter 7)
... a key thing was to enable groups to begin work earlier in their process than implied in Process 2005
... over the years, we've discovered that groups begin getting reviews/testcases/implementations earlier in their process
... are more effective at getting out of the process
... one of the stages was Last Call
... which unfortunately had two things attached
... when it was created it was "The WG thinks it has completed its work, Object now or forever hold your peace"
... when the Patent Policy was created, it needed an anchor point
... they says "since, LC is when the group finished its work", "we'll attach the Patent process to that"
... but, it turns out the group wasn't finished there
... and secondly, WGs weren't serious about LC = Done
... we'd get a series of LCs
... there's an objection that the series of LCs
... you could do LC->CR->LC->CR
... because patent consideration was tied to LC
... we've transferred the Patent Commitment to CR
... it's less likely to cycle
... and if you cycle, you cycle in CR
... this had a bad side-effect
... it removed an opportunity to say
... "speak now or forever hold your peace"
... before i go down that route
... the other thing we did in terms of wide-review
... reviews at LC was way too late in many circumstances
... another thing, was to drop LC as the review step
... saying "review when things are stable"
... review incrementally as things go along
... review continuously
... changing from "met requirement of making an announcement"
... to "met requirement if people have reviewed it"

[ 7.2.3.1 Wide Review ]

SteveZ: so instead of a check for publishing a request for review
... you have to show reviews
... one thing was Disposition of Comments (DoC)
... if all comments came from Implementers, it probably wasn't Wide Review
... there's an emphasis to reach out early
... some groups have an IG (Interest Group) list
... and send things out there
... show by getting comments from outside, that they've met the requirement
... so, change from checkbox, to actual substantive response

nigel: going through this, the W3C has a Liaison list
... in addition to the Charter dependencies, I also went to the Liaison list
... perhaps it would be beneficial to update that list to include Liaisons in the process

SteveZ: you're doing Timed Text
... clearly an area where groups outside W3C have a documented interest
... other things where it would be less useful
... because the technology is more internal

nigel: but then there wouldn't be anything in the liaisons page, so it wouldn't occur

[ nigel will create an action in #w3process to open an issue to add liaisons to 7.2.3.1 ]

barry: Pete and I are IETF application areas

<Ralph> [Ralph departs]

<nigel> trackbot: Created ACTION-37 - Open an issue to add liaisons to what's considered in the wide review paragraph 7.2.3.1 [on Steve Zilles - due 2014-11-05].

<trackbot> Sorry, nigel, I don't understand 'trackbot: Created ACTION-37 - Open an issue to add liaisons to what's considered in the wide review paragraph 7.2.3.1 [on Steve Zilles - due 2014-11-05].'. Please refer to <http://www.w3.org/2005/06/tracker/irc> for help.

barry: i was attracted by this
... we too tried to solicit it
... people are often trying to do a semi-final review
... but resist multiple reviews

SteveZ: the process we did before was largely based on IETF process at the time this was written

barry: we thought as well

SteveZ: what are we trying to do?
... one is dependencies
... adding liaisons is relevant there
... we're trying to advise people to, if you have a dependency, go talk to those people and work out a schedule for review
... some are dependent to horizontal WGs
... who are way overloaded w/ work
... so you want to give them time when it's stable enough to review
... but not before it's concrete -- that it can't be changed
... LC isn't that, because that's you think it's complete, what did we miss
... this is a goal, i don't how it will work out
... it's an extra burden on the Chairs

barry: on the Chairs, and the reviewers

SteveZ: presumably the reviewers want to review, because they don't want their ox gored

barry: we're trying on a case-by-case basis, you as a chair can ask for a review from a particular group for review
... it doesn't scale
... it's only seldom requested, there's a hope
... we're only getting 15% success
... it's hard to get people to review early/intermediate versions
... we're stumped

SteveZ: i'm not offering a panacea
... mentioning a discussion, the other group may come back and say "here's what's critical to us"
... so you know what might trigger them, to be more successful in getting reviews
... In our WG, we'll get a Review request, and the answer is no
... unless you can get something likely to be in the document that's likely to be a problem, no one wants to look at it
... hopefully conversations may lead to better results, than just throwing it over the wall

barry: agreed

SteveZ: WG members may be active and doing this 24 hours a day
... other reviewers aren't
... you're lucky to get one review, you need to make it count
... when you clearly have an identified dependency, you can reach out
... the other thing suggested
... fantasai had an action item to propose
... something to help w/ the review process
... we've observed that the development of a standard doesn't progress at a uniform rate
... some pieces (often core) progress faster than others
... it would be useful to mark sections of the document w/ stability
... CSS developed a system called Shepherd
... that scans through our documents, we have markup, and ToC
... and our test-database is driven off those IDs
... we use IDs, section numbers may change
... IDs don't change as much
... we can automatically generate a version of the document indicating which tests have passed in which browsers
... it wouldn't take too much effort to do this, if we could indicate stability
... to automatically generate section warnings
... and if changes are tied to ids, we could include links to change fields
... for people reviewing it more than once, they can see which sections have changed recently
... since they last saw it
... and therefore see what they need to re-review
... trying to make the documents reviewer friendly is another piece
... we don't have that technology in place
... there's no funding
... it's a couple of CSS WG members working on it

<nigel> +1 to making it easier for people to answer the question "how stable is this [section of this] document?"

SteveZ: we have it for testing level
... we can in principle extend it

barry: are all W3C drafts developed with the same markup

timeless: there are 2 that are used

barry: that helps
... we have many different

SteveZ: i understand you have an effort to try to standardize

barry: yes

SteveZ: there's a wiki, has most of the information that I said
... except why the process changed
... in the previous session, nigel pointed out it'd be useful to say what the goals of wide review are

barry: in IETF, groups will write a document, and not know who their dependencies are
... they'll stick something in, and not realize that they've created a dependency
... getting it wrong, they'll break things
... getting them to know that when they say URI/UTF,
... that they need to consult

SteveZ: we have more process in forming a Group
... we say a Charter needs to list dependencies
... Horizontal WGs need to review charters
... doesn't really address your problem
... because Charters are approved by AC members as a whole
... there's a Vote
... there's a better chance that the AC member looks at a Charter to see if a reviewer thinks there's a relation

barry: we have a similar process
... but someone says "this string is a character encoded in UTF-8"
... all of a sudden it has comparison
... but it wasn't in the Charter

pete: often in a routing protocol

barry: we have this problem, you have this problem, we're both trying to solve it

SteveZ: no, we're both doing experiments to try to improve this

nigel: you talked about typical duration for review

[ Working Groups should announce to other W3C Working Groups as well as the general public, ]

[ especially those affected by this specification, a proposal to enter Candidate Recommendation (for example in approximately four weeks). ]

nigel: when you talk to industry bodies
... they might not meet very often
... and their secretarial skills aren't great
... they might sit down in six months
... it probably conflicts with a good practice
... to state in the document when the end of review period is

SteveZ: i think that review end is required

nigel: reviewers will see the document after the review ends
... and not know what to do

SteveZ: for a Liaison, you should have set this up

timeless: is there a stopword lbarry, ist that you could write a bot for?

barry: for some, yes

<SteveZ> consider: https://www.w3.org/wiki/DocumentReview

timeless: types of reviews

barry: if we flag "this mentions UTF8", that's easier to get someone to review
... since they only have to look for UTF8 in the document

pete: security has been good to have a directorate
... they farm out documents to people

barry: that works at LC
... the area has asked about early reviews
... and Directorate has said NO

pete: because they'd get double their work

SteveZ: i discussed that w/ Accessibility
... and they said it creates as many problems as it solves
... this document (DocumentReview)

barry: we could have written that
... we can get more than one shot for a particular document
... but if we try for lots of documents, it would collapse

SteveZ: if the reviewing group indicates the kinds of concerns the chair should be aware of
... he may go ask for advice earlier if he sees those red flags being raised
... i think opening dialogue will make this work better

nigel: back to time-frames
... how do you prevent that from happening
... the other question for the WG
... what is the status for comments after Review closes
... you got a bunch of comments that came during and you got out of LC
... then you get more comments

SteveZ: there's an official answer and an unofficial answer
... there's a risk of a DoS
... practically, if the problems are real, the WG should want to try to fix them
... if someone feels they're being unfairly treated, there's an appeals process

nigel: worst-case scenario
... comments come @ CR
... you decide to go to PR
... you have comments w/o DoC
... it'd be clear if you should have a DoC on all, or all

SteveZ: typically the DoC would be postponed to a future version

nigel: so there's a requirement on the WG to make a decision (postponed) on the comment

SteveZ: there's a default-decision "came after comment deadline"
... you could record that w/o the WG making a decision
... good practice would be that you should note the comment
... and the person noting should respond saying "i think the WG should postpone, do you agree?"
... we've had cases where a group was totally overloaded

barry: we have who are people who aren't totally abusive, but they keep drawing the discussion out beyond where it makes sense

SteveZ: these are all judgement calls, you don't want to make rules that don't make them judgement

mdjp: github comments
... we have lots of issues, they're commented on outside W3C
... that's continuous

SteveZ: if you have a way of quickly summarizing that, i think that would be perfectly adequate
... there's no requirement for an official wide review
... if you can provide evidence
... i believe you've established there's been wide review
... it's important to show people beyond implementers
... and the key groups are in that set

mdjp: you can identify logs where you have gaps
... and you can target those groups

barry: now one who's expert in that has reviewed it

SteveZ: automating is great
... we're not trying to make work
... we're trying to make sure the right eyes have seen it

barry: a lot of this is having Chairs do the right thing
... and training the chairs
... what do you do in the line of Chair Training?

SteveZ: not much/enough
... this is the only SDO i know of where an appointment doesn't involve Chair training

barry: there's Chair training they sometimes do the Sunday before the meeting
... and there's a Wednesday lunch for Chair training

SteveZ: w3c has recently established a number of Chair Training Phone Calls
... for topics relevant to Chairs
... Tools
... Human Element
... How to be Effective

<nigel> http://www.w3.org/2014/Talks/chairs-part4/#/

mdjp: As a new chair, these sessions would have been really useful before the F2F

<nigel> http://www.w3.org/2014/06/17-chairing-minutes.html

mdjp: without the crossover to the outgoing chair, it would have been really difficult to run the meetings

<nigel> http://www.w3.org/2014/04/24-chairing-minutes

<nigel> nigel: There are others too - I'm not sure if they're collated onto a single page for ease of reference

SteveZ: no rules, and talk to the people you need to talk to

nigel: you mentioned if all reviews are from implementers, that's not enough, it's not wide
... but evidence of implementation can be evidence of review
... since you can't implement without reading

SteveZ: the way our process is
... you weren't going to get anywhere unless there were implementations anyway
... but it isn't the reviewers we're concerned about
... it's dealing with the communities that could be disrupted or disenfranchized if this becomes a standard

<nigel> timeless: I've dealt with implementers who see problems and ignore them

<nigel> fantasai: +1

<nigel> timeless: They pick an answer silently, without notifying anyone. They do /something/ but nobody realises it's not what was intended in the spec until much later.

<nigel> ... Most implementers aren't noisy like that.

<inserted> scribe: nigel

fantasai: An implementer who says something would be counted for wide review evidence

timeless: there are implementers that will just jump over problems without reporting them so implementations are not total proof of review
... When browser devs implement something what actually matters is usage in websites.
... Website devs are even worse than browser devs!
... You have to go and look at website code to see if they've put nasty comments in about your spec!
... It's a nasty thing to go and hunt those comments down.
... We've begged for years for this kind of feedback, and we're lucky to get feedback from 5 sites, out of billions!
... A way to harvest those comments would be beneficial.
... A way to demonstrate searches on stackoverflow, etc would be good.
... Conversely if people don't comment on the likely sites e.g. stackoverflow, then that may be evidence that there isn't a problem

<SteveZ> timeless: reviewing the comments in code on key websites and/or on stack overflow will show evidence of wide review

<timeless> scribe: timeless

nigel: W3C generally wants to make
... everyone wants to include "the general public" in "wide review"
... the best way to make things visible
... is "the w3c home page"
... if you put things in /TR/, you can ask staff to put things on the w3 home page
... it's been proposed to have a Blog or something for announcing reviews

SteveZ: the DocumentReview wiki documents making reviews happen
... it's asking the W3C to make a mailing list
... there's an issue of what the name should be
... i think that will happen
... it's not last-call
... the intent is that you identify what you want reviewed in the Status section
... because it may not be the whole document, perhaps just a recent change

nigel: some organizations have an "Official Journal"
... it's widely known to the world
... a clear statement from W3 that this is the place
... your XXXs should be looking there
... boring, but effective

SteveZ: the issue of publicizing this list is known to Team

<Zakim> timeless, you wanted to talk about flagging content

<Zakim> joanie, you wanted to suggest Twitter and hashtag that stateholders are likely to see. (Not a joke)

joanie: i did this by ranting, and i got a lot of replies
... if your stakeholders are web developers, and spit out a single tweet with a link to the document and a few appropriate hashtags
... a couple of stakeholders will retweet

SteveZ: we should consider using all channels that are available

joanie: i've used Twitter to complain about web developers doing things
... and then developers reply explaining why they did it

SteveZ: two other things in consideration
... besides the ML
... one is a dashboard page listing which active reviews
... and secondly a Calendar
... listing reviews closing
... I think, if we could automate it
... different views of the same information
... you have an announcement, it shows up in the right places

fantasai: CSS WG
... W3 will announce it
... we'll announce it on our blog
... we'll send it to places
... we'll tweet a link
... we make our editors write the announcement, because we want them to explain the key points
... otherwise, W3 staff does boilerplate
... there's people out there, you want their feedback, try to get it

SteveZ: developing an Interest community is a valuable way to get comments from others than implementers
... not the public exactly, but people who are following
... for CSS, it's www-style
... but we can get answers in the 50s
... when we ask

fantasai: i think we'd benefit from a less technical channel
... lower traffic
... people have wanted to participate, but don't want to subscribe to the ML
... it's high traffic
... high signal to noise

SteveZ: we tag messages
... you can sort, i consciously sort
... i don't do animation

fantasai: we ask people to tag w/ the spec shortname

SteveZ: i wish to thank our scribe, immensely

barry: indeed

SteveZ: and i'd like to thank everyone for participating
... i'm trying to do the process, not as written, but how it is/should be in practice...
... a very different question

[ Adjourned ]

[ Lunch ]

- DRAFT -

What is Wide Review and How do we achieve it

29 Oct 2014

Attendees

Contents

Summary of Action Items

Scribe.perl diagnostic output