Getting early and wide review of a document is very important yet in practice, it can be challenging. Rather than defining explicit mandatory steps for getting wide and constructive review, this document provides some considerations, guidelines and best practices that groups should follow.
- Tool to Assist with Calls for Specification Review
- Process Document 2017 (aka
Process-2017)- the 2017 Process applicable starting March 1st 2017.
- 1 Introduction
- 2 TL;DR:
- 3 Considerations and Best Practices
- 4 Request for Comments (RfC) Template
- 5 Horizontal Groups
- 6 FAQ
- 7 Issues
- 8 Terminology and Abbreviations
- 9 Enhancement Requests
Getting early review and feedback is important, especially reviews from those groups that have identified a document as a dependency, and the so-called horizontal groups. It is also important to get review from other stakeholders including Web developers, technology providers and implementers not active in the Working Group, external groups and Standards organisations working on related areas, etc.
In Process-2017, the focus is on actually getting evidence that wide review has occurred. The considerations, guidelines and Best Practices (BP) in this document should provide reasonable guidance for document review.
When you have published a First Public Working Draft, you should work through available "self-review" materials, and request review of your results, in at least the following areas. "Long enough*" before you request a transition to CR, you should do the same again, identifying substantive specification changes since the first review.
The meaning of "Long enough*" depends on how many changes there are, and how clearly you have explained them. Pointing to 14 concise points for a small spec means a short time, pointing to 900 diffs from commits and hoping people understand them in a 300 page spec means it will take a long time to get review. If you have effectively identified issues for review during development, and got feedback on them, the review time will probably be shortened. It also depends how busy the groups are, so planning in advance is useful.
- Work through this questionnaire then ask APA for review: firstname.lastname@example.org
- Work through the [Checklist], then request i18n review by email to email@example.com (A public list watched by the Working Group). If for some reason you want a member-confidential review, you can ask on firstname.lastname@example.org (a member-only list watched by the Working Group).
- Go through this checklist and the privacy/security checklist, then ask the Privacy Interest Group for a review: [mailto:email@example.com firstname.lastname@example.org
- Use your results from the privacy/security questionnaire (see privacy), then ask the Security IG for review: email@example.com
- Groups listed in your charter
- If there are groups listed in your charter as depending on some specification, you should ask them too.
You should familiarise yourself with the rest of this document. This section is just a quick reminder for when you are in the middle of doing the work.
Considerations and Best Practices
Among the considerations and best practices, regardless of the type of review (early, thorough wide review, etc.) are:
Who to ask for review?
- Which group(s) should be asked to review a document? All group charters should include information about the groups and external liaisons that are interested in particular documents. At a minimum, those groups should be included in all review request for their related document(s). Additionally, it may be appropriate to explicitly seek review from others such as: other Consortium groups; external organizations and SSOs; stakeholders such as implementers, testing and interop experts, application developers, etc. If a document is explicitly identified as a joint deliverable with another group, it is especially important to make sure that group is included in all review requests.
- Which Horizontal Groups should be asked to request a review? This can be a bit tricky because in some cases the set of relevant of horizontal groups (enumerated below) should be relatively obvious but in other cases it might not be. For example, Working Group Charters often explicitly identify Horizontal Group dependencies, but post-charter development may introduce features that require review by Horizontal Groups not called out in the charter. - SNZ
- BP: if in doubt, before formally asking a horizontal group to review a document, talk to them informally (f.ex. via e-mail).
(comment): I agree that it may not be obvious that a document needs horizontal review, but I tend to take a stronger line: even if you have not identified a horizontal group dependency, you should always discuss whether a review is necessary or even request a review (saying if you think there are no issues of interest to the given group). --Addison Phillips (talk)
- What should be done if it appears a document might be relevant to a W3C liaison but the liaison is not explicitly listed in the group's charter? The Chair(s) and Team Contacts should consult with their group, and if there is consensus within the group to contact the liaison, then that external group should be included in all review requests.
- BP: That group should be included in review requests unless they ask not to be --CMN
When should review be requested?
- When is a group required to seek wide review of a Process-2017 document? For a document to enter CR, the group must show the document has had wide review. The requesting group is free to determine the timing for review requests.
- When should reviews by Horizontal Groups be scheduled? Horizontal Groups are very often resource limited. This means that they can often only do one review and may have difficulty scheduling that review in a timely way. Secondly, to do a reasonable review, they need a document that is pretty stable, but still flexible enough to allow changes where issues are identified. -SNZ
- BP: Contact the chair of each relevant Horizontal Group to arrange a suitable time in their schedule to review the document. - SNZ
- What circumstances should trigger a RfC? When a new feature is added to a document after its FPWD is published or after the document has already been a target of a wide review, a group should seek wide review of that new feature.
- Other reasons to consider wide review include: some other group expresses keen interest in a document.
- What part of a document should be reviewed and when? The general rule of thumb is to get review early and often.
- BP: Ensure that each section or feature is reviewed when it is considered stable (this enables changes to be made early, which lowers the risk that deployed legacy will impact the possibilities for fixing something).
Making reviews happen
- Length of the Comment Period. The duration for comments (aka comment period) depends on a number of factors including the version of the Process Document being used. As such, consult the version of the Process being used: Process-2005 or Process-2015.
- BP: before a group asks another group to review a document, the requesting group should work with the review group to determine a mutually agreeable review timeline.
- How to initiate a review request. Review requests should be sent to the review groups' Public mail list(s).
- BP: please avoid cross-posting by using Bcc: if necessary.
- BP: use the RfC template below.
- Announce LCWD and pre-CR publications on the chairs list. The group Chair or W3C staff should announce all LCWD and pre-CR publications on the chairs list.
- Announce all Requests for Review (f.ex. LCWD and pre-CR publications) on the
public-review-announce e-mail list. This announcement may be made by a group Chair, Consortium Staff, Editor or any other designated member of a group.
- BP: use the RfC template below.
- Use Public e-mail lists for review feedback. The requesting group's Public mail list should be used for review feedback.
- BP: use the requesting group's main Public mail list for comments and do not use (nor create) a list whose sole purpose is for comments.
- How can a group get acknowledgement that some person and/or group actually reviewed a document? Naturally, this isn't an issue if the reviewing group and/or individuals in the reviewing group actually provides review comments. However, in some cases, this can be somewhat difficult to determine and achieve (f.ex the reviewing group is small, the overlap in scope is minimal, etc.).
- BP: when asking for review, explicitly ask the reviewing group to submit all feedback, even if the feedback is something like "I read it and have no comments".
Request for Comments (RfC) Template
A RfC should include the following general information:
- Name of the group seeking comments.
- Deadline for comments.
Subject:header: include the maturity level, name of the document and document maturity level of the document
Subject: RfC: FPWD of Name of Spec; deadline MMM DD.
- Mail list for comments; include the mailto: address plus a link to the Public mail list archive
- BP: avoid cross-posting to multiple lists (and if this must be done, use Bcc:), except for the i18n WG, which needs to track issues on its technical list as well. (i18n WG members cannot subscribe to all the lists of the groups they review specs for.)
- Alternate instructions for providing commentary (if needed). Some groups use bugzilla or github as a means of communication and for tracking issues. --Addison Phillips (talk) If your main or preferred tracking location is a database, make that clear in the SOD, since moving and tracking comments between mail threads and databases can waste time for both the reviewer and the reviewed group.
Reply-to:header: set the
Reply-to:header to the mail list where all comments should be sent
- State that all review be acknowledged, even if a reviewer or group has reviewed the document and has "no comments".
Information for Each Document
For each document that will be reviewed, provide at least:
- Document title
- BP: the contents of the document should be stable (i.e. not changed) during the review period.
- BP: if the contents of the document might change during the review, identify the specific section(s) that might change; or consider postponing the review until after those sections are stable.
- Maturity level: FPWD, LCWD (for Process-2005 documents), pre-CR (for Process-2015 documents), CR, ...
- Scope of the review: this can be the entire document, specific sections, specific issues, etc.
- BP: Identify the specific sections of the document that are newly stabilised, as well as those already considered stable. This allows reviewers who follw "at a distance" to focus on what's new, and allows new reviewers to focus on things that are ready for review, instead of providing a lot of comments on sections that are likely to shift.
- BP: Specifications should include information about stability and known implementation on a "section by section" basis.
- If a review was previously requested, include the URL of a diff, using the previous version of the document as the diff's reference/base.
- BP: use W3C HTML Diff service
- BP: also provide a human-authored summary of the changes.
Example Request for Comments
Here are some example RfCs:
The consortium's so-called Horizontal Groups are groups that include review and coordination functions in their charter. Typically, these groups consist of individuals that can provide advice and guidance (to other groups) regarding their domain expertise.
Groups with a horizontal review function, including their mail list are (alphabetical order):
- Accessible Platform Architectures Working Group; public-apa; APA Spec Review tracking
- Internationalization Working Group; www-international Reviews by Internationalization generally also involve the Interest Group, but are arranged through the WG.
- Privacy Interest Group; public-privacy
- Technical Architecture Group (TAG); www-tag
- Web API Consistency and Web IDL Conformance; public-script-coord; (this is a Web Platform WG list that liases with ECMA's TC-39)
- Web Accessibility Interest Group; w3c-wai-ig
- Web and Mobile Interest Group; public-web-mobile
- Web Security Interest Group; public-web-security
- I18n Request a review
- Internationalization Best Practices for Spec Developers; Richard Ishida ; this document is also available as an expanding collapsing checklist
- Self-Review Questionnaire: Security and Privacy; Mike West
- Guidelines for Writing RFC Text on Security Considerations (RFC3552)
- Mitigating Browser Fingerprinting in Web Specifications
- Specification Privacy Assessment (SPA) (Creating Privacy Considerations for W3C Technical Specifications); Frank Dawson
- Technical Architecture:
- Web Accessibility:
- Q: When is the best time to seek early review?
- A: For a number factors (including document maturity, implementation status, etc.), there is no best time that applies to all documents. However, for most documents, after a First Public Working Draft is published is generally a good time to seek review of a specification.
- Q: Is some type of security and/or privacy review mandatory before a First Public Working Draft is published?
- A: Strictly speaking, the answer is No. However, getting early review for documents with features that might have security and/or privacy implications is strongly encouraged. See the Horizontal Groups section below for information about security and privacy groups that can provide review and related review resources.
- Q: Does a group using Process-2017 need to prove its documents have had wide review before proceeding to CR?
- A: The answer is yes. -SNZ
- Q: How does a group using Process-2017 prove its documents have had wide review before proceeding to CR?
- A: See How to work with (experiment with) Wide Review -SNZ
- A: Keep a Disposition of Comments (DoC) Document that shows review comments and acknowledgements that have been received. It is the distribution of the reviewers that shows the review has been wide.-SNZ
- A: Ask the W3C Team if your proposed approach is likely to meet the requirements. -SNZ
- Q: Is there such a thing as too many reviews?
- A: Not really
- Q: Is it possible to make too many requests for review?
- A: unfortunately, yes there is. Multiple requests for reviews can quickly result in a situation of diminishing returns. To help address this, subsequent review requests should clearly identify the exact differences between the last review and current review.
- A: This is also the reason that the 2017 Process clearly suggests there should be (TR) Working Drafts published when there are "significant changes that would benefit from review beyond the Working Group", rather than every day or only twice in the life of a spec…
- A: TR Working Drafts are also useful for reviews since they provide a dated snapshot which can be recovered when the review comments are being discussed. Trying to discuss review comments against a document which has changed out of all recognition can be a frustrating and inefficient experience.
- Is pre-CR the best term to describe a Process-2014 snapshot that precedes CR or is there a better term?
- Proposals: 2014-LC, 2014-Snapshot, pre-CR snapshot, 2014-Review Doc, pre-CR Review Doc, ...
- Proposal: Don't do this in order to demonstrate "wide review". A "Last call" for a document that has already been reviewed in development is appropriate. Waiting until you think the document is finished to ask for review is a bad idea.
How can a group or individual be automatically notified when a document has been published at one of the transitions that could/should initiate the start of an early or wide review? For Process-2005 documents, the transitions of interest are FPWD, LCWD and CR and for Proc-Doc-2015 documents the transitions of interest are: FPWD, pre-CR, and CR.This Issue was resolved via the creation of the
- For all processes Working Drafts should be relevant for review - and should identify which parts are new or newly stabilized or changed, to facilitate timely review.
- Potential solution: when a document is published on /TR, announce its publication on a Public mail list. See a related Call for Consensus to create such a list, deadline for comments is October 15].
Terminology and Abbreviations
- pre-CR - This is a version of a Working Draft that is created to get wide review.
- note that this is a bad way to get review. In general, features should be reviewed as they are developed. Waiting for a "Last Call" for most review means that when reviews suggest changes it is far harder to make them, due to a commonly observed and logical reluctance to break deployed systems or content. Charles McCathie Nevile (talk) 11:18, 13 October 2014 (UTC)
- requesting group - a group that is requesting a review
- BP = Best Practices
- CR = Candidate Recommendation
- RfC = Request for Comments (aka Review Request)
- TR = Technical Report, i.e. a formal W3C publication.
- WD = Working Draft
- See the Document Review Dashboard document for information about creating a dashboard type service to facilitate document reviews.