Getting early and wide review of a document is very important, yet in practice it can be challenging. This document provides some best practices for getting document review; it does not define explicit mandatory steps.
This page is linked to from The Guide.
Feedback on this document is welcome, preferably by directly editing this document.
- 1 Introduction
- 2 How to get horizontal review
- 3 Considerations and Best Practices
- 4 Request for Comments (RfC) Template
- 5 Working with Horizontal Review Labels
- 6 Horizontal Review Boards
- 7 FAQ
- 8 Terminology and Abbreviations
- 9 Enhancement Requests
- 10 Resources
Getting early review and feedback is important, especially reviews from those groups that have identified a document as a dependency and the horizontal groups. It is also important to get review from other stakeholders including Web developers, technology providers and implementers not active in the Working Group, external groups and standards organizations working on related areas, etc.
In Process-2019, the focus is on getting evidence that wide review has occurred.
How to get horizontal review
When you have published a First Public Working Draft, you should work through available "self-review" materials and request review of your results in at least the following areas. "Long enough*" before you request a transition to CR, you should do the same again, identifying substantive specification changes since the first review.
The meaning of "Long enough*" depends on how many changes there are, how clearly you have explained them, and how much discussion is needed to resolve issues. Pointing to 14 concise points for a small spec means a short time if they are simple fixes, pointing to 900 diffs from commits and hoping people understand them in a 300 page spec means it will take a long time to get review, and potentially a long time to also discuss and agree on how to solve the issues. If you have effectively identified issues for review during development and received feedback on them, the review time will probably be shorter. Horizontal review groups sometimes get bogged down; planning in advance is useful.
- Read the Request a review page, then work through the Short Checklist, then request a review via GitHub.
- Write a "Privacy Considerations" section for your document, taking into account the Self-Review Questionnaire: Security and Privacy and Mitigating Browser Fingerprinting in Web Specifications, then ask the Privacy Interest Group for a review: email@example.com
- Write a "Security Considerations" section for your document, taking into account the Self-Review Questionnaire: Security and Privacy, then request a review on: firstname.lastname@example.org
- Groups listed in your charter
- If there are groups listed in your charter as depending on some specification, you should ask them too.
- General announcement
- Use email@example.com for general announcements regarding wide and horizontal reviews
You should familiarize yourself with the rest of this document. This section is just a quick reminder for when you are in the middle of doing the work.
Considerations and Best Practices
Among the considerations and best practices, regardless of the type of review (early, thorough wide review, etc.) are:
Who to ask for review?
- Groups listed in the WG's charter
- Groups jointly responsible for a particular document (if any) (duh!)
- All horizontal groups listed above. If it appears that one of those is not relevant (and is not listed in your charter), talk to them informally to confirm that.
- Other groups in your discretion, even if not in the WG charter: other W3C groups, external organizations and SSOs, implementers, application developers, etc.
When should review be requested?
- For most documents, after a First Public Working Draft is published
- Process-2019 requires wide review before a document is published at CR (Candidate Recommendation)
- When a document is both reasonably stable and still flexible enough to allow changes where issues are identified
- When new features are added after a document has already gotten wide review (perhaps requesting a review limited to the new features)
Recognize that horizontal review groups may be resource limited and may only be able to do one review or may have difficulty scheduling your review quickly. Give them as much time as you can, consistent with asking for review while it is still reasonable to change the technology to accommodate the issues they find.
Request for Comments (RfC) Template
The Internationalization WG has a series of issue templates that guide you through the information you need to provide when requesting a review. The information that follows is for other HR groups that don't use that approach.
What to include
- Name of the group seeking comments
- Preferred timeline for comments
- Preferred forum for comments (e.g. a github link or the group's mailing list)
- Suggested: Set the
Reply-to:header to the group's mailing list
- Document title(s)
- Document URL(s)
- Maturity Level
- Scope of the requested review: this can be the entire document, specific sections, specific issues, etc.
- If a review was previously requested, include a human-authored summary of the changes as well as a link to a diff
- W3C's HTML Diff service might be useful
Example Request for Comments
Here are some example RfCs:
Working with Horizontal Review Labels
Applying these labels doesn't replace the need to schedule a review of your spec at an appropriate time. (See How to Get Horizontal Review above.) Horizontal groups participants can find detailed process information here.
Day-to-day use of labels
Apply thelabel in your own repository to draw a horizontal review group’s attention to an issue in one of your own repositories. Horizontal review groups may also apply the label if they are interested in tracking a particular issue. Tooling will automatically notify the horizontal group that you attached the label.
If you want some specific advice from the horizontal group, describe that request in the issue thread.
Horizontal review groups may apply thelabel to issues they expect to be resolved before the specification moves to a new maturity level. Working Groups must not remove or add this label (not even when you close your issue).
If the horizontal group believes that an issue with a *-tracker label needs to be resolved before a transition, they may apply a *-needs-resolution label to the issue. Automatic tooling will later remove the *-tracker label.
At the end of a review, and before attempting to transition the spec, you should clarify that a resolution is described for all of the issues with a *-needs-resolution label, and check that the horizontal group is aware of those resolutions. You don't have to do this for issues with the *-tracker label. (The horizontal group was only tracking those issues, not expecting a particular resolution.)
What happens to unresolved issues marked *-needs-resolution?
As lead technical architect, the W3C Director is tasked (among many things) to assess consensus within W3C for architectural issues and to decide on the outcome of Formal Objections. When a horizontal issue gets flagged as *-needs-resolution and a Group chooses to request a new Maturity level despite the lack of consensus with the horizontal group, it is the task of the Director to assess the issue and the outcome of the request. A horizontal group MAY choose to elevate an horizontal issue as a Formal Objection to elevate further the importance of an issue per the W3C Process.
In the case where an horizontal issue hasn’t been addressed and the document was allowed to move forward, it is recommended that the issue remains open in the horizontal group repository (it MAY get closed in the specification repository unless the Director requests otherwise). Some issues may take years to get resolved, but that doesn’t mean those should be forgotten.
Horizontal Review Boards
The horizontal groups maintain repositories containing issues that track those raised in the WG repos. Key information about the current state of those tracker repositories is reflected in a set of tracker boards, one per horizontal function. The board points to WG issues and the corresponding tracker issues, and groups issues by specification.
Horizontal groups participants can find detailed process information here.
- Q: Is security and/or privacy review mandatory before a First Public Working Draft is published?
- A: Strictly speaking, no. However, getting early review for documents with features that might have security and/or privacy implications is strongly encouraged - security and privacy issues can require significant changes in specs, and it helps to identify them early.
- Q: Does a group need to prove its documents have had wide review before proceeding to CR?
- A: The answer is yes, since Process-2019.
- Q: How does a group prove its documents have had wide review before proceeding to CR?
- A: See How to work with (experiment with) Wide Review -SNZ
- A: Keep a Disposition of Comments (DoC) Document that shows review comments and acknowledgements that have been received. It is the distribution of the reviewers that shows the review has been wide.-SNZ
- A: Ask the W3C Team if your proposed approach is likely to meet the requirements. -SNZ
- Q: Is there such a thing as too many reviews?
- A: Not really
- Q: Is it possible to make too many requests for review?
- A: unfortunately, yes there is. Multiple requests for reviews can quickly result in a situation of diminishing returns. To help address this, subsequent review requests should clearly identify the exact differences between the last review and current review.
- A: This is also the reason that the Process clearly suggests there should be (TR) Working Drafts published when there are "significant changes that would benefit from review beyond the Working Group", rather than every day or only twice in the life of a spec…
- A: TR Working Drafts are also useful for reviews since they provide a dated snapshot which can be recovered when the review comments are being discussed. Trying to discuss review comments against a document which has changed out of all recognition can be a frustrating and inefficient experience.
Terminology and Abbreviations
- pre-CR - This is a version of a Working Draft that is created to get wide review.
- note that this is a bad way to get review. In general, features should be reviewed as they are developed. Waiting for a "Last Call" for most review means that when reviews suggest changes it is far harder to make them, due to a commonly observed and logical reluctance to break deployed systems or content. Charles McCathie Nevile (talk) 11:18, 13 October 2014 (UTC)
- requesting group - a group that is requesting a review
- BP = Best Practices
- CR = Candidate Recommendation
- RfC = Request for Comments (aka Review Request)
- TR = Technical Report, i.e. a formal W3C publication.
- WD = Working Draft
- See the Document Review Dashboard document for information about creating a dashboard type service to facilitate document reviews.