This report includes:
A future version of this report should complete some items that could still user more attention: an outline of a whole-spec template for SpecGL; a closer look at some individual-CP templates for SpecGL; and, consideration of templates for a handful of remaining OpsGL checkpoints.
The rest of this report uses these terms, acronyms, and abbreviations::
Composite abbreviations such as OpsGL/ET are also used.
The usability of any one of the GL documents is a result of several factors in combination. For example, the following bits in combination determine the usability of SpecGL:
The same usability factors apply OpsGL also.
An exhaustive analysis of the Candidate Recommendation (CR) text of SpecGL/ET was done. This uncovered lots of editorial and a few substantive issues with SpecGL. The latter pertain to the exact definition of the requirements of some checkpoints, hence their implementability. This gives a clear picture of where the SpecGL/SpecET combination needs to be enhanced with templates, and/or Examples ("E"), and/or techniques ("T").
Less of an analysis of OpsGL/ET was done. "Low fruit" has already been picked with OpsET and its two templates, the Charter template [CHART-TEMPL] and the QA Process Document template [QAPD-TEMPL]. However the results of SpecGL analysis suggest that there will be value in performing an OpsGL analysis for clarity, testability, and implementability issues; and, editorial problems -- this is recommended for future work or for the W3C editors.
What is a template? The existing OpsGL/ET Charter template and QAPD template fit a traditional concept of templates -- language, structure, and organization is laid out, with indications of where customization by the user is needed. In some cases, analysis suggested that a simple model or prototype sentence in the techniques parts of SpecET or OpsET would be sufficient. Does a simple model sentence qualify as a template? In another case, a "Quick Tips" (QT) supplement to OpsET -- discussing and instructing how to do the checkpoint, with sections of model language -- seemed to be an ideal solution. Is a QT a template?
Since the goal is to improve the usability of SpecGL and OpsGL, we decided to take a broad view of what qualifies as a template:
The SpecGL/ET analysis took a look at 1, 2, and 3 (above). The results are condensed into a SpecGL analysis table [SpecGL-data].
#1 resulted in a collection of editorial and substantive comments (issues) about SpecGL, which may also be found in the analysis results.
#2 and #3 are necessarily somewhat subjective. I read the checkpoint in SpecGL, and then followed the link to SpecET. I tried to imagine how an average user (not a QAWG expert) would understand the SpecET content and what he/she would gain from it.
These data are contained in the per-checkpoint table of analysis results [SpecGL-data].
The table also has a "Where" datum for each checkpoint. With two exceptions where SpecGL requires a particular spec location, I considered and chose a "good" location (or set of locations) for a CP's fulfillment. These are not unique or normatively exclusive of other possibilities, but ought to result in a clear and nicely arranged spec. This is the purpose of the templates ajdunct to SpecGL/ET -- give one good, easy way to do it, rather than explore all possibilities.
The "where" data turned out to be more useful than expected -- collectively, they suggested what I consider to be the most useful of the templates: an ideal-CC (Conformance Clause) skeleton, and an all-spec skeleton.
For "Templates potential" item, I tried, per-CP, to assess the degree to which the checkpoint's implementability would be enhanced by having a template (presumably linked from SpecET). Note that the potential refers only to whether the particular checkpoint (or small collection of closely related CPs) has potential to benefit from a template focused on that CP. The assessment ignores what might be the highest priority template -- a super- or composite template that outlines a complete specification, reflecting (almost) all of SpecGL's CPs, and probably embedding a number of specific and focused templates. In other words, a AAA-compliant skeleton specification.
We defer consideration of templates for these guidelines and checkpoints, because we think that either their precise meaning and intent is still too unclear, or there are other substantive issues that need to be resolved before designing supporting templates:
See the SpecGL analysis [SpecGL-data] for details about the problems with these.
Our initial focus in the analysis was on the potential of each checkpoint, in isolation, or a small number of closely related checkpoints. Analysis revealed relatively limited possibilities here.
The analysis of where in a specification the checkpoints might ideally be realized suggested that one or more multi-checkpoint templates, similar to [CHART-TEMPL] and [QAPD-TEMPL], might be most useful to SpecGL implementors (already done for OpsGL).
During the analysis of SpecGL/ET, a few cases suggested that a template for a single guideline and its checkpoints might be useful. This is an interesting possibility that ought to be looked at more closely in the future.
Of the options, a Conformance Clause (CC) template seems to be the single first choice for most value and utility to SpecGL implementors. The CC template will have a placeholder, sub-section, choice, slot, or model language, in a single document chapter (comprising the Conformance Clause), for each of the CP's that the SpecGL analysis table [SpecGL-data] contains "CC" in the Where column:
There may also be possibilities for the two checkpoints of GL10, but this needs further thought.
The CC template will result in a skeleton Conformance Clause chapter for the specification that gives at least some support for 31 of the total 46 checkpoints of SpecGL.
For many of the checkpoints, the support might be relatively limited, e.g., "do it here for CPx.y". That is, it may do no more than instruct the user, at a certain location in the skeleton CC chapter, to do what is described in the CP's Conformance Requirements. This alone -- a well-organized CC chapter outline that addresses all of these checkpoints -- is still of great value. There may be further opportunities for enhancement by tieing it together with some individual-CP templates (see below).
Currently, we give second priority to a full-spec template. This would be similar to the CC template, but would include template-section bits for each of a number of other checkpoints that are, for example, required to be in the specifications "Introduction" chapter.
The main features of this template would be:
It is a topic of further study, whether it could be productive to try to provide templates (of at least the "do it here" variety) for some of the CP's for which the SpecGL analysis table [SpecGL-data] indicates as "body" in the Where column.
These might be used standalone (linked from the techniques section of SpecET), or they might be linked from a CC template (see above) or a full-spec template (see above), or both.
Some CP are identified in SpecGL analysis table [SpecGL-data] as having substantive problems. The issues are briefly identified in the "Checkpoint" column, below the statement of the checkpoint. Queue these as SpecGL/ET issues and resolve them.
In the ET/E and ET/T columns of the SpecGL analysis table [SpecGL-data], some of the CP have status entries "unr". This means that the example or technique is unrelated to the apparent intended meaning of the CP. Replace these with examples and/or techniques that properly reflect the CP's meaning. Note. Such occurrences may indicate problems with the clarity of the CP's statement, conformance requirements, etc.
In the ET/E and ET/T columns of the SpecGL analysis table [SpecGL-data], some of the CP have status entries "0" or "minus". This means that there are either no examples/techniques, or that some are present but more are needed. These should be filled out more.
It is beyond the scope of this report to define, CP-by-CP, exactly what is needed to bring these cases to "ok" or "plus" status. This will have to be considered on a case-by-case basis.
Note that the implementation of the two recommended templates (below) will result in SpecET links from about 35 of the CPs. There will be opportunities, at the placeholder locations of many of the CPs in the template outline, to develop and include additional material.
Above we identified a Conformance Clause template as the top-priority template. This should be implemented. A CC template outline is provided, which can (should) be used as a starting point.
Above we discussed a full-spec template as a second priority template effort. This should be implemented, after the CC template and some of its individual-CP suggestions. In a future version of this report, a full-spec template outline may be provided.
Given the value that the SpecGL analysis provided, for identification of GL/ET editorial and substantive issues, I recommend that the OpsGL editors perform such an analysis.
However, for the purposes of this project, a different approach was expedient. This is because significant template support has already been built for OpsGL/ET: Charter template for "QA Framework: Operational Guidelines" [CHART-TEMPL], and QA Process Document template for "QA Framework: Operational Guidelines" [QAPD-TEMPL]. These two templates collectively support the implemention of 21 out of the total 31 checkpoints of OpsGL:
For the purposes of this project, we will consider that the existing templates supply adequate (cost-effective) support for their corresponding checkpoints. That is, we consider that the existing template implementations for these checkpoints have realized the highest-priority opportunities -- the most return for effort expended. There is some support for this view, for example in this user testimonial: "Thank you [...] I like the QAPD, it is a great tool for keeping our group organized!" [From unarchived email from a member of WCAG, Jenae Andershonis, who has been applying OpsGL/ET and the templates.]
However, we recommend, for a minor future improvement, that each of those template-supported checkpoints be looked at to assess two details:
Remaining opportunities for OpsGL/ET templates will be found in the 10 checkpoints that are not already in the template-supported list of checkpoints (see preceding section):
Checkpoint 2.3 reads: "Request allocation of QA resources to the Working Group." This refers to solicitation for QA-specific resources during the WG's Call for Participation. In email to the qa-chairs (team-only), Karl Dubost proposed a Quick Tips that addresses this checkpoint. This is a template in the general sense considered by this project report. It would be an excellent addition to OpsET, and would satisfy template support of CP2.3. We recommend that it be linked from OpsET.
All of the checkpoints of guidelines 1, 2, and 6 are therefore done -- the have adequate template support already built.
For the remaining 9 checkpoints, the following table contains our recommendations.
Guideline 3. Synchronize QA activities with the specification milestones. |
||
Nbr | Checkpoint | Recommendation |
---|---|---|
3.1 | Synchronize the publication of QA deliverables and the specification's drafts. [P2] |
tbd |
3.2 | Support specification versioning/errata in QA deliverables. [P1] |
tbd |
Guideline 4. Define the QA process. |
||
Nbr | Checkpoint | Recommendation |
4.1 | Appoint a QA moderator. [P1] |
simple task, no templates needed; "T" in OpsET should suggest where and how "appoint" may be done and documented. |
4.2 | Appoint a QA task force. [P2] |
refer to CP2.3 template |
[...checkpoints 4.3, 4.4, 4.5 are done...] | ||
Guideline 5. Plan test materials development. |
||
Nbr | Checkpoint | Recommendation |
5.2 | Ensure test materials are documented and usable for their intended purposes. [P1] |
simple task (and too variable per TS) -- no templates needed, but OpsET should point to good example(s). |
[...checkpoints 5.1, 5.3, 5.4, 5.5 are done...] | ||
Guideline 7. Plan the transfer of test materials to W3C if needed. |
||
Nbr | Checkpoint | Recommendation |
7.1 | Perform a quality assessment of any test materials that are candidates for transfer. [P2] |
tbd |
7.2 | Identify sufficient staff resources to meet the needs of any transferred test materials. [P1] |
tbd |
7.3 | For any transferred test materials, resolve all IPR issues with the external party that produced the test materials. [P1] |
tbd |
Guideline 8. Plan for test materials maintenance. |
||
Nbr | Checkpoint | Recommendation |
8.1 | Provide for the long-term maintenance of the contribution and review procedures. |
tbd |
[...checkpoints 8.2, 8.3 are done...] |
In addition to producing a full set of templates to aid implementation of OpsGL (this also applies to SpecGL), the templates could be made easier to use. Right now, the existing OpsGL/ET templates are XHTML documents with instructions for filling in blanks, chosing one option from an enumerated set, etc.
These templates could be made easier to use by writing a form or script, that would interactively solicit choices, options, and content from the user, and would generate an XHTML skeleton document. We recommend that the level of effort be assessed, and interactive API (form or script) for the bigger, composite (multi-CP) templates be considered.
This report was prepared with the support of National Institute of Standards and Technology, Gaithersburg, MD 20899-8970.