Template Assessment Report

Author:
Lofton Henderson
Date of this version:
31 January 2004

Table of Contents


Introduction

This report includes:

A future version of this report should complete some items that could still user more attention: an outline of a whole-spec template for SpecGL; a closer look at some individual-CP templates for SpecGL; and, consideration of templates for a handful of remaining OpsGL checkpoints.

Terminology

The rest of this report uses these terms, acronyms, and abbreviations::

Composite abbreviations such as OpsGL/ET are also used.

Methodology

Basic principles

The usability of any one of the GL documents is a result of several factors in combination. For example, the following bits in combination determine the usability of SpecGL:

  1. The clarity and completeness of SpecGL.
  2. The number and quality of the Examples in SpecET.
  3. The number and quality of the Techniques in SpecET.
  4. The provision of good templates as adjuncts to the Techniques of SpecET.

The same usability factors apply OpsGL also.

Method

An exhaustive analysis of the Candidate Recommendation (CR) text of SpecGL/ET was done. This uncovered lots of editorial and a few substantive issues with SpecGL. The latter pertain to the exact definition of the requirements of some checkpoints, hence their implementability. This gives a clear picture of where the SpecGL/SpecET combination needs to be enhanced with templates, and/or Examples ("E"), and/or techniques ("T").

Less of an analysis of OpsGL/ET was done. "Low fruit" has already been picked with OpsET and its two templates, the Charter template [CHART-TEMPL] and the QA Process Document template [QAPD-TEMPL]. However the results of SpecGL analysis suggest that there will be value in performing an OpsGL analysis for clarity, testability, and implementability issues; and, editorial problems -- this is recommended for future work or for the W3C editors.

Issues & resolutions

What is a template? The existing OpsGL/ET Charter template and QAPD template fit a traditional concept of templates -- language, structure, and organization is laid out, with indications of where customization by the user is needed. In some cases, analysis suggested that a simple model or prototype sentence in the techniques parts of SpecET or OpsET would be sufficient. Does a simple model sentence qualify as a template? In another case, a "Quick Tips" (QT) supplement to OpsET -- discussing and instructing how to do the checkpoint, with sections of model language -- seemed to be an ideal solution. Is a QT a template?

Since the goal is to improve the usability of SpecGL and OpsGL, we decided to take a broad view of what qualifies as a template:

SpecGL template assessment

SpecGL/ET analysis results

The SpecGL/ET analysis took a look at 1, 2, and 3 (above). The results are condensed into a SpecGL analysis table [SpecGL-data].

#1 resulted in a collection of editorial and substantive comments (issues) about SpecGL, which may also be found in the analysis results.

#2 and #3 are necessarily somewhat subjective. I read the checkpoint in SpecGL, and then followed the link to SpecET. I tried to imagine how an average user (not a QAWG expert) would understand the SpecET content and what he/she would gain from it.

These data are contained in the per-checkpoint table of analysis results [SpecGL-data].

The table also has a "Where" datum for each checkpoint. With two exceptions where SpecGL requires a particular spec location, I considered and chose a "good" location (or set of locations) for a CP's fulfillment. These are not unique or normatively exclusive of other possibilities, but ought to result in a clear and nicely arranged spec. This is the purpose of the templates ajdunct to SpecGL/ET -- give one good, easy way to do it, rather than explore all possibilities.

The "where" data turned out to be more useful than expected -- collectively, they suggested what I consider to be the most useful of the templates: an ideal-CC (Conformance Clause) skeleton, and an all-spec skeleton.

For "Templates potential" item, I tried, per-CP, to assess the degree to which the checkpoint's implementability would be enhanced by having a template (presumably linked from SpecET). Note that the potential refers only to whether the particular checkpoint (or small collection of closely related CPs) has potential to benefit from a template focused on that CP. The assessment ignores what might be the highest priority template -- a super- or composite template that outlines a complete specification, reflecting (almost) all of SpecGL's CPs, and probably embedding a number of specific and focused templates. In other words, a AAA-compliant skeleton specification.

Deferred consideration

We defer consideration of templates for these guidelines and checkpoints, because we think that either their precise meaning and intent is still too unclear, or there are other substantive issues that need to be resolved before designing supporting templates:

See the SpecGL analysis [SpecGL-data] for details about the problems with these.

Potential SpecGL/ET templates

Individual

Our initial focus in the analysis was on the potential of each checkpoint, in isolation, or a small number of closely related checkpoints. Analysis revealed relatively limited possibilities here.

Composite

The analysis of where in a specification the checkpoints might ideally be realized suggested that one or more multi-checkpoint templates, similar to [CHART-TEMPL] and [QAPD-TEMPL], might be most useful to SpecGL implementors (already done for OpsGL).

per-GL

During the analysis of SpecGL/ET, a few cases suggested that a template for a single guideline and its checkpoints might be useful. This is an interesting possibility that ought to be looked at more closely in the future.

Recommendations for SpecGL/ET templates

Top priority

Of the options, a Conformance Clause (CC) template seems to be the single first choice for most value and utility to SpecGL implementors. The CC template will have a placeholder, sub-section, choice, slot, or model language, in a single document chapter (comprising the Conformance Clause), for each of the CP's that the SpecGL analysis table [SpecGL-data] contains "CC" in the Where column:

There may also be possibilities for the two checkpoints of GL10, but this needs further thought.

The CC template will result in a skeleton Conformance Clause chapter for the specification that gives at least some support for 31 of the total 46 checkpoints of SpecGL.

For many of the checkpoints, the support might be relatively limited, e.g., "do it here for CPx.y". That is, it may do no more than instruct the user, at a certain location in the skeleton CC chapter, to do what is described in the CP's Conformance Requirements. This alone -- a well-organized CC chapter outline that addresses all of these checkpoints -- is still of great value. There may be further opportunities for enhancement by tieing it together with some individual-CP templates (see below).

Full-spec template

Currently, we give second priority to a full-spec template. This would be similar to the CC template, but would include template-section bits for each of a number of other checkpoints that are, for example, required to be in the specifications "Introduction" chapter.

The main features of this template would be:

It is a topic of further study, whether it could be productive to try to provide templates (of at least the "do it here" variety) for some of the CP's for which the SpecGL analysis table [SpecGL-data] indicates as "body" in the Where column.

Individual-CP templates

These might be used standalone (linked from the techniques section of SpecET), or they might be linked from a CC template (see above) or a full-spec template (see above), or both.

Summary of SpecGL/ET recommendations

Fix GL issues

Some CP are identified in SpecGL analysis table [SpecGL-data] as having substantive problems. The issues are briefly identified in the "Checkpoint" column, below the statement of the checkpoint. Queue these as SpecGL/ET issues and resolve them.

Fix ET bugs

In the ET/E and ET/T columns of the SpecGL analysis table [SpecGL-data], some of the CP have status entries "unr". This means that the example or technique is unrelated to the apparent intended meaning of the CP. Replace these with examples and/or techniques that properly reflect the CP's meaning. Note. Such occurrences may indicate problems with the clarity of the CP's statement, conformance requirements, etc.

Include more E & T

In the ET/E and ET/T columns of the SpecGL analysis table [SpecGL-data], some of the CP have status entries "0" or "minus". This means that there are either no examples/techniques, or that some are present but more are needed. These should be filled out more.

It is beyond the scope of this report to define, CP-by-CP, exactly what is needed to bring these cases to "ok" or "plus" status. This will have to be considered on a case-by-case basis.

Note that the implementation of the two recommended templates (below) will result in SpecET links from about 35 of the CPs. There will be opportunities, at the placeholder locations of many of the CPs in the template outline, to develop and include additional material.

Implement Conformance Clause template

Above we identified a Conformance Clause template as the top-priority template. This should be implemented. A CC template outline is provided, which can (should) be used as a starting point.

Implement Full-Spec template

Above we discussed a full-spec template as a second priority template effort. This should be implemented, after the CC template and some of its individual-CP suggestions. In a future version of this report, a full-spec template outline may be provided.

OpsGL template assessment

OpsGL/ET analysis

Given the value that the SpecGL analysis provided, for identification of GL/ET editorial and substantive issues, I recommend that the OpsGL editors perform such an analysis.

Existing OpsGL/ET template support

However, for the purposes of this project, a different approach was expedient. This is because significant template support has already been built for OpsGL/ET: Charter template for "QA Framework: Operational Guidelines" [CHART-TEMPL], and QA Process Document template for "QA Framework: Operational Guidelines" [QAPD-TEMPL]. These two templates collectively support the implemention of 21 out of the total 31 checkpoints of OpsGL:

For the purposes of this project, we will consider that the existing templates supply adequate (cost-effective) support for their corresponding checkpoints. That is, we consider that the existing template implementations for these checkpoints have realized the highest-priority opportunities -- the most return for effort expended. There is some support for this view, for example in this user testimonial: "Thank you [...] I like the QAPD, it is a great tool for keeping our group organized!" [From unarchived email from a member of WCAG, Jenae Andershonis, who has been applying OpsGL/ET and the templates.]

However, we recommend, for a minor future improvement, that each of those template-supported checkpoints be looked at to assess two details:

  1. sufficiency of the examples in OpsET;
  2. whether the technique, as implemented by the linked section in the template, might be further improved by linking its template section to a more detailed per-checkpoint template.

Candidates for new OpsGL/ET templates

Remaining opportunities for OpsGL/ET templates will be found in the 10 checkpoints that are not already in the template-supported list of checkpoints (see preceding section):

Checkpoint 2.3 reads: "Request allocation of QA resources to the Working Group." This refers to solicitation for QA-specific resources during the WG's Call for Participation. In email to the qa-chairs (team-only), Karl Dubost proposed a Quick Tips that addresses this checkpoint. This is a template in the general sense considered by this project report. It would be an excellent addition to OpsET, and would satisfy template support of CP2.3. We recommend that it be linked from OpsET.

All of the checkpoints of guidelines 1, 2, and 6 are therefore done -- the have adequate template support already built.

For the remaining 9 checkpoints, the following table contains our recommendations.

Guideline 3. Synchronize QA activities with the specification milestones.

Nbr Checkpoint Recommendation
3.1

Synchronize the publication of QA deliverables and the specification's drafts. [P2]

tbd
3.2

Support specification versioning/errata in QA deliverables. [P1]

tbd

Guideline 4. Define the QA process.

Nbr Checkpoint Recommendation
4.1

Appoint a QA moderator. [P1]

simple task, no templates needed; "T" in OpsET should suggest where and how "appoint" may be done and documented.
4.2

Appoint a QA task force. [P2]

refer to CP2.3 template
[...checkpoints 4.3, 4.4, 4.5 are done...]

Guideline 5. Plan test materials development.

Nbr Checkpoint Recommendation
5.2

Ensure test materials are documented and usable for their intended purposes. [P1]

simple task (and too variable per TS) -- no templates needed, but OpsET should point to good example(s).
[...checkpoints 5.1, 5.3, 5.4, 5.5 are done...]

Guideline 7. Plan the transfer of test materials to W3C if needed.

Nbr Checkpoint Recommendation
7.1

Perform a quality assessment of any test materials that are candidates for transfer. [P2]

tbd
7.2

Identify sufficient staff resources to meet the needs of any transferred test materials. [P1]

tbd
7.3

For any transferred test materials, resolve all IPR issues with the external party that produced the test materials. [P1]

tbd

Guideline 8. Plan for test materials maintenance.

Nbr Checkpoint Recommendation
8.1

Provide for the long-term maintenance of the contribution and review procedures.

tbd
[...checkpoints 8.2, 8.3 are done...]

Automating the templates

In addition to producing a full set of templates to aid implementation of OpsGL (this also applies to SpecGL), the templates could be made easier to use. Right now, the existing OpsGL/ET templates are XHTML documents with instructions for filling in blanks, chosing one option from an enumerated set, etc.

These templates could be made easier to use by writing a form or script, that would interactively solicit choices, options, and content from the user, and would generate an XHTML skeleton document. We recommend that the level of effort be assessed, and interactive API (form or script) for the bigger, composite (multi-CP) templates be considered.


Acknowledgements

This report was prepared with the support of National Institute of Standards and Technology, Gaithersburg, MD 20899-8970.


Change history

References

[CC-TEMPL]
A Conformance Clause Template outline, corresponding to the highest priority template recommended by this report. Available at CC-template-outline.html .
[CHART-TEMPL]
A Working Group Charter template, for demonstrating how to add QA-related information to the Charter. This is a fill-in-the-blank template that can be used by the WGs directly for implementing charter-related requirements of the Operational Guidelines [QAF-OPS] specification. September 2003, L. Henderson & L. Rosenthal, Eds. Version concurrent with this report is available at http://www.w3.org/QA/WG/2003/09/OpsET-charter-20030922.html .
[OpsET]
QA Framework: Operational Examples & Techniques, L. Henderson, L. Rosenthal, K. Gavrylyuk, D. Dimitriadis, Eds., W3C Note, (initially) September 2003, companion version to this document, latest companion version available at http://www.w3.org/QA/WG/2003/09/qaframe-ops-extech .
[OpsGL]
QA Framework: Operational Guidelines, L. Henderson, D. Hazaël-Massieux, L. Rosenthal, Eds., W3C Candidate Recommendation, 12 September 2003, companion version to this document, available at http://www.w3.org/TR/2003/CR-qaframe-ops-20030922/ .
[QAPD-TEMPL]
a Working Group QA Process Document template (QAPD), demonstrating how to satisfy all of the QAPD-related checkpoints of the operational guidelines. Both of these are fill-in-the-blank templates that can be used directly by the WGs for implementing QAPD-related requirements of the Operational Guidelines [QAF-OPS] specification. September 2003, L. Henderson & L. Rosenthal, Eds. Version concurrent with this report is available at http://www.w3.org/QA/WG/2003/09/OpsET-qapd-20030922.html
[SpecET]
QA Framework: Specification Examples & Techniques, Karl Dubost, Lofton Henderson, Lynne Rosenthal, Eds., current companion to [QAF-SPEC], available at http://www.w3.org/QA/WG/2003/09/qaframe-spec-extech-20030912 .
[SpecGL]
QA Framework: Specification Guidelines, D. Hazaël-Massieux, L. Rosenthal, L. Henderson, Eds., W3C Candidate Recommendation , available at http://www.w3.org/TR/2003/CR-qaframe-spec-20031110/ .
[SpecGL-data]
The raw SpecGL/ET analysis data for this project, available at qaframe-spec-tmpl-analysis.html .