Index to Alex Rousskov's comments & issues:
- AR-001: fulfillment criteria, requirements, test
assertions, etc
- AR-002: wording of scope CP (1.1) and use of
scenario in fulfillment criteria
- AR-003: confused use of "scenarios" between CP1.2
and 1.1, clarification of intents in CP1.2
- AR-004: (deleted)
- AR-005: accuracy of rationale explanation in
CP1.2
- AR-006: in CP1.3, clarify "examples of what?"
- AR-007: (CP1.3 and throughout) clarify that
"include" or "link to" is authors' perogative
- AR-008: remove CP1.4 because it's
untestable.S
- AR-009: suggest that GL2 can be removed without
significant loss to SpecGL
- AR-010: GL3 and GL10 are redundant -- combine
them.
- AR-011: question the usefulness and priority of
CP3.1
- AR-012: meaning and intent of CP3.3 are
unclear
- AR-013: editorial
- AR-014: suggest improved wording of CP3.4, to
make more generally applicable
- AR-015: suggest improved wording of CP3.5, to
make more generally applicable
- AR-016: CP4.3 is not testable as written
- AR-017: CP5.1 is not testable as written
- AR-018: CP7.1 is not testable as written
- AR-019: suggest changing MUST to SHOULD in
CP7.4
- AR-020: remove "technique" from fulfillment
criteria of CP8.1; and fix untestable parts.
- AR-021: CP8.2 is not testable as written.
- AR-022: reword CP8.3 for clarity, precision; and,
is the CP needed?
- AR-023: remove CP8.4; or, at least clarify
undefined dependencies.
- AR-024: remove the "for example" part of GL9
verbiage.
- AR-025: wording of CP9.4 needs to be clarified
and terms/concepts defined.
- AR-026: remove CP9.6, because it's too narrow
- AR-027: checkpoint 10.2 is not testable as
written
- AR-028: editorial -- document needs spell
check
On Tue, 24 Dec 2002, Lofton Henderson wrote:
> ...
> These versions are the base documents for discussion at the QA
WG/IG
> face-to-face meeting next month (6-8 January 2003) in Seattle [1].
> We welcome comments, on this list. If you comment before the
> Seattle meeting, we will be able to discuss and resolve your
> comments in time to potentially affect the Last Call versions.
Here are a few comments. Original SpecGL text is quoted using a ">"
prefix.
AR-001:
> * a statement of how to fulfill the checkpoint
All these statements should be reworded to remove "To fulfill this
checkpoint," phrases that only pollute requirements.
[LH note. This ties into issue 99 and mark's
issue (@@link) about TA vs. requirements.]
AR-002(2):
> Checkpoint 1.1. Define the scope of the
specification.
>
> To fulfill this checkpoint, a specification MUST
define what
> scenarios are in scope and SHOULD identify out of
scope scenarios.
The above wording is not precise enough. It is not clear whether ALL or
just
some in- and out-of-scope scenarios must/should be defined. Is one
scenario
enough? This gets pretty messy because "scenario" (or use case) is not a
well-defined term.
If ALL scenarios must be defined, then the checkpoint becomes untestable
because it is impossible, under general assumptions, to know whether all
scenarios have been covered by the scope definition.
Defining out of scope scenarios should be a MAY. There is usually no
harm
whatsoever in not explicitly listing out-of-scope scenarios. In fact,
the presence of out-of-scope scenarios often indicates shortcuts that
the
authors had to take: "security considerations are currently out of
scope".
To solve the above problems, I would suggest that the above wording is
rephrased as follows:
Checkpoint 1.1. Define the scope of the
specification.
A specification MUST define its scope.
A specification MAY define what is outside of its
scope.
[LH note. Add this sub-issue: is the scope definition a sum of scenarios,
as currently worded? I think not. Consider the scope definition of OpsGL
or SpecGL.
They involve no scenarios at all.]
AR-003(2):
> Checkpoint 1.2. Provide Usage Scenarios. [Priority
2]
> To fulfill this checkpoint, a specification MUST
include or link
> to Usage Scenarios, SHOULD have an extensive list
of the usage
> scenarios that the authors have in mind
The difference between "usage scenarios" in checkpoint 1.2 and
"scenarios in scope" in 1.1 is not clear. If there is no difference,
than checkpoint 1.2 is identical to 1.1. If there is a difference it
should be explained. Note that the problem does not exist if the
alternative wording for checkpoint 1.1 is used; in that case, 1.1 may be
the same as 1.2 (depending on how the spec authors choose to define
"scope"), but that's OK.
Also, what is the difference between "includ[ing] Usage Scenarios" and
"hav[ing] a list of the usage scenarios"? Isn't that the same thing,
except the latter is more specific (too specific, perhaps).
To solve the above problems, I would suggest that the above wording is
rephrased as follows:
Checkpoint 1.2. Provide Usage Scenarios. [Priority
2]
A specification MUST document at least one Usage
Scenario.
Non-normative text should probably say that "documented Usage Scenarious
do not define or limit the scope of the specification unless noted
otherwise." We should make it clear that Usage Scenarios are their for
illustrative purposes only and, in general, cannot cover all valid uses.
In other words, Usage Scenarios are non-normative _examples_.
AR-005:
> Rationale: Usage scenarious help to assess what
features are
> missing and what features are superfluous in the
specification.
Usage scenarios on their own cannot detect missing or extra features.
Usage scenarios just illustrate scope or applicability of the
specification. The above can be reworded to make more practical sense:
Rationale: documenting usage scenarios helps
specification
authors to assess specification scope
AR-006:
> Checkpoint 1.3. Provide examples. [Priority 1]
> To fulfill this checkpoint, a specification MUST
include or link to
> at least one example.
An example of what? Are we talking about usage examples/scenarios (then
it is a repetition of 1.2), something else?
AR-007:
Here and elsewhere, replace "include or link" and "include or reference"
with "document" or "specify". It should be up to the authors whether
linking is an acceptable documentation/specification method (and the
choice will probably depend on the context).
AR-008:
> Checkpoint 1.4. Provide an example for each section
that is
> difficult to understand . [Priority 2]
Remove this checkpoint: it is not testable for several reasons and it
restricts authors in how complexity should be addressed. The intent here
is clear: we want specifications to be easy to understand. We cannot
demand that via a testable checkpoint though. It should be our hope that
other [testable] checkpoints and authors goodwill will lead to specs
that are easier to understand.
AR-009:
> Guideline 2. Identify what needs to conform and
how.
What negative impact will removing this guideline from SpecGL have? Its
only
contribution is an attempt to standardize spec categories (without
defining
those proposed categories!). Instead of having a special guideline, can
we
just suggest that proposed categories are used when "conformance" is
defined
in the specification (subject of Guidelines 3 and 10)?
AR-010:
> Guideline 10, "conformance clause" is related to
this guideline.
> This Guideline 3 focuses on the establishment and
scope of
> definition of a conformance policy, while Guideline
10 focuses on
> where and how to document it. That is, the
verification of these
> checkpoints will require looking at the Conformance
Clause.
There is so much overlap between GL3 and GL10 that I have to question
the rationale for keeping those two guidelines separate. I can
understand "where and how to document" focus. I am having trouble to
interpret "the establishment and scope of definition" focus. Does not
documentation imply establishment? Does not [good] documentation of
conformance policy implies its definition? I would suggest that GL3 and
GL10 are merged into one guideline:
GL?: Define conformance.
AR-011(2):
> Checkpoint 3.1. Specify any universal requirements
for minimum
> functionality. [Priority 1]
>
> To fulfill this checkpoint, a specification MUST
include a
> normative section detailing any universal
requirements for minimum
> functionality. It is not applicable if there are
not any universal
> requirements.
This checkpoint seems to be too specific/narrow and not practically
useful given a good conformance policy. That is, in complex specs with
many subjects, there cannot be a universal minimum functionality
requirement. In simpler specs the minimum functionality should be
obvious from the conformance definition/rules. In many specs, it would
be impractical to confine the description to one section.
I also question the priority level assigned to this checkpoint. Again,
given a good conformance clause, having "minimum functionality"
documented is a nice-to-have not must-have.
If you think the checkpoint is still useful enough, I would suggest to
reword it for clarity:
Checkpoint 3.1. Specify universal requirements for
minimum
functionality. [Priority 3]
If specification has several subjects, and if all
specification
subjects have common minimum functionality, then
specification
SHOULD explicitly document that minimum.
AR-012(2):
> Checkpoint 3.3. Distinguish requirements from
product-specific
> extra features [Priority 1]
>
> To full this checkpoint, a specification MUST state
in its
> conformance section all facets of the requirements
where the
> required features represent the maximum allowable
capabilities.
To tell you the truth, I do not understand what this checkpoint and
accompanying rationale clause are trying to accomplish. The example in
the rationale is an example of an extension (support for a graphical
resolution other than those explicitly required to support). On the
other hand, the rational says "implementations may be allowed to exceed
the specified capabilities in ways other than having extensions". If I
am not the only one confused by this checkpoint, perhaps the author
could rephrase or clarify the intent.
Also, it is often not practical to list all applicable "facets"
(extensions or whatever) in the conformance section as there may be
hundreds of them.
[LH note. This is similar toissue #110 raised by
DHM]
AR-013:
> If profiles are used (see Guideline 4), state
whether ...
The above paragraph has two identical "If levels are used ..."
sentences.
AR-014:
> Checkpoint 3.4. If special conformance terms are
used, include a
> definition in the specification. [Priority 1]
>
> To fulfill this checkpoint, a specification MUST
either
> exclusively use conformance terms as defined in
this document or
> define any other conformance terms used in it and
reference them
> from the conformance clause.
Consider more general:
Checkpoint 3.4. Conformance terms must be defined.
Conformance terms MUST be defined.
Checkpoint comments may explain that external definitions are allowed
(of course) as long as their sources are explicitly cited. The current
checkpoint wording gives SpecGL a special status (only its definitions
can be used without re-defining them) and requires copying definitions
from any other spec.
AR-015(2):
> Checkpoint 3.5. Justify any usage of a dimension of
variability by
> reference to usage scenarios and requirements
[Priority 1]
> To fulfill this checkpoint, a specification MUST
provide a
> justification for each Dimension of Variability the
specification
> uses by reference to usage scenarios and
requirements.
Let's not suggest a specific and rather limited solution/technique.
Consider more general:
Checkpoint 3.5. Justify used dimensions of variability.
[Priority 1]
If a specification uses Dimensions of Variability,
it MUST justify each Dimension it
uses.
Same comment for Checkpoint 9.2, BTW.
AR-016:
> Checkpoint 4.3. If profiles are chosen, define
their relationships
> and interaction with other dimensions of
variability. [Priority 2]
> To fulfill this checkpoint, a specification MUST
identify all the
> relations and interactions between profiles and the
other
> dimensions of variability. It is not applicable if
profiles are not
> used.
This checkpoint is not testable. The authors may claim that all
interactions are identified, but it is not possible, in general, to
verify that the authors did not miss any implicit interactions. If we
are only concerned about explicit interactions, then all explicitly
defined interactions are, of course, "identified" (i.e., any spec would
satisfy the above checkpoint).
AR-017:
> Checkpoint 5.1. If modules are chosen, indicate any
mandatory
> conditions or constraints on their usage. [Priority
1]
>
> To fulfill this checkpoint, a specification MUST
document any
> mandatory conditions or constraints on their
usage.
This checkpoint is not testable. See comments for checkpoint 4.3 above.
AR-018:
> Checkpoint 7.1. Identify each deprecated feature.
[Priority 1]
>
> To fulfill this checkpoint, a specification MUST
identify in a
> normative section each deprecated feature. It is
not applicable if
> there is no deprecated feature.
This checkpoint is not testable. See comments for checkpoint 4.3 above.
AR-019:
> Checkpoint 7.4. Include examples to illustrate how
to avoid using
> deprecated features. [Priority 3]
>
> To fulfill this checkpoint, a specification MUST
provide an
> example for each deprecated feature showing how to
avoid using it.
> It is not applicable if there is no deprecated
features.
This should be SHOULD-level requirement because it is talking about
examples, which cannot be MUSTs because some specifications (i.e., those
written in formal languages) do not need them and cannot have them.
Also, we may want to keep inmind that requiring such examples often does
not
help much. For example, recent HTML specs often would say something like
this:
"<foo> is depricated, use style sheets instead" which is kind of an
example,
but may not specific enough for a non-guru to actually avoid using
<foo>.
AR-020(2):
> Checkpoint 8.1. State the circumstances for when
discretionary
> items are allowed [Priority 1]
>
> To fulfill this checkpoint, a specification MUST
indicate the
> rationale for the discretionary items by providing
a reference or
> link to its use cases and/or project requirements
and SHOULD
> identify by labeling all discretionary items. It is
not applicable
> for specifications that don't have discretionary
items.
SpecGL should not prescribe a specific technique ("by providing a reference
or
link" and "by labeling").
The SHOULD part is not testable. See comments for checkpoint 4.3 above.
AR-021:
> Checkpoint 8.2. For implementation dependencies,
address the
> allowable differences between implementations
[Priority 1]
> To fulfill this checkpoint, a specification MUST
describe any
> permitted variations or constraints for how the
implementation
> dependency is realized by implementations.
This checkpoint is not testable. See comments for checkpoint 4.3 above.
To summarize, pretty much all checkpoints of the "MUST idedtify all X"
form, where X is introduced by the specification itself are either not
testable (if X can be introduced implicitly) or self-referential (if X
must be defined to exist).
AR-022(3):
> Checkpoint 8.3. Indicate any constraints on the
number of
> choices/options that can be implemented [Priority
2]
>
> To fulfill this checkpoint, a specification MUST
indicate whether
> zero, exactly one, or a multiple of choices/options
are allowed to
> be implemented. It is not applicable for
specifications that don't
> use discretionary items or for implementation
dependent values among
> them.
What does "them" refer to? Also, let's not introduce any magic constants
like "zero" or "one":
Checkpoint 8.3. Indicate any constraints on the number
of
choices/options that can be implemented [Priority
2]
A specification MUST document how many choices/options a
single
implementation can support at a time.
I also question the necessity of this checkpoint. Why is it important?
There
is no rationale clause to explain. Should we demand documentation of
options
interaction instead? Most comments for checkpoint 8.4 below apply here as
well
("single implementation", "at a time").
AR-023:
> Checkpoint 8.4. Promote consistent handling of
discretionary
> choices. [Priority 2]
>
> To fulfill this checkpoint, the specification MUST
state that
> given identical conditions, the effect of a
discretionary choice is
> consistent within a single implementation.
>
> Rationale. Users have an expectation of what to
expect and should
> be able to count on getting the same results under
the same
> conditions with a given implementation.
This checkpoint should be removed. It introduces many undefined and/or
specific (not general enough terms):
- "identical conditions" (is current time or random
number generator
state included in the "condition"?)
- "effect of a choice" (effect on user? on log files? on
CPU
utilization? temperature in the room?)
- "consistent effect" (consistent with what? user
expectations or
design rationale? why not identical?)
- "single implementation" (if a user reconfigures a
product
run-time, is it the same implementation or a different
one?
what if the implementation gets automatic updates,
is
self-configurable, or uses true entropy source?)
- "same result" (see "effect of a choice")
If somebody believes the checkpoint must stay, please provide more
convincing arguments why every good specification on Earth that uses
discretionary items MUST state "given identical conditions, the effect
of a discretionary choice is consistent within a single implementation"
and close the above terminology holes. At the very least, MUST should be
replaced with MAY and priority should be decreased.
AR-024:
> For example, a specification that allows private
extensions (e.g.,
> proprietary) is highly likely to impede
interoperability, whereas a
> specification than permits only registered
extensions partially
> mitigates the negative impacts.
This is a classic example of wishful thinking. Even if the above thesis
was true in theory (public is always better than private), in real
world, it is usually impossible to enforce registration constraints and,
hence, private extensions are at least as easy to introduce as public
ones. In other words, when you talk about proprietary interoperability,
it really does not matter what specification permits because private
extensions usually do not have to be compliant to "work". I would
suggest that the above example is removed.
AR-025:
> Checkpoint 9.4. Use a standard mechanism to define
the extension.
> [Priority 3]
>
> To fulfill this checkpoint, a specification MUST
provide a
> standard way of defining the extension. It is not
applicable if
> extensions are not allowed.
What is a "standard mechanism/way"? Which standard does this checkpoint
refer to?
AR-026:
> Checkpoint 9.6. Require implementations to include
a way to "turn
> off" extensions. [Priority 3]
>
> To fulfill this checkpoint, a specification MUST
indicate via
> conformance requirements that implementations
provide a mode under
> which they produce only conforming content. This
checkpoint is not
> applicable if extensions are not allowed.
This checkpoint is probably too narrow and should be removed. To start
with, many implementations do not produce any content. Also, if
extensions do not hurt interoperability, it is not clear why a
conformant implementation MUST be able to disable them. Presence of
extensions does not imply conformance violations.
AR-027(2):
Checkpoint 10.2 is not testable. See comments for checkpoint 4.3 above.
AFAIK, IETF requires standards to have Security Considerations sections
in
addition to conformance sections. Should W3C do the same?
AR-028:
Finally, somebody needs to spellcheck SpecGL :-).