This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.

Bug 1041 - Conformance is not a yes/no proposition (wrt filling an ICS)
Summary: Conformance is not a yes/no proposition (wrt filling an ICS)
Status: RESOLVED REMIND
Alias: None
Product: QA
Classification: Unclassified
Component: QASpec-GL (show other bugs)
Version: LC-2004-11-22
Hardware: All All
: P2 normal
Target Milestone: ---
Assignee: Karl Dubost
QA Contact: Karl Dubost
URL: http://lists.w3.org/Archives/Public/w...
Whiteboard:
Keywords:
Depends on:
Blocks: 1157
  Show dependency treegraph
 
Reported: 2005-01-21 09:37 UTC by Dominique Hazael-Massieux
Modified: 2005-04-28 11:53 UTC (History)
0 users

See Also:


Attachments

Description Dominique Hazael-Massieux 2005-01-21 09:37:05 UTC
"1.2 Good Practice B" suggests that an ICS form be provided with
yes/no questions: "1. Create a list, table or form listing all
features (capabilities) and indicating if it is mandatory or not. 2.
Provide space for the implementer to check: Yes, No, Not Applicable".
However, this is unrealistic. For example, take CSS user agents. How
is an implementor to determine if he has implemented margin collapsing
correctly? All that can really be said is that the user agent passes a
certain set of tests. For any even mildly complicated specification it
will always be possible to show that a user agent is in some way
non-compliant, it's just a matter of finding a suitable test.

Therefore I would suggest changing this section to instead suggest
leaving space for the implementor to list the URIs to (publically
available) tests that the implementor has used to verify
interoperability and compliance, listing which tests the implementor
determined passed in the user agent and which failed (if any).
Specification authors may wish to provide a list of URIs to the tests
that form part of the specification's formal test suite (as used to
check for interoperability as per the CR exit criteria), although
naturally such a test suite can never be complete enough to really be
used to claim conformance so implementors would be expected to also
provide links to other tests that they used.

(The existing suggestions could be kept for the rare specs in which a
test suite is inappropriate, such as the two examples the spec
currently gives: the QA spec guidelines and the WCAG. However, this
applies to very few specifications and so should not IMHO be the
primary suggestion in the document.)
Comment 1 Karl Dubost 2005-03-03 17:33:17 UTC
http://lists.w3.org/Archives/Public/www-qa-wg/2005Feb/0021.html

(dh) What I understand from Ian's email is that you cannot ever say  
that you are fully conformant to a spec; it's not a yes/no proposition.
(pc) Conformance is not binary. While it's not possible to claim with  
100% certainty that an implementation is conformant to every aspect of  
a specification, but rather as to whether the conformance _REQUIREMENT_  
has been met (for example, passing these tests etc). It's up to the  
spec writers to make it "binary" or easy to answer. We use the term  
conformanace in a special sense. If we require conformance in this  
sense, we need to rewrite specifications to meet this need.
(lr) To me, a conformance statement is a vendor declaration that they  
implement this or that feature.
(kd) [minuter: sound breaking up]... maybe we can add a clause to Spec  
GL ...
(dm) the ICS is also an input to the processor running the test
(lr) I'd say selecting the tests that are run. I'll take a stab at  
review and modify wording as necessary, by feb 11.
Comment 2 Karl Dubost 2005-03-03 19:22:18 UTC
http://lists.w3.org/Archives/Public/www-qa-wg/2005Feb/0028.html
       http://www.w3.org/Bugs/Public/show_bug.cgi?id=1041
       Ongoing: LH will modify the wording. Due date is February 21.
Comment 3 Karl Dubost 2005-03-03 19:24:05 UTC
http://lists.w3.org/Archives/Public/www-qa-wg/2005Feb/0014.html

Action Item  to modify 1.2 B GP ICS
I tweaked the what does it mean, removing 'conformance' from the first 
sentence, changing some works and adding a new last sentence.

What does it mean?
An Implementation Conformance Statement (ICS) provides information about an 
implementation to a specification, by presenting in a uniform manner the 
capabilities and optional features that have been implemented as well as 
the limitations of the implementation..   An ICS typically takes the form 
of a questionnaire or checklist for implementor to complete.  An ICS 
provides the implementor a way to indicate what has been implemented.
Comment 4 Dominique Hazael-Massieux 2005-03-08 14:26:42 UTC
Lynne has an action item to provide clarification wrt real meaning of ICS:
AI-20050303-1 LR to provide a 1 sentence disclaimer in the"What does it mean"
section that the ICS should positively emphasize that it's not about conformance.
Comment 5 Dominique Hazael-Massieux 2005-04-08 09:29:01 UTC
Lynne's proposal integrated as amended by later proposal.
Comment 6 Dominique Hazael-Massieux 2005-04-28 11:53:52 UTC
setting version to LC in case of future use