This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.
As far as I understand, a test-case should be considered 'fail' if: - it is reported as a FAIL - a test-case that is present in the test suite for which no result has been reported Yet a test-case reported as 'notRun' with an additional comment attribute specifying the reason for which the test-case was not run counts as 'fail' in the reports generated by the XSL transformation. Here is an example: If one reports <fots:test-case name="fn-collection-4" result="notRun" comment="Default collection is always an empty sequence."/> the test-case counts as 'fail' in the reports although having a way of setting default collection to something different than () is *not* a required functionality. I see 2 approaches to fix this: 1) remove the 'notRun' status altogether and add new dependencies for the test-cases that require functionality outside let's say "minimal conformance". This way the test-cases would be reported as pass/fail or n/a (meaning a dependency was not met). Because, IMHO, having a way of setting additional collations or setting the default collection to something different than () should be treated as a dependency. As a result, the following paragraph would be remove from "Running QT Test Suite": "Also: XQuery and XPath do not explicitly require implementations to provide a way for all values in the static and dynamic context to be set. For example, an implementation that does not allow any collations to be defined (other than the Unicode Codepoint Collation) would be technically conformant; so also would be an implementation in which the default collection is always an empty sequence. There is therefore an implicit dependency, when values are set in the environment, that the implementation allows such values to be set. An implementation that cannot set the context as required should behave in the same way as if a dependency has not been satisfied, ideally reporting the test as "notRun" with a reason." 2) update XSL stylesheet to count as 'fail' only test-cases reported as 'notRun' and do not have a comment attribute stating the reason why they were not run.
We think the reporting is consistent with the categories in the schema. In the schema for notRun it states: "Indicates that the test was not run for unspecified reasons, perhaps because the test uses a feature that is not yet implemented in the product under test. For statistical purposes this outcome is treated in the same way as "fail". This is the default outcome assumed for tests that are present in the test catalog on the date of the test run, but which are not present in the results submission." It seems that in the situation you are describing it would be more appropriate to use 'n/a' meaning the test is not applicable to your product. The following sentence needs to be changed: "An implementation that cannot set the context as required should behave in the same way as if a dependency has not been satisfied, ideally reporting the test as "notRun" with a reason." Here we need to change 'notRun' to 'n/a'.
> The following sentence needs to be changed: > > "An implementation that cannot set the context as required should behave in > the same way as if a dependency has not been satisfied, ideally reporting > the test as "notRun" with a reason." > > Here we need to change 'notRun' to 'n/a'. This is one approach, I agree. But in this case one more thing needs to happen: one would have to add 2 more dependencies in QT and add these dependencies to the impacted test-cases. Otherwise the meaning of 'n/a' is not consistent throughout QT test-suite and it will be a cause further ambiguities.
#11 test-cases needing the dependency for defining collations other than Unicode Codepoint Collation: "compare-010", "K-SeqDeepEqualFunc-56", "K-SeqDeepEqualFunc-57", "K-SeqDeepEqualFunc-58", "K-SeqDeepEqualFunc-59", "K-SeqDeepEqualFunc-60", "K-SeqDeepEqualFunc-61", "K-SeqDeepEqualFunc-62", "K-SeqDeepEqualFunc-63", "fn-substring-after-24", "fn-substring-before-24" #21 test-cases needing the dependency for setting default collections to something other than the empty sequence: "fn-collection-4", "fn-collection-4d", "fn-collection-5", "fn-collection-5d", "fn-collection-6", "fn-collection-8", "fn-collection-9", "fn-collection-10", "fn-collection-10d", "collection-004", "collection-008", "fn-function-lookup-370", "fn-function-lookup-372", "fn-function-lookup-374", "fn-function-lookup-376", "generate-id-012", "generate-id-013", "function-literal-370", "function-literal-372", "function-literal-374", "function-literal-376"
(In reply to comment #3) Any suggestions on the dependency names? I am thinking of: 1) For test-cases needing the dependency for defining collations other than Unicode Codepoint Collation: "non_unicode_codepoint_collation" 2) For test-cases needing the dependency for setting default collections to something other than the empty sequence: "non_empty_sequence_collection"
(In reply to comment #4) > (In reply to comment #3) > > Any suggestions on the dependency names? > > I am thinking of: > > 1) For test-cases needing the dependency for defining collations other than > Unicode Codepoint Collation: > > "non_unicode_codepoint_collation" +1 > 2) For test-cases needing the dependency for setting default collections to > something other than the empty sequence: > > "non_empty_sequence_collection" +1
Bug fixed. Added dependencies suggested in comment #4 to the test cases specified in comment #3
Sorry but test-case 'K-SeqDeepEqualFunc-63' from 'fn-deep-equal' needs a dependency for 'non_unicode_codepoint_collation' added. Here is the test-case below: <test-case name="K-SeqDeepEqualFunc-63" covers="named-function-ref"> <description>Use case-blind collation, if supported, in dynamic call</description> <created by="Michael Kay" on="2012-05-30"/> <modified by="O'Neil Delpratt" on="2013-08-02" change="Added feature dependency - Fix to bug issue #22423" /> <environment> <collation uri="http://www.w3.org/2010/09/qt-fots-catalog/collation/caseblind"/> </environment> <dependency type="spec" value="XQ30+"/> <dependency type="feature" value="higherOrderFunctions"/> <test><![CDATA[ let $deep := deep-equal#3 return $deep(<a><b>abc</b></a>, <a><b>ABC</b></a>, "http://www.w3.org/2010/09/qt-fots-catalog/collation/caseblind") ]]></test> <result> <assert-true/> </result> </test-case>
(In reply to comment #7) Thanks. Dependency now added for the test-case ''K-SeqDeepEqualFunc-63'