This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.
I've raised this in the last telco, but failed to find my own email or reproduce this issue, so here comes a bug report: I'm running the XQUTS test cases and there a tests containing states like this (from revalidate-valstrict-ins-007): <state time="0"> <query name="valstrict-ins-007" date="2008-03-25"/> <input-file role="principal-data" variable="books">books</input-file> <expected-error>XUST0026</expected-error> </state> The test running guidelines says a state is successful if "one or more expected errors, no expected results - the state is successful if one of the expected errors is raised". So this seems to say this needs to fail with XUST0026. However I think that is not correct, the test (and this state) is intended to be successful but returning no results, and implementations may raise an XUST0026 alternatively.
If the author of the test case believes that both an an error and successful execution are acceptable for this state, then I suggest that the metadata be changed to: <state time="0"> <query name="valstrict-ins-007" date="2008-03-25"/> <input-file role="principal-data" variable="books">books</input-file> <output-file role="principal" compare="Text">empty-sequence.txt</output-file> <expected-error>XUST0026</expected-error> </state> Where empty-sequence.txt is an empty file. In our design of the catalog, we didn't anticipate the need for "accept any successful result", so this is the best we can do. A similar state can be seen in test case parts-q3-2.
I am running into this issue as well. If you end up changing the guidelines, notice that there is some risk of allowing false positives for tests like id-comma-expr-04. Here is my list of effected tests: revalidate-valstrict-del-002 revalidate-valstrict-del-004 revalidate-valstrict-ins-003 revalidate-valstrict-ins-005 revalidate-valstrict-ins-007 revalidate-valstrict-ren-003 revalidate-valstrict-repnode-003 revalidate-vallax-del-002 revalidate-vallax-del-004 revalidate-vallax-ins-003 revalidate-vallax-ins-005 revalidate-vallax-ins-007 revalidate-vallax-ren-003 revalidate-vallax-repnode-003 removeType-valskip-ins-001 removeType-valskip-ins-002 removeType-valskip-ins-003 removeType-valskip-ins-004 removeType-valskip-ins-005 removeType-valskip-ins-006 removeType-valskip-ins-007 removeType-valskip-ins-008 removeType-valskip-ins-009 removeType-valskip-ins-010 removeType-valskip-ins-011 removeType-valskip-ins-012 removeType-valskip-ins-013 removeType-valskip-ins-014 removeType-valskip-del-001 removeType-valskip-del-002 removeType-valskip-del-003 removeType-valskip-del-004 removeType-valskip-del-005 removeType-valskip-ren-001 removeType-valskip-ren-002 removeType-valskip-ren-003 removeType-valskip-ren-005 removeType-valskip-repnode-001 removeType-valskip-repnode-002 removeType-valskip-repnode-003 removeType-valskip-repnode-004 removeType-valskip-repnode-005 removeType-valskip-repnode-006
I'm going to bite the bullet on this one. The intended purpose of these tests is to demonstrate that a product that supports revalidation does so correctly. The tests should be designed so that a product that does not support this feature does not run the test (it's a mockery for it to claim it has run the test and passed it). I think Andrew's solution proposed in comment #1 leaves too many options open for passing tests spuriously (as comment #2 observes). My proposal is to add to the GroupInfo for a test group an element such as <depends-on> <feature>revalidation:strict</feature> <description>The tests in this group should be run only by a product that supports revalidation mode strict</description> <spec-citation spec="XQUPDATE" section-number="2.2.1" section-title="Revalidation Declaration" section-pointer="id-revalidation-declaration"/> </depends-on> We can also add a test that a product that does not support the feature raises the error, using <feature>!revalidation:strict</feature> I would propose putting the feature "names" as an enumeration type into the schema, and using the same "names" in the <feature name=""> attribute in the report of test results. I know this might appear disruptive at this stage of the game, but I think this needs to be done properly.
I am not a big fan of making the catalog more complicated. If you decide that biting this bullet will break too many teeth, why not just let implementations fail these tests (e.g. In your example, take out XUST0026 as an option and add empty-sequence.txt as the expected result)? For testing XUST0026, you could make sure a few tests are in the minimal conformance section that allow the error or the proper result. But tests that aren't in minimal conformance only allow the proper result.
Related to this issue, the following tests need to be fixed too: fn-put-003 fn-put-004 fn-put-006 These tests only allow the error FOUP0001. My implementation supports putting text, pi, and comments so I should not have to throw this error to pass the test.
I have implemented the solution proposed in comment #3.
(1) Did you forget to annotate these groups with "feature" information? RevalidateStrict RevalidateTransform RevalidateLax (2) Your change breaks the HTML report generated by XQUTSResults.xsl. No implementation will currently receive "100% minimal conformance" in the report due to mutually exclusive tests like "fn-put-003" and "fn-put-003-fail". Implementations which do not implement all revalidation modes should not be penalized in the minimal conformance section of the report. An implementation that does not pass any revalidation tests in the minimal conformance should be penalized in the report.
(1) Many of the revalidation tests appear twice in the catalog as a result of a previous attempt to cater for all varieties of product. I'm now trying to sort this out - it should be possible to eliminate one of the versions. (2) I agree there's a need to change the reporting, I'll look into this.
I have now: (a) eliminated the duplicated revalidation tests (b) moved all tests for optional features into the optional features group (removing the distinction between optional XQuery features and optional Update features, which was not properly reflected in the content anyway) (c) revised the stylesheet for displaying results to ensure that products get a green light if they pass (i) all tests for mandatory features, or (ii) all tests that they choose to run for optional features
Created attachment 860 [details] A screen shot of the "Features" section in the generated report
The report looks good now except for one thing. In the detailed report, section "Features: " --> "Update Facility Static Typing Feature" lists "Anonymous: not specified" many times. (screen shot attached). Before your change, the only item was "Anonymous: true".
(In reply to comment #11) > The report looks good now except for one thing. In the detailed report, > section "Features: " --> "Update Facility Static Typing Feature" lists > "Anonymous: not specified" many times. (screen shot attached). Before your > change, the only item was "Anonymous: true". I have identified what was causing this and have fixed it. The problem was that the stylesheet uses "//feature", and elements named <feature> now appear in the new metadata associated with tests that are dependent on specific features. I'm now marking this as FIXED.