Bug 19782 - test cases using 'assert-eq' for numeric values instead of 'assert'
Summary: test cases using 'assert-eq' for numeric values instead of 'assert'
Status: RESOLVED FIXED
Alias: None
Product: XPath / XQuery / XSLT
Classification: Unclassified
Component: XQuery 3 & XPath 3 Test Suite (show other bugs)
Version: Working drafts
Hardware: PC Linux
: P2 normal
Target Milestone: ---
Assignee: O'Neil Delpratt
QA Contact: Mailing list for public feedback on specs from XSL and XML Query WGs
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-10-30 16:06 UTC by Sorin Nasoi
Modified: 2012-11-08 15:45 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sorin Nasoi 2012-10-30 16:06:08 UTC
All test sets under 'math' folder have test cases where only 'assert-eq' is used for checking the result.

Please correct the tests such that reasonable differences to be allowed.
Possible fixes:
- replace the 'assert-eq' with 'assert'
- replace the 'assert-eq' with a combination of 'assert-any-of', 'assert-eq' and 'assert'


Here are 2 example of test cases that are wrong:

1) Test set 'math-acos', test case 'math-acos-002':
<test-case name="math-acos-002">
 <description>Evaluate the function math:acos() with the argument 0</description>
 <created by="O'Neil Delpratt, Saxonica" on="2010-12-10"/>
 <environment ref="math"/>
 <test>math:acos(0)</test>
 <result>
   <assert-eq>1.5707963267948966e0</assert-eq>
 </result>
</test-case>

Possible fix:
<test-case name="math-acos-002">
[...]
 <result>
   <assert>$result gt xs:double (1.57079632679489)</assert>
 </result>
</test-case>

2) Test set 'math-exp', test case 'math-exp-008':
<test-case name="math-exp-008">
  <description>Evaluate the function exp() with the argument set to xs:double('INF')</description>
  <created by="O'Neil Delpratt, Saxonica" on="2010-12-10"/>        
  <environment ref="math"/>        
  <test>math:exp(xs:double('INF'))</test>
  <result>
    <assert-eq>xs:double('INF')</assert-eq>
  </result>
</test-case>

Possible fix:
<test-case name="math-acos-002">
[...]
 <result>
   <assert>$result eq xs:double('INF')</assert>
 </result>
</test-case>
Comment 1 Sorin Nasoi 2012-10-30 23:11:25 UTC
Additional information: there are 74 total test cases inside the 'math' test sets that should be corrected.
Comment 2 Sorin Nasoi 2012-10-30 23:12:39 UTC
And there are 48 test cases part of the test sets from 'fn' folder.
Comment 3 Sorin Nasoi 2012-10-30 23:38:43 UTC
Here are 2 more examples of misuse of the 'assert-eq':

1) "op-numeric-add" test set, "op-numeric-addflt2args-3" test case:

<test-case name="op-numeric-addflt2args-3">
  <description> Evaluates The "op:numeric-add" operator with the arguments set as follows: $arg1 = xs:float(lower bound) $arg2 = xs:float(mid range) </description>
  <created by="Carmelo Montanez" on="2004-12-13"/>
  <test>xs:float("-3.4028235E38") + xs:float("0")</test>
  <result>
    <assert-eq>xs:float(-3.4028235E38)</assert-eq>
  </result>
</test-case>

Instead of :
<test-case name="op-numeric-addflt2args-3">
[...]
  <result>
   <assert>$result eq xs:float(-3.4028235E38)</assert>
  </result>
</test-case>

2)test set "op-numeric-divide", test case "op-numeric-dividesht2args-5":

<test-case name="op-numeric-dividesht2args-5">
 <description> Evaluates The "op:numeric-divide" operator with the arguments set as follows: $arg1 = xs:short(lower bound) $arg2 = xs:short(upper bound) </description>
 <created by="Carmelo Montanez" on="2004-12-13"/>
 <test>xs:short("-32768") div xs:short("32767")</test>
 <result>
   <any-of>
     <assert-eq>-1.000030518509475997</assert-eq>
     <assert-eq>-1.000030518509475997192297128208258</assert-eq>
     <assert-eq>-1.000030518509</assert-eq>
   </any-of>
 </result>
</test-case>

What if the XQuery processor returns '-1.0000305185094759972' ?
Comment 4 Sorin Nasoi 2012-10-31 00:09:52 UTC
<test-case name="fn-abs-1">
 <description>numeric types. Author: Oliver Hallam Date: 2010-03-15</description>
 <created by="Oliver Hallam" on="2010-03-15"/>
 <environment ref="empty"/>
 <test>string-join(for $x in (1, xs:decimal(2), xs:float(3), xs:double(4)) return 
           if ((abs($x)) instance of xs:integer) then "integer" 
           else if ((abs($x)) instance of xs:decimal) then "decimal" 
           else if ((abs($x)) instance of xs:float) then "float"
           else if ((abs($x)) instance of xs:double) then "double" else error(), " ")</test>
 <result>
  <assert-eq>"integer decimal float double"</assert-eq>
 </result>
</test-case>

instead of:
<test-case name="fn-abs-1">
[...]
 <result>
  <assert>$result eq "integer decimal float double"</assert>
 </result>
</test-case>
Comment 5 Michael Kay 2012-10-31 07:24:08 UTC
1. If you have enumerated all the affected tests then it would be helpful to supply the list. It would be even better to supply a shorter list where the changes are essential, since resources are limited.

2. I don't understand comment #4. What's wrong with the test as written, and how does it relate to this bug report?
Comment 6 Sorin Nasoi 2012-10-31 10:16:14 UTC
(In reply to comment #5)
> 1. If you have enumerated all the affected tests then it would be helpful to
> supply the list. It would be even better to supply a shorter list where the
> changes are essential, since resources are limited.
> 
> 2. I don't understand comment #4. What's wrong with the test as written, and
> how does it relate to this bug report?
Let me try to answer both issues in the same time because they are connected:
I implemented the function for checking the "assert-eq" into a FOTS driver.
I run all the tests in the FOTS that have this assertion defined and I analysed the failures.

I did not find any way of implementing an "assert-eq" check that is able to account for all the misuses of this assertion into the FOTS.

1) if I make the assumption that the "assert-eq" was created to compare the string values of the actual and expected results, than a bunch of tests fail.

Here are some examples:
<result><assert-eq>1.5707963267948966e0</assert-eq></result> compared to an actual result of 1.5707963267948966
<result><assert-eq>0e0</assert-eq></result> compared to an actual result of 0
<result>"X"</assert-eq></result> compared to an actual result X (because of the additional commas in the expected result)

2) if I make the assumption that the "assert-eq" was created to compare the values of the actual and expected results but only after casting them both to the type of the actual result another set of tests fail.
Some of the above mentioned tests pass, because 
xs:double(1.5707963267948966e0) eq xs:double(1.5707963267948966) and also xs:double (0e0) eq xs:double(0)

IMHO one of the main causes for these confusions (both for those writing tests and for those trying to write drivers that are able to run the tests correctly) is the lack of information in the FOTS catalog for the "assert-eq":

"assert-eq  denotes an element whose assumption when run must equal the expected test result."

IMHO we should give more specific details about how the result and actual results are compared:
- is there any serialization done to the actual result?
- the things that are compared are the string values or should there be some casting done first?

Also some common sense advices would also be very helpful, something like:
- if one is interested in checking the type of the results please consider using "assert-type" instead of "assert-eq"
- if one needs to compare numeric values that have decimals please consider using "assert" instead of "assert-eq".

So here is what I would propose:
- please add the necessary details into he FOTS schema for "assert-eq"
- after I implement the "assert-eq" in the driver it would be very simple for me to provide a verbose list of all tests in the FOTS that misuse "assert-eq" (according to the complete description from the FOTS schema).

Would this work?
Comment 7 Michael Kay 2012-10-31 11:13:09 UTC
>"assert-eq  denotes an element whose assumption when run must equal the
expected test result."

What my copy says is:

"The assert element contains an XPath expression (usually a simple string or numeric literal) which must be equal to the result of the test case under the rules of the XPath 'eq' operator. "

which seems clear enough. Are you using the latest CVS copy?
Comment 8 Sorin Nasoi 2012-10-31 11:18:07 UTC
(In reply to comment #7)
> >"assert-eq  denotes an element whose assumption when run must equal the
> expected test result."
> 
> What my copy says is:
> 
> "The assert element contains an XPath expression (usually a simple string or
> numeric literal) which must be equal to the result of the test case under
> the rules of the XPath 'eq' operator. "
> 
> which seems clear enough. Are you using the latest CVS copy?

I looked at this:
http://dev.w3.org/cvsweb/~checkout~/2011/QT3-test-suite/catalog-schema.html?rev=1.1;content-type=text%2Fhtml#elem_assert-eq
Comment 9 Sorin Nasoi 2012-10-31 11:29:54 UTC
(In reply to comment #8)
> (In reply to comment #7)
> > >"assert-eq  denotes an element whose assumption when run must equal the
> > expected test result."
> > 
> > What my copy says is:
> > 
> > "The assert element contains an XPath expression (usually a simple string or
> > numeric literal) which must be equal to the result of the test case under
> > the rules of the XPath 'eq' operator. "
> > 
> > which seems clear enough. Are you using the latest CVS copy?
> 
> I looked at this:
> http://dev.w3.org/cvsweb/~checkout~/2011/QT3-test-suite/catalog-schema.
> html?rev=1.1;content-type=text%2Fhtml#elem_assert-eq

There are some issues I see:
- the catalog-schema.xsd says what you have, that is true
- the catalog-schema.html says is not updated (so it mentions what I stated)
- the description in the catalog-schema.xsd does not account for the "empty environment" issue I reported in bug 19783: that is the fact that the "assert-eq" is not checked in the same environment defined for the test itself but in a default/empty one (where math prefix is not defined for instance).
Comment 10 Sorin Nasoi 2012-10-31 11:41:16 UTC
"For example, <assert-eq>12&</assert-eq> asserts that the result of the test expression is an atomic value that compares equal to the integer 12 (which means it might be the double value 12.0 or the float value 12.0 or the untyped atomic value "12.0", for example)."

Means that
<assert-eq>"integer decimal float double"</assert-eq>
would not succeed when compared to ('integer', 'decimal', 'float', 'double')
and would succeed when compared to ('"', 'integer', 'decimal', 'float', 'double', '"')
right?
Comment 12 Michael Kay 2012-10-31 13:20:27 UTC
>Means that
<assert-eq>"integer decimal float double"</assert-eq>
would not succeed when compared to ('integer', 'decimal', 'float', 'double')
and would succeed when compared to ('"', 'integer', 'decimal', 'float',
'double', '"') right?

This test, no doubt for historic reasons, returns a single string, not a sequence of four (or six) strings, and the assertion compares the result by comparing the value of that single string using the "eq" operator.
Comment 13 Sorin Nasoi 2012-11-01 15:18:52 UTC
As discussed, here is the list of test cases that I propose to be investigated:

- "math-acos" test set, "math-acos-003"

- "op-numeric-divide" test set, "op-numeric-dividesht2args-5" -replace alternate results with "assert"

- "prod-CastExpr" test set, "CastFOCA0001-1,CastFOCA0003-1" -replace alternate results with "assert"

- "prod-CastExpr" test set, "K2-SeqExprCast-17" - provide alternate result for <error code="FOAR0002"/>

- "prod-TryCatchExpr" test set "try-catch-optimizations-1" - the result is a sequence of 2 items
Comment 14 O'Neil Delpratt 2012-11-08 15:45:29 UTC
Bug fixed and committed to cvs.

(In reply to comment #13)
> - "math-acos" test set, "math-acos-003"

There is also a problem with the test case math-acos-003
Changed the expected result to the following:

<assert>abs($result - 1.5707963267948966e0) lt 1e-14</assert>

> - "op-numeric-divide" test set, "op-numeric-dividesht2args-5" -replace
> alternate results with "assert" 
> - "prod-CastExpr" test set, "CastFOCA0001-1,CastFOCA0003-1" -replace
> alternate results with "assert"
 
DONE

> - "prod-CastExpr" test set, "K2-SeqExprCast-17" - provide alternate result
> for <error code="FOAR0002"/>

DONE as suggested

> - "prod-TryCatchExpr" test set "try-catch-optimizations-1" - the result is a
> sequence of 2 items

My suggestion here (and I have made the change) is the following:
<assert-deep-eq>0, 1</assert-deep-eq>