This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.

Bug 2407 - Rounding problems, implementation defined
Summary: Rounding problems, implementation defined
Status: CLOSED FIXED
Alias: None
Product: XML Query Test Suite
Classification: Unclassified
Component: XML Query Test Suite (show other bugs)
Version: 0.7.0
Hardware: PC Linux
: P2 normal
Target Milestone: ---
Assignee: Carmelo Montanez
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2005-10-21 16:37 UTC by Martin Probst
Modified: 2006-06-22 12:56 UTC (History)
0 users

See Also:

Attachments

Description Martin Probst 2005-10-21 16:37:58 UTC 
We have problems running the following tests because of rounding errors. E.g.
the tests generally pass in our implementation and return a correct result, but
the string comparison fails because of rounding. This is because the precision
an xs:decimal must provide is generally implementation defined (though required
to be at least 18). See http://www.w3.org/TR/xmlschema-2/#decimal.

InternalVariablesWithout VarDecl020
  Expects an xs:decimal to support 27 digits after the colon.

The following tests fail because they expect less precision than we provide. It
seems they expect 17 digits of precision, so it's actually impossible to meet
both tests.

NumericDivide op-numeric-divideint2args-2
NumericDivide op-numeric-divideint2args-3
NumericDivide op-numeric-divideint2args-4
NumericDivide op-numeric-divideint2args-5

NumericDivide op-numeric-divideintg2args-2
NumericDivide op-numeric-divideintg2args-4

NumericDivide op-numeric-dividedec2args-2
NumericDivide op-numeric-dividedec2args-4

NumericDivide op-numeric-dividelng2args-2
NumericDivide op-numeric-dividelng2args-4

NumericDivide op-numeric-dividenint2args-2
NumericDivide op-numeric-dividenint2args-4

NumericDivide op-numeric-dividepint2args-4

NumericDivide op-numeric-dividenpi2args-2
NumericDivide op-numeric-dividenpi2args-4

NumericDivide op-numeric-dividesht2args-4
NumericDivide op-numeric-dividesht2args-5

Please note that these tests do not produce doubles or floats, which have a
defined precision. It would probably be best to do a round-to-half-even($num,
18) on all of these values before comparing the results.
Comment 1 Martin Probst 2005-10-24 13:56:17 UTC 
Same goes for:

op-divide-dayTimeDuration-by-dTD-1
op-divide-dayTimeDuration-by-dTD-11
Comment 2 Martin Probst 2005-10-24 14:02:07 UTC 
Plus:

op-divide-yearMonthDuration-by-yMD-11
Comment 3 Carmelo Montanez 2005-10-28 19:20:38 UTC 
Tests changed.  Added the "round-to-half-even" function to tests and
adjust precision to match the type.  Please note that precision is
not the main point of the test.  So long as not asking for precision beyond
what is stated in teh specs, the test is ok.

Please close the bug if you agree with solution.

Thanks,
Carmelo Montanez