This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.

Bug 3759 - [F+O] round-half-to-even and float precision
Summary: [F+O] round-half-to-even and float precision
Alias: None
Product: XPath / XQuery / XSLT
Classification: Unclassified
Component: Functions and Operators 1.0 (show other bugs)
Version: Candidate Recommendation
Hardware: PC Windows XP
: P2 normal
Target Milestone: ---
Assignee: Ashok Malhotra
QA Contact: Mailing list for public feedback on specs from XSL and XML Query WGs
Depends on:
Reported: 2006-09-21 08:05 UTC by Michael Kay
Modified: 2007-02-25 23:34 UTC (History)
0 users

See Also:


Description Michael Kay 2006-09-21 08:05:16 UTC
This issue arises from an XSLT test case bug:

but it is not specific to XSLT. The question is, what exactly does it mean in F+O 6.4.5 when it says "If two such values are equally near (e.g. if the fractional part in $arg is exactly .500...), returns the one whose least significant digit is even. "

The case in question is 

round-half-to-even(xs:float(150.0150e0), 2)

At first sight 150.015 is equally close to 150.01 and 150.02, so it should be rounded up. However, xs:float(150.015e0) actually returns a number whose precise value is 150.0149993896484375, which would suggest rounding down. (This is what I think should be done, and what the published test results do). 

However, Joanne has argued:

xs:float(150.01) = 150.00999450683594
xs:float(150.015) = 150.01499938964844
xs:float(150.02) = 150.02000427246094

and therefore 150.015 is equally close to both. But I can't see any justification for this line of reasoning in the spec. The spec says we should compare with "a multiple of ten to the power of minus $precision.", not with an xs:float approximation to this.
Comment 1 Ashok Malhotra 2006-09-27 22:29:48 UTC
I'm not sure what to do abt this.  I understand both Mike and Joanne's arguments.
The summary of the function currently says "The value returned is the nearest (that is, numerically closest) value to $arg that is a multiple of ten to the power of minus $precision. If two such values are equally near (e.g. if the fractional part in $arg is exactly .500...), returns the one whose least significant digit is even."

Would it help to add something like "floating point numbers are compared using a decimal representation"?  But then we need to say something abt the number of decimal places.
Comment 2 Michael Kay 2006-09-27 22:55:29 UTC
I'm not sure what do about it either. I'd like to be consistent with other parts of the spec, but we are very strict about numeric precision in some places, and very lax in other places. My own implementation of round-half-to-even(xs:float) does the arithmetic in double precision, which I think is reasonable but it's hard to argue that it's the only reasonable approach - you still get rounding errors, but they are smaller than with float.

I think at this stage it's probably best to add a note saying that in the case of xs:float and xs:double, the arithmetic used to implement round-half-to-even may use floating point arithmetic, and that we don't prescribe the exact algorithm which means that rounding errors may occur and are implementation-dependent; the effect is that a number that is close to half-way between two powers of ten may round either up or down. 
Comment 3 Michael Kay 2006-10-24 15:50:06 UTC
The working group accepted the following proposal, which was made by email at (member-only):

OPTION 2: define the function to work by converting the argument to decimal,
and then converting the result back to the original type. Below is suggested
text for that option.

fn:round-half-to-even($arg as numeric?) as numeric?
fn:round-half-to-even($arg as numeric?, $precision as xs:integer) as

Summary: The value returned is the nearest (that is, numerically closest)
numeric to $arg that is a multiple of ten to the power of minus $precision.
If two such values are equally near (e.g. if the fractional part in $arg is
exactly .500...), the function returns the one whose least significant digit
is even. 

If type of $arg is one of the four numeric types xs:float, xs:double,
xs:decimal or xs:integer the type of the result is the same as the type of
$arg. If the type of $arg is a type derived from one of the numeric types,
the result is an instance of the base numeric type.

The first signature of this function produces the same result as the second
signature with $precision=0.

For arguments of type xs:float and xs:double, if the argument is NaN,
positive or negative zero, or positive or negative infinity, then the result
is the same as the argument. In all other cases, the argument is cast to
xs:decimal, the function is applied to this xs:decimal value, and the
resulting xs:decimal value is cast back to xs:float or xs:double as
appropriate to form the function result. If the resulting xs:decimal value
is zero, then
positive or negative zero is returned according to the sign of the original

Note that the process of casting to xs:decimal may result in an error

This produces a well-defined and interoperable result provided that the
implementation supports sufficient decimal range and precision: though not
always the result that a naive user might expect.  For the example in

round-half-to-even(xs:float(150.0150e0), 2)

any implementation that supports the required 18 digits for xs:decimal will
convert the argument to the xs:decimal


which will then be rounded to the xs:decimal


which will be converted back to the xs:float whose exact value is


which will typically be displayed as


Michael Kay
Comment 4 Jim Melton 2007-02-25 23:34:57 UTC
Closing bug because commenter has not objected to the resolution posted and more than two weeks have passed.