This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.

Bug 29409 - [FO31] fn:format-number with xs:double/float and (non-)rounding
Summary: [FO31] fn:format-number with xs:double/float and (non-)rounding
Alias: None
Product: XPath / XQuery / XSLT
Classification: Unclassified
Component: Functions and Operators 3.1 (show other bugs)
Version: Candidate Recommendation
Hardware: PC Windows NT
: P2 normal
Target Milestone: ---
Assignee: Michael Kay
QA Contact: Mailing list for public feedback on specs from XSL and XML Query WGs
Depends on:
Reported: 2016-01-31 02:24 UTC by Abel Braaksma
Modified: 2016-03-22 09:32 UTC (History)
1 user (show)

See Also:


Description Abel Braaksma 2016-01-31 02:24:20 UTC
Section 4.7.5 Formatting the Number says, under item 6:

"The mantissa is converted (if necessary) to an xs:decimal value, using an implementation of xs:decimal that imposes no limits on the totalDigits or fractionDigits facets. "

And, on xs:double/float specifically:

"the one that is chosen should be one with the smallest possible number of digits not counting leading or trailing zeroes (whether significant or insignificant). For example, 1.0 is preferred to 0.9999999999, and 100000000 is preferred to 100000001" 

I fail to see how 100000000 is preferred over 100000001 (both have the same amount of digits, the second having in part non-zero trailing digits). I also don't see how this is supposed to work. Consider:

format-number(1E26, '#')

This can return '10000000000000000905969664' because converting the double value 1E26 to xs:decimal is 10000000000000000905969664.

However, the text may imply that this should in fact return '10000000000000000000000000', but I don't see how that is supposed to happen.

The way I understand the text is that the example above should format the same way as format-number(xs:decimal(1E26), '#'), where there's no limit in the decimal digits. This seems in line with other parts of the spec where conversion from double to decimal retains the insignificant digits.

Relevant tests that show a different interpretation are numberformat60a, numberformat60m, numberformat60n, numberformat60o, numberformat60p, numberformat60q. Taking the latter:

format(1E100 div 3, '#####################')

I think this should (may, ought to, must?) return 

and not 

But apart from the relevant tests, I raised this as a spec bug because, even if the text is considered correct, perhaps an example could be given for this particular entry.
Comment 1 Andrew Coleman 2016-02-04 14:02:17 UTC
At the teleconference on 2016-02-02, the joint WG agreed to close this bug with no action.
Comment 2 Abel Braaksma 2016-02-04 14:42:52 UTC
Just a short rationale and FYI for the record: the text in the spec says "not counting leading and trailing zeroes". This includes non-fractional zeroes.

As such, *if and only if* the number xs:float(120001 is represented by the same exact IEEE-754 representation as xs:float(120000), then format-number should (not must!) output the string "120000", as it is 2 digits (trailing zeroes removed) against 6. It was recognized during the meeting that such an algorithm is known to be non-trivial, hence the "SHOULD" and not "MUST" in the spec.

Bugs may separately be raised against the spec to add a dependency, or add an alternative outcome, but this bug will not be moved to QT3.