Section 5.1.7 states:
The doctype-public parameter MUST be ignored unless the doctype-system parameter is specified.
It is not clear whether "ignoring" the parameter means treating it as absent for the purposes of the rule in 6.1.7:
[if] the doctype-public and doctype-system serialization parameters are both absent ...
Test case output-0229 in the XSLT 3.0 test suite depends on the outcome.
I think treating it as absent for the purposes of 6.1.7 would be most reasonable. My own opinion, not that of the working groups.
As was decided at the joint XSLT/XQuery call of 18 December 2012, I will change the text in section 6.1.7 from "if the doctype-public and doctype-system serialization parameters are both absent" to "if the doctype-system serialization parameter is absent," to make it clear that if doctype-system is absent, doctype-public has no effect for the XHTML output method.
This will be reflected in the next public working draft of Serialization 3.0.
As you were present when this decision was made, I will assume you are in agreement with these changes and mark the bug as CLOSED.
 https://lists.w3.org/Archives/Member/w3c-xsl-query/2012Dec/0054.html (Member-only link)