I'm changing this to a spec bug. The documentation of the schema for serialization parameters claims that it is extensible, but the only extension point appears to be (uselessly) in the character-map element. I think the schema should contain an xs:any wildcard in the complexType of element serialization-parameters if it is to match its documentation.

I think it could be extended by creating another schema that uses the substitution group, e.g.

<xs:import namespace="http://www.w3.org/2010/xslt-xquery-serialization"/>
...
<xs:element name="xindent" type="..." substitutionGroup="output:serialization-parameter-element"/>
...

Am I missing something?

As agreed at the telecon on 2014-12-02, the schema definition for serialization-parameters should be a repeating choice group containing the parameters defined in this spec together with a wildcard (xs:any) for extension parameters.  I have modified the schema to the following...

  <xs:element name="serialization-parameters">
    <xs:complexType>
      <xs:choice minOccurs="0" maxOccurs="unbounded">
        <xs:element ref="output:serialization-parameter-element"/>
        <xs:any namespace="##other"
                processContents="lax"/>
      </xs:choice>
    </xs:complexType>
  </xs:element>

I am reopening this because the change is not backwards compatible.  If a 3.0 implementation currently extends the schema as described in comment 2, that schema is no longer be valid with the change described in comment 3.  The element declaration "xindent", which would not be in the standard namespace [2], now causes a violation of the "Unique Particle Attribution Constraint" [1].  It falls under the case "One is a wildcard and the other an element declaration, and the {target namespace} of any member of its ·substitution group· is ·valid· with respect to the {namespace constraint} of the wildcard".

[1] http://www.w3.org/TR/xmlschema-1/#non-ambig
[2] http://www.w3.org/2010/xslt-xquery-serialization

One possible way forward is to describe differently the way validation is performed on the parameters.  

If I'm reading the spec and our overall goals right, we want the effect of the rules to be:  if a serialization parameter defined in our spec is used, it should be valid against the definition in the spec (including the schema), but it's legitimate to have other serialization parameters, not defined in the spec.

(The PSVI is designed to provide just the information needed here, with a simple validation against the schema as given in the spec:  unrecognized parameters will cause the output:serialization-parameters element to be invalid, but the individual parameters will be valid or invalid, and recorded as such.  We cannot use that property here, though, because XQuery and XSLT both made the design error of confusing invalidity against a schema with an unconditional error condition, and our validation actions never return the PSVI if the validation root is invalid.  It was a mistake to define our validation primitives this way, and we are now reaping the reward of that design mistake.  And I would like to redeem my I-told-you-so chit now.)

We can work around the problem by changing the rule to say NOT:  validate the output:serialization-parameters element, and then raise an error if one of the parameters defined in the spec is invalid, but instead:  validate each child of output:serialization-parameters individually, against the schema in the spec, and raise an error if any of them are invalid.  Parameters not defined in the spec will not be valid, but they also won't be invalid:  they will have validity = notKnown.  

(Unless, of course, our validation routines also treat notKnown as equivalent to invalid, in which case we must say to raise an error if any parameter defined in the spec is invalid.)

So for 

  (validate lax { document { . } })
     /output:serialization-parameters
     /output:*[local-name() eq $param-name]/data(@value)

we should (I think) read

  (validate lax { document { ./output:serialization-parameters
     /output:*[local-name() eq $param-name] } })
     /data(@value)

and for

  <xsl:sequence>
    <xsl:variable name="validated-instance">
      <xsl:document validation="lax">
        <xsl:sequence select="."/>
      </xsl:document>
    </xsl:variable>
    <xsl:sequence select="$validated-instance
                          /output:serialization-parameters
                          /output:*[local-name() eq $param-name]/data(@value)"/>
  </xsl:sequence>

we should read

  <xsl:sequence>
    <xsl:variable name="validated-instance">
      <xsl:document validation="lax">
        <xsl:sequence select="./output:serialization-parameters
                          /output:*[local-name() eq $param-name]"/>
      </xsl:document>
    </xsl:variable>
    <xsl:sequence select="$validated-instance
                          /data(@value)"/>
  </xsl:sequence>

I apologize that I have not worked through the underbrush of the QT specs' descriptions of schema validation to see whether there are any other gotchas.  I intended to but am out of time.

Discussion suggested we might simplify this to

document{.}/
  output:serialization-parameters/
  (validate lax{output:*[local-name()=$param-name]})

The WG decided to adopt the proposal in comment #5 with the modification in #6

The changes suggested have been made and the editors' draft has been updated accordingly.  (That draft will, in due course, be made public, but I don't
know where; if anyone wishes to inspect this change but currently lacks access
to the editors' draft at [1], I will be happy to post a detailed list of the changes here.)

[1] https://www.w3.org/XML/Group/qtspecs/specifications/xslt-xquery-serialization-31/html/Overview-diff.html (member-only link)