The specification says in 14.7.3 fn:serialize

"The key of the entry is an xs:string value in the cases of parameter names defined in these specifications, or an xs:QName (with non-absent namespace) in the case of implementation-defined serialization parameters."

and test case serialize-xml-120

let $params := map { QName("","indent") : true() }
return serialize(., $params)

expects SEPM0017.

However, in 1.5 Options we read

"It is not an error if the options map contains options with names other than those described in this specification. Implementations may attach an ·implementation-defined· meaning to such entries, and may define errors that arise if such entries are present with invalid values. Implementations must ignore such entries unless they have a specific ·implementation-defined· meaning. Implementations that define additional options in this way should use values of type xs:QName as the option names, using an appropriate namespace."

which seems to say this option can be ignored, or handled in an implementation-defined way.