Rule 6 for the option parameter conventions (§1.5) states:

For each named option, the function specification defines a required type for the option value. The value that is actually supplied in the map is converted to this required type using the function conversion rules. A type error [err:XPTY0004] occurs if conversion of the supplied value to the required type is not possible, or if this conversion delivers a coerced function whose invocation fails with a type error. A dynamic error occurs if the supplied value after conversion is not one of the permitted values for the option in question: the error codes for this error are defined in the specification of each function.

Now suppose the supplied value is map{"indent", xs:untypedAtomic('2')} (XSLT 3.0 test case xml-to-json-C104).

The second sentence of the rule above ("The value that is actually supplied in the map is converted to this required type using the function conversion rules") implies an error FORG0001.

But the third sentence ("A type error [err:XPTY0004] occurs if conversion of the supplied value to the required type is not possible") implies an error XPTY0004.

I think that the third sentence was intended as a gloss on the second, and the second sentence is normative. Suggested rewording:

For each named option, the function specification defines a required type for the option value. The value that is actually supplied in the map is converted to this required type using the function conversion rules. This may result in an error (typically [err:XPTY0004] or [err:FORG0001]) if conversion of the supplied value to the required type is not possible. A type error [err:XPTY0004] also occurs if the conversion delivers a coerced function whose invocation fails with a type error. A dynamic error occurs if the supplied value after conversion is not one of the permitted values for the option in question: the error codes for this error are defined in the specification of each function.