There is normative guidance in the spec that says currentTime can be set on a media element when no src is defined, and non-normative guidance in a note that says doing this should cause an invalidStateError.  I just did a quick check, and behaviors for various browsers vary:

-  IE11:  returns an invalidStateError
-  Chrome:  returns a DOMException with a message that says the element readyState is HAVE_NOTHING
-  Firefox:  returns an invalidStateError

I believe the non-normative note is affecting the behaviors, and that it should be acceptable to set the default position on a media element with no src yet attached.  The correct behavior would seem to be setting the currentTime value with error.

--------------------------------
Details:

The non-normative note is just under http://www.w3.org/html/wg/drafts/html/master/embedded-content.html#offsets-into-

media . currentTime [ = value ]
Returns the official playback position, in seconds.
Can be set, to seek to the given time.
Will throw an InvalidStateError exception if there is no selected media resource or if there is a current media controller.

The normative guidance is in a paragraph lower down in the section:

The currentTime attribute must, on getting, return the media element's default playback start position, unless that is zero, in which case it must return the element's official playback position. The returned value must be expressed in seconds. On setting, if the media element has a current media controller, then the user agent must throw an InvalidStateError exception; otherwise, if the media element's readyState is HAVE_NOTHING, then it must set the media element's default playback start position to the new value; otherwise, it must set the official playback position to the new value and then seek to the new value. The new value must be interpreted as being in seconds.

The same language is in the WHATWG living HTML spec.