Re: Audio-ISSUE-105 (MIDI timestamp resolution): timestamps in MIDI should use High Resolution Time [MIDI API]

I had some further discussion off-list with Chris, where he helped me
coalesce my thinking a little better.

I think there are three design factors to a timestamp-

   1. What type the actual numerical value is (e.g. WebAudio's
   currentTime/time params are double, as is DOMHighResTimeStamp - some others
   are long ints, or long longs in Windows).  More importantly for JS devs,
   though is what the numeric value *represents*: i.e., number of seconds
   (in WebAudio) or number of milliseconds (DOMHRTS and MIDI proposal).
   2. What the "zero reference" (or start point) is - e.g. in WA, it's when
   the audioContext is created, for DOMHRTS in the window.performance.now()
   usage it's navigationStart (although by itself, the DOMHRTS type does not
   imply a "start point" - we need to fix this if we want to use it in the
   MIDI API, even if it's just to say it's the same as
   window.performance.now().)  For Date.now(), the zero reference/start point
   is 1 January 1970 00:00:00 UTC.
   3. On what clock the time proceeds.  This is somewhat subtle, because
   this is really to account for different clocks possibly running on
   different crystals - e.g., for WA this is the audio hardware clock. Like
   Data.now(), the performance.now() clock is just the system clock (except
   it's explicitly not supposed to adjust for system time[1]).  But just
   because you're using DOMHRTS type doesn't explicitly mean you're using the
   system clock - sections 4.3 and 4.4 are specific to the usage in
   window.performance.now(), and the rest of the spec simply defines the type
   (= #1 above).

For the purposes of the MIDI API, I don't have any real problem with using
the DOMHRTS type - but I'm a) not really psyched about having to use
window.performance.now() in a MIDI application to get the zero-reference,
because it seems like that is really designed to be a performance-timing
thing, and b) a little less pleased that Web Audio times are in seconds,
and this timestamp would be in milliseconds.  Neither of those are
completely disastrous, but they both seem less than optimal.  Anyone else
have thoughts on that?  Any thoughts about how developers usually think?

-C

[1]  http://www.w3.org/TR/2012/CR-hr-time-20120522/#sec-monotonic-clock.
 This makes complete sense in the performance context - you wouldn't want
performance measurements to be thrown by Daylight Savings time adjustments
or the user or network time adjust changing the current time.  That would
also make sense in a MIDI sequencer, so I guess mostly my objection a)
boils down to "I think performance is the wrong place to put this," which
is beyond our scope.  Sigh.


On Fri, Jun 1, 2012 at 1:34 PM, Chris Wilson <cwilso@google.com> wrote:

> This was what was making me uncomfortable - it's defined as relative to
> the Performance interface, navigation start time, which I think is a bit
> odd.
>
>
> On Fri, Jun 1, 2012 at 1:12 PM, Jussi Kalliokoski <
> jussi.kalliokoski@gmail.com> wrote:
>
>> Ah, you're right, in that case I don't see a reason to duplicate it.
>>
>> Cheers,
>> Jussi
>>
>> On Jun 1, 2012 11:09 PM, "Adam Goode" <agoode@google.com> wrote:
>> >
>> > On Fri, Jun 1, 2012 at 4:02 PM, Jussi Kalliokoski
>> > <jussi.kalliokoski@gmail.com> wrote:
>> > > The reason I kind of like the idea of having the timestamps specified
>> as
>> > > DOMHighResTimeStamps is that it will allow the accuracy to live
>> outside the
>> > > spec, for example if in the future it somehow becomes desirable to
>> have more
>> > > accuracy than double precision, the DOMHighResTimeStamp will probably
>> be
>> > > updated by then to use a higher precision as well. Although I don't
>> think a
>> > > use case for higher resolution than double will come along very soon.
>> Having
>> > > the timestamps be related to the creation time of the MIDIAccess is a
>> very
>> > > good idea actually, because it makes the problem of accuracy
>> deterioration a
>> > > slightly smaller problem. We probably need to introduce some method
>> to get
>> > > the current timestamp of the MIDIAccess as well.
>> > >
>> >
>> > If I'm reading the spec correctly, DOMHighResTimeStamp is defined to
>> > be relative to the start of navigation to the page. I don't think we
>> > should redefine this, but we could store the creation timestamp into
>> > each MIDIAccess object if necessary.
>> >
>> > http://www.w3.org/TR/2012/CR-hr-time-20120522/#sec-DOMHighResTimeStamp
>> >
>> >
>> > Adam
>> >
>> >
>> > > Cheers,
>> > > Jussi
>> > >
>> > >
>> > > On Fri, Jun 1, 2012 at 10:39 PM, Chris Wilson <cwilso@google.com>
>> wrote:
>> > >>
>> > >> Gah!  yes, sorry, didn't hit reply-all. Only thing in Gmail I'm
>> still not
>> > >> quite used to, somehow.
>> > >>
>> > >> Yes, I agree that it's not great to have so many different timestamp
>> > >> formats and reference points.  If the desire is to divorce from
>> wallclock
>> > >> time, then I supposed we could do like audioContext does - from when
>> > >> MIDIAccess is created.  As written in Jussi's last edit, though, it's
>> > >> "current time" (unfortunately, the definition of what that means (ms
>> since
>> > >> UNIX epoch) was removed).  I don't have strong feelings.  I mostly
>> disliked
>> > >> DOMHighResTimeStamp because it's one more reference, for what is
>> essentially
>> > >> a trivial thing (monotonically increasing, number of milliseconds,
>> unrelated
>> > >> to wallclock time), but that spec is really defined for uses
>> relating to
>> > >> Performance, so it's confusing to read as a solution for this
>> problem.  I
>> > >> think we would need to define our own zero point.
>> > >>
>> > >> I like seconds just because I think if it's not integer anyway, it's
>> > >> easier for humans to think that way, but I don't care that strongly.
>>  The
>> > >> newer MIDI interfaces in Windows, I note, use a longlong (64bit int)
>> of
>> > >> units of 100ns (i.e. tenths of a microsecond, or 0.0001
>> milliseconds).  I
>> > >> think that is kind of confusing, personally.  Seconds are prevalent
>> in the
>> > >> Web Audio API, but milliseconds (as ints) are common in other web
>> > >> programming APIs, so I could be okay with either.
>> > >>
>> > >> On Fri, Jun 1, 2012 at 12:17 PM, Adam Goode <agoode@google.com>
>> wrote:
>> > >>>
>> > >>> On Fri Jun 01 13:53:52 GMT-400 2012, Chris Wilson <
>> cwilso@google.com>
>> > >>> wrote:
>> > >>>>
>> > >>>> Well there you go - it's been quite a while since I wrote Windows
>> code.
>> > >>>>  :)
>> > >>>>
>> > >>>>
>> > >>>> >The point of DOMHighResTimeStamp is that it is divorced from
>> > >>>> > wallclock time.
>> > >>>>
>> > >>>> So is audioContext.currentTime.
>> > >>>>
>> > >>>
>> > >>> Hmmm. It's not great to have so many different timestamp formats and
>> > >>> reference points. It does make sense for audioContext to have its 0
>> point at
>> > >>> its start time. And there is no "start time" for these raw MIDI
>> events. So
>> > >>> deferring to page load time seems fine.
>> > >>>
>> > >>> But the units are different (seconds in float vs. milliseconds in
>> > >>> double), and that seems worth addressing.
>> > >>>
>> > >>>
>> > >>> (Did we drop off the public list with this thread?)
>> > >>>
>> > >>> Adam
>> > >>>
>> > >>>> On Fri, Jun 1, 2012 at 10:49 AM, Adam Goode <agoode@google.com>
>> wrote:
>> > >>>>
>> > >>>> On Fri, Jun 1, 2012 at 12:47 PM, Chris Wilson <cwilso@google.com>
>> wrote:
>> > >>>> >
>> > >>>> > Although I'm not completely opposed to this change, I'd argue
>> against
>> > >>>> > the point that millisecond resolution is insufficient.  If using
>> hardware
>> > >>>> > MIDI ports, it takes approximately 1/4 of a millisecond to SEND
>> a single
>> > >>>> > byte of data - so it will take approximately 3/4 of a
>> millisecond to simply
>> > >>>> > transfer the data anyway - and the latency in processing at the
>> other end is
>> > >>>> > typically much, much higher than 1ms (I seem to recall around
>> 4-7ms was not
>> > >>>> > atypical for hardware synths, but can't find my reference ATM).
>> > >>>> >
>> > >>>>
>> > >>>> The issue is more of jitter, not of processing delay. Though 1ms
>> seems
>> > >>>> totally sufficient to me, I could imagine issues with the single
>> byte
>> > >>>> timing code (F8) getting some unwanted jitter. But the real win of
>> > >>>> this change is monotonicity.
>> > >>>>
>> > >>>> >
>> > >>>> > That said, of course, it's not a bad idea to future-proof better
>> than
>> > >>>> > that; many MIDI use cases will never actually see a 5-pin-DIN
>> cable.
>> > >>>> >  However,
>> > >>>> >
>> > >>>> > 1) I find the usage of DOMHighResTimeStamp very confusing, as
>> it's
>> > >>>> > deliberately chained to (in terms of "zero" point) to the
>> Performance
>> > >>>> > interface.  It doesn't seem to add any value to reference here,
>> since it's
>> > >>>> > simply a double; we would still need to provide a way to get
>> system time in
>> > >>>> > double units, as I don't think using the PerformanceTiming
>> interface is the
>> > >>>> > most intuitive thing to do.  Or suggest that people use
>> Date.now() (even
>> > >>>> > though it's millisecond-precision), which is livable, I suppose.
>>  But we do
>> > >>>> > need to define that.  I would recommend either a) using a double
>> for number
>> > >>>> > of milliseconds, and recommending people use Date.now, or b) (my
>> preference)
>> > >>>> > use a double to represent number of seconds, to be uniform with
>> the Web
>> > >>>> > Audio API.  I'm ambivalent about whether we use the same
>> currentTime from
>> > >>>> > the audioContext as WA or Date.now().
>> > >>>> >
>> > >>>>
>> > >>>> The point of DOMHighResTimeStamp is that it is divorced from
>> wallclock
>> > >>>> time. All the MIDI implementations use this kind of time stamp
>> (even
>> > >>>> Windows, read on).
>> > >>>>
>> > >>>>
>> > >>>> >
>> > >>>> > 2) I would absolutely recommend that we (similar to
>> > >>>> > DOMHighResTimeStamp) explicitly state that implementations are
>> allowed to
>> > >>>> > have millisecond-only precision in their implementation.  The
>> underlying
>> > >>>> > system APIs on Windows are based in milliseconds, for example -
>> unless
>> > >>>> > they're building another API, the time stamps on MIM_DATA are in
>> > >>>> > milliseconds. The underlying API on OSX is a bit harder to
>> determine
>> > >>>> > precision, but I think it is higher.
>> > >>>> >
>> > >>>>
>> > >>>> Actually the ONLY part of DirectMusic that is undeprecated (it
>> > >>>> disappeared briefly in Vista, then was replaced in a service pack)
>> is
>> > >>>> high resolution monotonic MIDI timestamps:
>> > >>>>
>> > >>>>
>> http://msdn.microsoft.com/en-us/library/ee416788(VS.85).aspx#ID4EFEAC
>> > >>>> http://support.microsoft.com/kb/943253
>> > >>>>
>> > >>>>
>> > >>>> So yes, we can specify that the timestamps might only have ms
>> > >>>> resolution, but I don't think it's really required.
>> > >>>>
>> > >>>>
>> > >>>> Adam
>> > >>>>
>> > >>>> >
>> > >>>> >
>> > >>>> >
>> > >>>> > On Fri, Jun 1, 2012 at 8:48 AM, Jussi Kalliokoski
>> > >>>> > <jussi.kalliokoski@gmail.com> wrote:
>> > >>>> >>
>> > >>>> >> This issue is now pending review per
>> > >>>> >> https://dvcs.w3.org/hg/audio/rev/b78b7c5e906e .
>> > >>>> >>
>> > >>>> >>
>> > >>>> >> On Fri, Jun 1, 2012 at 6:22 PM, Jussi Kalliokoski
>> > >>>> >> <jussi.kalliokoski@gmail.com> wrote:
>> > >>>> >>>
>> > >>>> >>> Good catch, thank you! As I planned it, the timestamp should
>> have
>> > >>>> >>> been a floating point value, allowing for sub-millisecond
>> precision, but
>> > >>>> >>> actually DOMHighResTimeStamp is actually more fit fore this.
>> > >>>> >>> I will make the necessary changes to the spec.
>> > >>>> >>>
>> > >>>> >>> Cheers,
>> > >>>> >>> Jussi
>> > >>>> >>>
>> > >>>> >>>
>> > >>>> >>> On Fri, Jun 1, 2012 at 6:16 PM, Audio Working Group Issue
>> Tracker
>> > >>>> >>> <sysbot+tracker@w3.org> wrote:
>> > >>>> >>>>
>> > >>>> >>>> Audio-ISSUE-105 (MIDI timestamp resolution): timestamps in
>> MIDI
>> > >>>> >>>> should use High Resolution Time [MIDI API]
>> > >>>> >>>>
>> > >>>> >>>> http://www.w3.org/2011/audio/track/issues/105
>> > >>>> >>>>
>> > >>>> >>>> Raised by: Adam Goode
>> > >>>> >>>> On product: MIDI API
>> > >>>> >>>>
>> > >>>> >>>> The current MIDI API specifies timestamp as a long
>> representing
>> > >>>> >>>> "milliseconds from the UNIX Epoch".
>> > >>>> >>>>
>> > >>>> >>>> For MIDI applications, millisecond resolution is insufficient
>> and
>> > >>>> >>>> can cause noticeable jitter.
>> > >>>> >>>>
>> > >>>> >>>> Using absolute wallclock time is also problematic, as it is
>> subject
>> > >>>> >>>> to system clock skew.
>> > >>>> >>>>
>> > >>>> >>>> The MIDI timestamp should use High Resolution Time
>> > >>>> >>>> (DOMHighResTimeStamp), which solves these problems:
>> > >>>> >>>>
>> > >>>> >>>>
>> http://dvcs.w3.org/hg/webperf/raw-file/tip/specs/HighResolutionTime/Overview.html
>> > >>>> >>>>
>> > >>>> >>>>
>> > >>>> >>>>
>> > >>>> >>>
>> > >>>> >>
>> > >>>> >
>> > >>>>
>> > >>>>
>> > >>
>> > >
>>
>
>