SV_MEETING_TITLE -- 27 Mar 2013

Web Midi

<olivier> ScribeNick: olivier

cwilso: introduction to MIDI, history, etc

… very simple message format with built-in extensibility

… to a lot of users MIDI became syn with standard midi file

… exciting part really is about connected controllers to computers

… used to control large systems too, lighting, etc

cwilso: looking at Web MIDI API. pretty simple

… ask for access, some privacy/security implications

… enumerate MIDI devices, get access to each of them

… MIDI connections are unidirectional

… input and output to a given controller will show up as two connections

… the actual data is a sequence of bytes

… MIDI has a well defined message syntax

… that's all the API does - passes messages

… MIDI is both hardware and software protocol. our api has nothing to do with the hardware protocol. We're mostly working with USB-MIDI devices here

… wrote a polyfill to implement using the Jazz MIDI plugin - https://github.com/cwilso/WebMIDIAPIShim

gmandyam: question about android and what would need to be implemented

cwilso: you wouldn't have to build the whole audio infrastructure.
... I think it will be implemented in android, although time is a more difficult question

… implementation would be different on osx, windows, linux and any other os we want to ship on

… fairly similar MIDI APIs

… at least similar in shape

cwilso: polyfill lets you try out the API but requires the jazz plugin to function

- demos -

cwilso: looking at webaudio made me realise how easy it would be to build synthesisers

… controllers are fairly cheap and easy to find

shepazu: why would they build a web app rather than native?

cwilso: I'd ask the opposite question

… web platform means that you can ship on all OSes

… and get a more natural hook in social/cloud platforms

shepazu: have you built something that would send the signal to XHR

cwilso: there is a community in Japan, someone built something like that using web sockets to exchange MIDI messages

crogers: jam with chrome app (shown at SXSW) does that kind of collaborative, online music

olivier: what do we still need to do

… the big bits of work that need doing

cwilso: real time messages need to be clarified

… virtual MIDI ports (powerful but complicated)

… some question about constructor - mostly for testing

… plus some minor renaming issues

cwilso: would like people to test the API, use the polyfill

… biggest challenge is to get understanding across vendors on whether this is useful

… went to NAMM - huge industry event

… every 3rd or 4th booth was using MIDI in some way

… also went to MMA (Midi Manufacturer's Association) and one of the topic was the Web MIDI API

cwilso: also working on native prototype implementation

… we need more than one implementation

… really needs the web audio API also implemented to make sense

ehsan: don't think anyone at Mozilla has reviewed the spec

… would rather avoid similar situation to web audio API where people look at it when it is already very mature

… realistically I don't think anyone at moz will work on it before we are done with webaudio

cwilso: feedback from people building on top of the API is good

[discussion on missing system level MIDI support on Android]

Koji: even without web audio api, it would be valuable to us

… to replace native software

… would like to know what we can do to make this happen

shepazu: lend resources to e.g mozilla?

cwilso: not necessarily, bottleneck is to get it reviewed first

takashi: btw Korg also interested in virtual MIDI port

koji: in the future yes

… currently we have latency in web audio API, so can't start with software synths

Testing

<chrislowis> olivier: we spoke to Tobie last week about the goals for testing.

<chrislowis> Our goal in the end is to have interoperable implementations.

<chrislowis> Firstly, we want to use tests to improve the spec - that is, when you have an assertion in the spec can you write a test for it, and if not can you improve the wording of the spec to make this more explicit.

<chrislowis> Secondly, to check whether the spec is implementable.

<chrislowis> Thridly, to check whether the implementations actually follow the spec.

<gmandyam> Check out http://dev.w3.org/geo/api/test-suite/ for an example of a test suite derived from the spec requirements (in this case Geolocation API)

<chrislowis> olivier: Fourthly, to test whether our spec interoperates correctly with the rest of the web platform.

<gmandyam> +q

<chrislowis> olivier: Fithly to test the integration with the rest of the web platform.

<chrislowis> olivier: and finally to look at regressions.

<chrislowis> olivier: But we should concentrate on the first three, according to Tobie.

<chrislowis> shepazu: we're going to be looking at browsers, but we shouldn't forget authoring tools.

<mdjp> chrislowis - copy out an existing test from webkit/gecko suite - verify offlineAudioContext in w3c suite.

Guest session

<chrislowis> playcanvas

<chrislowis> scira

Jory: talking about interactions with game dev.

… discussing game dev engines

<chrislowis> (^^ libraries for developing games which support web audio)

<chrislowis> Three.js.

<chrislowis> jory: these are games engines, not audio middle ware.

Eric & Neil from Khronos - talking about relationship between OpenSL and Web Audio API

Erik: mapping the two - 90% overlap. Also: overlap in use cases

… discussion on whether to build one upon the other

… we abstract away the node/graph approach

… giving access to effects and control

Crogers: explains a little more about history of how the web audio API architecture came to b

s/b$/be/

olivier: asking about ease of porting from one to the other

Erik: managed to port some code for one of our use cases in a day - mostly spent looking at javascript, as I am not a JS developer

Ehsan: any missing feature ?

Erik: if I were to do webSL I would look at where the specs do not overlap and get to full coverage
... notes there has been demand for a web version of OpenSL ES

(there was an interesting note that node graph is harder to accelerate, with considerations for Mobile platforms)

General agreement that building OpenSL on top of Web audio API would be a good thing ™

olivier: the only potential issue is if we don't coordinate well on communication, explain that the two would be complementary

crogers: asking about exposing OpenCL to the web

<chrislowis> scribeNick: chrislowis

further discussion on issues with the spec.

We're going to discuss prefixes.

ehsan: currently webkit, as used by chrome supports the most recent version of the spec, and also some of the deprecated API methods (start and stop for example).
... in Gecko I've been resisting implementing the deprecated names. If I do, there won't be any incentive for developers to implement those names.
... I've had people coming to me recently and saying that "ff doesn't support web audio" when people test HTML5 rocks tutorials, for examples, as they use the prefix.
... as a result, I think it will be a problem for the adoption of the spec for developers who are not interested in a vendor-specific solution.

olivier: one idea is for implementors to warn the user (via the console for example) if they are using deprecated methods.

crogers: there's a lot of code out there that would break today if there were no noteOn or noteOff. And most people use the noteOn/noteOff as safari doesn't implement it.

jernoble: is there a way to turn off the old APIs programatically in the webkit code?

crogers: yes, but even Chrome hasn't flipped that switch yet as we want to give people a while for developers to implement it.

ehsan: one issue here is that the current spec suggests that one "should consider" implementing the deprecated names.

shepazu: one way that people have dealt with this is to have a script loibrary that aliases the names.

cwilson: I wrote a monkey-patch library that takes calls to the deprecated names and aliases them. And the opposite.

(https://github.com/cwilso/AudioContext-MonkeyPatch)

crogers: I think there has to be a deprecation note, for historical interest, but we should take the statement out about recommending the deprecation.

RESOLUTION change the deprecation text to remove the recommendation to implement the deprecated methods

ISSUE: change the deprecation text to remove the recommendation to implement the deprecated methods

<trackbot> Created ISSUE-108 - Change the deprecation text to remove the recommendation to implement the deprecated methods; please complete additional details at <http://www.w3.org/2011/audio/track/issues/108/edit>.

jory: we need a way to communicate to developers these changes, my blog, or on the list etc.
... also deprecation notices (in the console) are really useful in this context.

crogers: we need a nuanced message, as we still need noteOn() to work in safari.

vendor prefixes

ehsan: browser vendors used to use prefixes for whatever CSS functionality they wanted to implement.
... I don't know what the webkit policy about DOM object vendor prefixes is. I'd like to urge webkit into un-prefixing as soon as possible, and not implement the deprecated methods on the unprefixed first.

crogers: the problem is that a lot of developers check for the existance of AudioContext first, then fall back to webkitAudioContext if not. So if we don't implement the methods in the unprefixed version, pages will break.

cwilson: I don't think webkit have promised to never removed prefixed versions. I think there's consensus that prefixing isn't a great idea in general. I'd like to see us remove prefixing as soon as possible.
... but we'd used the prefix in the past to indicate that something is experimental.

crogers: I think we'd support the prefixed version for a period of time.

cwilson: there's another task on me to go heavily evangilise how to write web audio api applications, and how to write them.

olivier: we've been talking about articles, but we haven't talked about whether we should put out software.

ehsan: I don't know if unprefixing is a solution to this problem.
... the reason I care about this is that webkit not removing the prefixes will hurt the adoption of a cross-platform web audio api.
... from my perspective if webkit can't change the policy it is much better to support the old names.
... we need to be pragmatic, and I'm not sure if there will be any short-term solutions.
... I'm talking at broad levels. When we want to decide exactly what to do we will need a deeper discussion.

crogers: one possibility is to implement the deprecated methods in ff, and then agree in few months time to simultaniously start issuing strict deprecation messages.

olivier: is there a lot of code out there that isn't a tech demo that uses these methods?

crogers: yes, absolutely. (cites Angry birds. Jory's ibooks). It would be a breaking change.

ehsan: I appreciate that it is very difficult. In Gecko we still support document.all for example, despite a decade of evangilising.

crogers: I don't understand why supporting noteOn and start is such a bad solution.

ehsan: because we're exposing the same functionality with two names.

crogers: and what's the downside of that?

olivier: I can answer that. There are specs in w3c land and beyond that have two ways of doing the same thing. One example is ttml. The criticism I've heard is that they're too many ways of doing the same thing.

crogers: I think there's already some confusion, but I think thats because safari doesn't support start(). Once we're over that hurdle we can heavily evangalise, and developers are smart, they'll understand that there's a new name for something.
... we could change "deprecation" to "alternate name" or something.

ehsan: we could do that as a partial interface.

olivier: am I hearing a consensus?

crogers: I think we're going to require the implementation of the alternate names.

joe: I want to check, is it possible to know that if "start" is present that it will accept those other argument.

ehsan: start is going to have the new interface, and noteOn will have the old arguments.

RESOLUTION: We'll support the two interfaces in the spec, have the noteOn etc in a separate section. Change the deprecation methods to "alternate names" and explain that they exist in the spec for historical reasons.

<olivier> https://www.w3.org/Bugs/Public/show_bug.cgi?id=17415

Web workers

crogers: now that scriptProcessorNode has been out for a while, I've noticed that a lot of developers choose a buffer size that is too small.

<ehsan> http://www.w3.org/TR/streamproc/#stream-mixing-and-processing

crogers: we should look at roc's proposal for how we the proposed API for webworks worked there.

(§4.4 Worker Processing)

And example number 9. (§9.9)

-- document.getElementById("out").src = new ProcessedMediaStream(new Worker("synthesizer.js")); --

<ehsan> http://people.mozilla.org/~roc/stream-demos/worker-generation.html

We're looking at the example code above.

crogers: You'd have something like "new ProcessedMediaStream(worker, 44100, 1);" in the example, but for createScriptProcessor.
... I don't like the idea of different sample rates for each worker.
... seems overly complex.

Something like createScriptProcessor(worker) and you'd pass that back to the buffer.

crogers: you might even implement it first in Gecko, ehsan?

ehsan: it's not entirely unpleasant to work with webworkers in gecko :)

joe: the event loop is very similar to what happens in flash.
... does this deliver performance improvements?

crogers: I think this would yes, as it takes the audio out of the main thread so that it is protected from DOM relayouts etc.

olivier: can we summarise this discussion in bugzilla?

(https://www.w3.org/Bugs/Public/show_bug.cgi?id=17415)

<olivier> RESOLUTION: we will allow the use of ScriptProcessorNode with Workers - see https://www.w3.org/Bugs/Public/show_bug.cgi?id=17415#c94

ehsan: I'll create an issue to discuss this further

crogers: notes that this doesn't solve the latency issue. That's a hard problem that we may never solve, but I'd love to see the ability to compile code using llvm into a node for zero-latency script processing.

jameswei: (a question about communication between threads)

crogers: the inter-thread communication isn't with javascript, it happens at a lower level.

<ehsan> https://www.w3.org/Bugs/Public/show_bug.cgi?id=21417

crogers: but it looks very nice from a JS level.

next call date

RESOLUTION: next call on the 25th April 2013 at the usual time.

We'll aim to have all the issues raised at this f2f in bugzilla, and crogers will try to address them all. Agenda of the next call will be to go through thse and make sure we're happy with them.

TPAC

olivier: we don't have a date for TPAC yet as it has changed. Might be good to have a presence there even if we don't have a meeting.
... once we have a date we can discuss how to do some developer advocacy there.
... just let us know if you know you won't be able to attend.

Meeting adjorned.

- DRAFT -

SV_MEETING_TITLE

27 Mar 2013

Attendees

Contents