Meeting minutes
scribe magic on
Raw Media Insertable Streams Status Report
Raw Media Insertable streams discussion
question from Bernard: anything we could do IN THE SPEC to improve performance?
HTA: not really. We are working on it and we will likely come back when we have more data. let's talk about that next month.
Beyond Insertable stream (breaking box)
Encoded data sits in the middle of the media engine (encoder - media transport coupling) and thus cuts the media path and the feedback path, which is not the case for non encoded data going into the encoder.
both media and feedback should be rewired.
That raises question about signalling as well.
One of the challenge will be to designs Api and objects, so that there are some objects accessible from JS and C++ alike.
Tim: I like the capacity it brings, but I'm nervous about the SDP part of it.
B.O.: you can not do an SDP O/A, then in JS decide do do an extra encryption step and expect the other side to just know about it.
Tim: like DC just tag the DC with the name of the protocolled tunnelled .....
HTA: it will not work if you need some kind of interoperability or exchange. For example for E2EE you need to exchange keys. Everything negotiated will need to be in SDP.
JIB: assumption 1: we need a feedback channel.
JIB: the example is nice (biter buffer in JS) but is there any other application for that?
HTA: once we have this design in place switching media transport (taking out RTP) is trivial. Moreover if you want to reuse webcodec, you need both raw frame and encoded frame APIs and chain able property. I made a demo code actually.
JIB: I still find it hard to comment. It s very .....
HTA: lose, yeah .....
JIB: also the naming (track and stream) is slightly confusing
JIB: It still feels like side-wire. Also, we are breaking up the objects and it s not sure which object will use it. Also there is a mixing of raw data and encode data ...
HTA: i want to re-use the object definition from webcodec.
JIB: why a stream, and not functions?
HTA: i don t like functions because they have return value. while streams feels more unidirectional.
but ... I m no t(100%) sure it s the best way.
JIB: it looks like this is more a setting up API, and I am also interested in the Feedback part which is not defined here.
HTA: oh yes, there will be a lot of work to be done.
JIB: I'm not sure I'm sold just yet.
TIM: let s not forget abut webaudio
… it could be quite clumsy
HTA: interestingly, webcodec and webaudio have the same buffer representation, so this idea is to keep things aligned, and i m hoping we can connect it all with a little bit of elegance.
… i want to write more demo to illustrate it
insertable stream as a Transform
(off main thread)
by Youenn Fablet
next step proposal
(s): 3 of them.
JIB: I like the approach, I like the API, it makes it easy to make the right thing.
Separation between data and model is nice.
JIB: I amin favour of proposal 1 and 2
HTA: I like proposal 2.
… I m nervous about the number of ports available as we might need 4 ports to handle in/out, media and feedback.
… we might want to have something else than transform.
… I agree with proposal 3: I think we should continue working on APIs.
… especially, we need an handle on the feedback control
… so yes to native transform, hesitant to adopt the transform API.
YF: re: transform having only two ports. In one example, we have a readable gater and a writable, that gives you 2 ports. We could add a readableControl / writableCOntrol and get the 2 additional ports.
… in my point of view it is orthogonal to proposal 2, and that s why I'm not nervous about adopting a transform attribute.
JIB: what I like about the transform is that it seems easy to do the right thing, and hard to do the bad part.
… i m worried that the plumbing of insertable streams is more dangerous.
… so i want 1 AND 2
Henrik: I like this, but it s hard to put it in perspective. How would you that today without this API?
YF: you would get a readable / writable stream, you would create a worker, pass it, then yo would have to do the pipe dance, ...
… I have an example. Here you do not need to do all of that.
… The transform allows you to carry all the internal objects around easily.
Henrik: I like it.
JIB: you expose readable/writeable only in the worker, and not in the main thread?
YF: that s one way, or we can use events.
JIB: ideally I believe we should not expose readabe/writable to the main thread, or people will use it and it will be terrible
TIM: webAudio can have multiple observer/output to a given filter. Here in this model this is one input one output model. It would be a shame if it would limit webaudio.
YF: if the frames are "clonal" then you can do that, it s up to the javascript.
JIB: if I assign the same sender to two transform.
YF: current implementation it would fails.
JIB: is the transform reusable after the streams has Stopped?
YF: not currently. I don t see a use case for it, and it s easier to implement.
YF: do we get consensus?
JIB: I m hearing support.
HTA: certainly support to move forward to get a more specific proposal (PR)
Resolution: consensus to move forward with a pull request on Proposals 1, 2 on issue #48
NOTE CONSENSUS to move forward to get a PR for proposal 1 and 2, and wait for more info on proposal 3
Cropping MST
Henrik:
… I like to be able to get a track which content is exactly what I want to be
… however, why is this a separate step? You are capturing it in one step, then cropping. Why not using the cropped element as a source of the capturer.
… that would remove an intermediate step.
Elad_Alon: I thought it would be user friendly the proposed way.
… also, if the application want to change the crop area during the capture you culd
JIB: the capture API is not fully spec'ed yet so we cannot base things on it
… then what is the use case? Over capture?
… we should think about all the possible source this API would impact, and i m not sure how the DIV coordinate would apply to some source
… I believe we should limit this API to some source type
… we tried in the past to do exactly that for screen capture.
… MEdiaCapture not being spec'ed let s solve at the point of capture.
… which is what henrik said, and it also solves the div coordinates management
Elad: I feel like you raised 4 points.
… point 1 ...
… point 2: we would still want to be able to capture an HTML element. There is a consent problem.
JIB: if the core problem is over capture, we can solve it without cropping, and we should.
Elad: I assume the user would provide consent.
HTA: we have to be careful about persistent permission and permission duration
JIB: please note i do not support the proposal at this time.
elad: i do not have a full green light from chrome security time too so this is not for decision at this time.
YF: We should have a more clearly define Use Case.
Bernard: any consensus (as we are running out of time) ?
DOM: and what is the right place to follow up on this.
DOM: open a ticket in github please.
Elad: will do.
Bernard: see you in feb.