07:19:57 RRSAgent has joined #webscreens 07:19:57 logging to https://www.w3.org/2019/05/24-webscreens-irc 07:20:04 Zakim has joined #webscreens 07:20:21 RRSAgent, make logs public 07:20:22 tidoust has joined #webscreens 07:20:43 Meeting: Second Screen WG/CG F2F - Day 2/2 07:20:52 Chair: Anssi 07:21:22 Agenda: https://www.w3.org/wiki/Second_Screen/Meetings/May_2019_F2F#Agenda 07:22:29 Present+ Anssi_Kostiainen, Chris_Needham, Eric_Carlson, Francois_Daoust, Louay_Bassbouss, Mark_Foltz, Peter_Thatcher 07:22:39 RRSAgent, draft minutes v2 07:22:39 I have made the request to generate https://www.w3.org/2019/05/24-webscreens-minutes.html anssik 07:24:47 Present+ Kasar_Masood 07:26:24 mfoltzgoogle has joined #webscreens 07:29:30 RRSAgent, draft minutes v2 07:29:30 I have made the request to generate https://www.w3.org/2019/05/24-webscreens-minutes.html anssik 07:32:53 scribe: tidoust 07:36:05 [agenda review] 07:37:51 anssik: more on technical topics for open screen protocol, the open screen protocol library, explainer for open screen protocol, plans for wide review, group rechartering, and upcoming F2F 07:40:48 Topic: Remaining Open Screen Protocol 1.0 issues 07:41:27 s/Topic: Remaining Open Screen Protocol 1.0 issues/Topic: General 1.0 protocol issues/ 07:41:58 cpn has joined #webscreens 07:42:24 mfoltzgoogle: type ID used by endpoints to decide which CBOR parsing code to use. 07:43:05 Louay has joined #webscreens 07:43:09 present+ Louay_Bassbouss 07:43:25 ... Also numeric identifier for requests/responses. 07:44:21 ... Third type of IDs are how endpoints correlate requests/responses. If you want to determine the availability of an endpoint, you create a "watch" with an identifier. 07:44:33 ... I want to review how these identifiers are assigned. 07:44:51 ... I don't think we have issues open for message type keys. 07:45:25 ... Let's start with message type IDs. [showing type ID structure slide] 07:46:04 ... How many bytes do we want to each type of message? 07:46:25 ... With one or two bytes, we already have a pretty big range of message types. I don't think we need to go beyond that. 07:47:02 ... The basic idea is to reserve the 1-byte key for messages that are most frequently sent. Hard to tell before we have implementations, but some are obvious. 07:47:44 ... presentation-connectionmessage is a good example, audio and video frames. One change I would make is to at remote-playback-state as well. 07:48:56 ... The majority of message types will be 2 bytes, from 64 to 8192 in the table. These are messages that should happen one or two times per session. 07:49:06 ... Note we reserve ranges each time for extensions. 07:50:06 ... Longer type IDs, let's say they are reserved for now, I don't think we'll have to bother about them any time. 07:50:22 ... The current allocation appears in the CDDL. 07:50:48 ... The spec is not explicit about the allocation scheme. 07:51:13 ... The spec should just say which ranges can be used. 07:52:44 cpn: [registry?] 07:53:07 scribenick: cpn 07:53:15 mfoltzgoogle: There are a couple more steps before we can have full support for extensions. We need a way for an endpoint to discover what extensions are supported. 07:53:46 Chris: Do we need a way for extensions to be able to register ranges of numbers for their own use, e.g., some form of registry? 07:54:01 ... I think it's important to have a registry of capabilities. We don't want two endpoints to understand extensions differently 07:54:23 ... Once you know the capabilities of the other endpoint, hopefully you can send/receive messages for the right IDs. 07:54:36 scribenick: tidoust 07:55:30 Peter: The spec currently says that if you get a type key that you don't understand, you just close the connection. 07:55:41 Eric: So you need to recompile to support extensions 07:55:57 Peter: You would need new code. Whether you recompile depends on how you implemented it. 07:56:25 mfoltzgoogle: The capabilities have to be mutually agreed. The other side has to acknowledge that it understands the capabilities. 07:56:25 Also, there may be a need to add a time synchronization protocol (e.g, similar to HbbTV companion screen) that may want to use low range message type IDs 07:56:51 Action: Chris to add a reference to the DVB / HbbTV sync message definitions 07:57:02 Peter: The endpoint will just indicate which type keys you understand. If you don't know what the capabilities mean, then you just won't send these messages. 07:58:19 Peter: We could add a section that explains how to add new types of messages. 07:58:44 mfoltzgoogle: There's a normative aspect that says that no implementation should send messages for a type key that the other endpoint has not declared support of. 07:59:43 Peter: Yes. There's no way for me to tell you that I cannot receive a certain type of message other than closing the connection. 07:59:51 anssik: OK, we should create an issue to track it. 08:00:10 mfoltzgoogle: Extensions is part of v1 scope, there may already be an issue opened. 08:01:25 ACTION:mfoltzgoogle Add remote-playback-state to the list of one-byte message type IDs. 08:01:33 [Example of an Extensibility section https://w3c.github.io/sensors/#extensibility] 08:03:01 Peter: We could reserve a range of numbers for capabilities. The sizes aren't too important, so we can reserve large numbers. 08:03:50 mfoltzgoogle: I think there's an assumption that all agents support certain types of messages. We could be pedantic and say that each message type maps to a capability. 08:03:53 RRSAgent, draft minutes v2 08:03:53 I have made the request to generate https://www.w3.org/2019/05/24-webscreens-minutes.html anssik 08:04:18 Eric: There's a basic set that you have to implement. 08:05:24 [Opened Add Extensibility section https://github.com/webscreens/openscreenprotocol/issues/175] 08:05:29 mfoltzgoogle: I think we should reserver something like a thousand capabilities for standard capabilities and reserve the rest for non standard capabilities. 08:05:40 s/reserver/reserve 08:07:58 RESOLUTION: Reserve 1-1000 for standard capability IDs. Extensions can pick numbers > 1000 08:09:03 mfoltzgoogle: We talked about length prefixing having been dropped yesterday. 08:09:47 ... We talked about an IANA registry for external (extensions) type keys. I don't know if we want to start with ours or register it at IANA. 08:10:00 anssik: Probably good to start small on our own to avoid overhead 08:10:15 mfoltzgoogle: It will probably be a file on GitHub for now. 08:10:23 anssik: That's fine. 08:11:38 mfoltzgoogle: For dictionaries, we use numeric IDs instead of keys on the wire to save bytes. In principle, we could let endpoints add additional entries to the dictionaries, but we need a way for endpoints to detail the mapping between IDs and keys. 08:12:26 ... The message has to make sense without the field being processed in principle. 08:12:39 Peter: If we have an extensibility section, we can explain how to do that there. 08:13:10 ... We didn't think of saying that fields added to regular messages should be optional. 08:13:21 mfoltzgoogle: Can you add this to the pull request? 08:13:23 Peter: Sure. 08:14:31 anssik: Right. Whoever might want to create extensions for HbbTV might want some checklist of things to be done. 08:14:49 mfoltzgoogle: Alright, that wraps up for message type IDs and fields. 08:15:00 ... Next category is request/response IDs. 08:15:48 ... Concretely, when you send a message that needs a correlated response, the request message includes a request-id, which must appear in the response. 08:16:29 ... We don't need the request-ids to be all unique across all endpoints. 08:16:50 ... But we do want to avoid collision between multiple connections between two agents. 08:18:42 Peter: 2 options. Request/Responses tied to transport. Problem is with reconnect which switches to another transport. 08:18:58 ... Other option is to include state. But what happens if an endpoint forgets to include state. 08:19:51 mfoltzgoogle: One solution is that each agent keeps track of the next request ID to use for an endpoint and increments it by 1 for each message. A request-id of 1 is a "reset your counter" message. 08:20:32 ... 1 becomes a "magic number". 08:20:46 ... Option 2 is Peter's option. 08:21:09 Peter: Basically, send a message to request state if you forgot about it. 08:21:45 ... Problem with this one is that if I forget everything, I may forget to send that reset request as well. 08:22:06 ... so we need to make this a requirement to send the request. 08:22:56 mfoltzgoogle: Basically a garbage collection mechanism. 08:23:26 Eric: Is that likely to be in the middle of a session? 08:23:36 ... I just wonder if this is a problem that really needs to be solved. 08:24:09 Peter: I can think of a restart use case. 08:24:32 Eric: After restarting, doesn't Chrome indicate that it is connecting for the first time? 08:24:41 Peter: That's basically what we're trying to specify here. 08:25:02 ... Third option is to include a state-id. 08:25:11 mfoltzgoogle: It could be a timestamp, for instance. 08:25:37 Eric: What's the initial message sent when a connection is created for the first time? 08:25:46 Peter: agent info, basically. 08:26:00 Eric: Could you include an optional field there, perhaps? 08:27:14 Peter: Right, sounds like option 3, but in agent-info, not agent-status. 08:29:11 RESOLUTION: For issue #139, add a random state token to agent-info to notify connecting agent to discard previous state (including ID counter). 08:30:31 RESOLUTION: The additional field can be named "state-token" 08:31:24 mfoltzgoogle: For completeness, we had a separate issue for watch IDs, but I think we should use a combined ID counter. 08:31:46 ... For request-ids and watch-ids. 08:32:05 Peter: The state-token automatically covers this. 08:32:52 ... I don't think we need to specify anything in the spec. Implementations can pick whatever ID they want. It will work, no need to combine the ID counters. 08:32:58 mfoltzgoogle: Fair enough. 08:33:28 anssik: Can we close the issue then? 08:33:55 mfoltzgoogle: Yes, it can be closed with the pull request for the previous issue we discussed, solved by the state-token proposal. 08:34:24 ACTION: Close Issue #145 when the pull request lands for the previous resolution (for Issue #139). 08:34:48 mfoltzgoogle: I believe that's all of the ID related issues that we had. 08:35:18 Topic: Remote Playback API Protocol Issues 08:35:56 mfoltzgoogle: Some of the remaining issues for v1. First one is separate for the next three. We need to finish the requirements for the Remote Playback API. 08:36:25 ... We did that for the Presentation API but not for the Remote Playback API. I prepared something in a pull request 08:36:41 ... PR #162. 08:37:11 ... I looked at use cases to prepare the pull request. Relatively short. 08:37:17 ... [reviewing PR] 08:37:26 https://github.com/webscreens/openscreenprotocol/pull/162 08:38:16 PR #162 HTML preview: https://pr-preview.s3.amazonaws.com/webscreens/openscreenprotocol/pull/162.html#requirements-remote-playback 08:38:41 ... If we had remoting to the scope, we may want to amend the third requirement to include media data. 08:38:51 s/we had/we add 08:39:56 Eric: What about the connectivity? LAN vs. bluetooth, etc. 08:40:17 mfoltzgoogle: This is more about what the messages need to be able to do 08:41:48 Peter: I think we do everything in the spec right now except number 6. 08:42:03 mfoltzgoogle: We do pass HTTP headers but I don't know if that's sufficient to meet the requirements. 08:42:09 Eric: No, that's not. 08:42:29 Peter: I think we would need to put the locale somewhere else. 08:42:40 anssik: How does Airplay handle locales if they are different? 08:42:44 Eric: I have no idea. 08:44:08 mfoltzgoogle: [looking at related notes in the Remote Playback API] 08:45:17 Peter: you wouldn't have different locale for one remoting and another? 08:46:07 Eric: Actually, especially for extract, there may not be encoding in your first language, but there may be in your second/third language. 08:46:30 ... We look at the options vs. the list of languages and try to make the best choice. 08:47:52 [discussion on controller/receiver language references] 08:48:05 mfoltzgoogle: Eventually, it's up to the receiver to choose the locale. 08:48:13 Peter: Is it enough to have a list of locales, then? 08:48:16 Eric: Yes. 08:49:41 RRSAgent, draft minutes v2 08:49:41 I have made the request to generate https://www.w3.org/2019/05/24-webscreens-minutes.html anssik 08:49:48 Peter: we could have that in agent-info. 08:50:12 mfoltzgoogle: That seems reasonable. I agree that this is information that is not specific to the Presentation API and Remote Playback API. 08:50:45 ACTION: Peter to add locale to agent-info and to create an agent-info event to advertise changes 08:52:35 mfoltzgoogle: The next three issues are related and future-looking. We may want a mechanism by which an application can reconnect to a remote playback started somewhere else through a remote playback ID. 08:52:43 Eric: Where does this ID come from? 08:52:52 Peter: Right now, that is not included. 08:53:41 mfoltzgoogle: Correct. One way to do this would be to expose a remote playback ID to the JS and then to add a reconnect method to the Remote Playback API. This does not exist yet. 08:55:46 [Eric describes how media playback works on iOS and Mac] 08:56:30 Peter: The distinction here is the idea that the receiver can continue playing even if the controller goes away. For Presentation API, that's in the spec. For remoting, that has not been envisioned. Is it worth trying to introduce the idea? 08:56:43 ... Does Airplay have this concept? 08:56:48 Eric: Not as far as I know. 08:57:08 Peter: It wouldn't work for remoting, obviously. 08:57:46 mfoltzgoogle: The flinging implementation could theoretically be made compatible with such a design. 08:58:41 Peter: Second related question on whether we want multiple controllers for remote playback. 09:00:24 Eric: There has to be some kind of a default behavior when the last connection drops. 09:01:36 ... You may want to start a remoting solution where you start watching something on a device and then switch to your mobile to control it. 09:01:48 Peter: We could do that. 09:02:16 Eric: It may make sense to flag this as v2 until we have more practical experience. 09:02:44 mfoltzgoogle: When the controller disconnects, it's implementation specific whether media playback continues or not. 09:03:45 anssik: Would it be possible to start a remote connection from a mobile phone, then put the phone to sleep, and then pause the playback one hour later? 09:03:51 Eric: It's implementation specific for now. 09:04:58 ... Application might get killed for various reasons. If you want to survive putting the screen off, you actually want to survive app restart. 09:05:54 Peter: As long as you remember the remote playback ID, then it would work at the user agent level. But not for the Web application. What happens if Safari gets backgrounded? 09:06:16 Eric: It depends on what the system is going to need in terms of resources. 09:06:30 ... The web app needs to store the info. 09:06:45 Peter: But it does not have a way to reconnect now. 09:07:15 ... The user agent can reconnect, but the Web app wouldn't know about it. 09:08:05 mfoltzgoogle: There is an issue opened. According to the discussion, the protocol supports it, but we don't have support at the API level. 09:10:02 Peter: If we decide that we're going to do something later, we may change the design right now. 09:10:28 mfoltzgoogle: Not to derail but I have slides for this afternoon from a colleague on multiple controllers. 09:10:43 https://github.com/w3c/remote-playback/issues/5 09:15:22 [Group discusses reconnecting ins and outs, reuse of media elements] 09:15:40 Peter: It's almost as if you would want to create a proxy but not reuse an existing thing. 09:16:08 Eric: Yes, it's a separate consideration from setting up a connection and sending full status. 09:16:53 ... Is the operation different from getting a MediaStream from a camera? 09:17:30 Peter: So you want to be able to do "new Element.srcObject = [something with ID in it]"? 09:17:32 Eric: Yes. 09:18:37 ... It would do everything but render the stream locally. 09:20:55 Peter: One discussion is bikeshed the current API for how to setup a proxy. Second is how we would want to specify the behavior. Original question is whether we want to do that at all. 09:22:02 mfoltzgoogle: The protocol for a second remote playback is going to be different from an intial remote playback. 09:22:17 ... Play/Pause/Seek should be available to secondary controllers, for sure. 09:22:55 ... What we've seen a lot from Presentation API usage is that, when controller disconnects, presentation continues. 09:23:08 ... So we'd want that to happen with remote playback as well. 09:25:01 ... If we want single controllers, we want to prevent the case where another controller can take over. 09:25:33 ... If we want multiple controllers, that's the opposite, we probably want to use GUIDs for remote playback as well. 09:25:58 ... It seems that there is support for addressing multiple controllers in the future. 09:26:12 Peter: If we want to do that later, we don't want to be stuck by the protocol. 09:26:30 mfoltzgoogle: The discussion on how to expose that at the API level would have to continue. 09:26:43 anssik: So, for protocol this would be v1. For the API, this would be v2. 09:27:13 Peter: Adding a message that allows you to reconnect to an existing one would be v2 for the protocol as well. 09:27:24 mfoltzgoogle: Yes, I'm trying to avoid scope creep. 09:28:02 RESOLUTION: Use a string GUID for remote-playback-id, similar to Presentation IDs. 09:29:11 RRSAgent, draft minutes v2 09:29:11 I have made the request to generate https://www.w3.org/2019/05/24-webscreens-minutes.html tidoust 09:29:58 Topic: Remaining streaming issues 09:31:00 Peter: I presented audio-frame, video-frame, and data-frame. But then I realized that I did not need data-frame so I removed it. 09:31:11 ... There's no text if I share a tab, right? 09:31:54 ... For remoting, you would have text, but we already have a mechanism to send text to the other side, through text track. 09:32:05 ... So it doesn't need to be in the form of a data-frame. 09:32:47 Eric: [question on remoting] 09:33:41 Peter: If we're trying to solve remoting, we can concentrate on solving text. But in the context of streaming, there is no need to stream text/data. 09:34:01 ... I'm proposing to drop generic data streams as long as we don't have a use case for them. 09:35:00 mfoltzgoogle: The main use case for data frames is when you have interactive content, e.g. an interactive display, touch screen. 09:35:18 Peter: Yes, but that's opposite direction, so you would set it up differently. 09:35:23 mfoltzgoogle: Why? 09:36:14 Peter: I don't want to try to solve a problem that we don't have for now. 09:36:48 Kasar: For sports stuff, we may have overlays in the stream. 09:37:15 Peter: If we want to put it back, we can do that. 09:37:40 Eric: The problem is not about sending text track. It's about sending arbitrary data. 09:38:27 Peter: If we have a use case, that's fine, we can put it back when needed. 09:39:21 mfoltzgoogle: The use cases do exist, they are just not documented yet. I think it's fine to postpone to v2. I'd like to see an action for someone who's interested in it to describe what they are willing to have. 09:40:04 ... Once we have use cases, we can decide whether we define specific messages or a generic data-frame. 09:40:27 ... Also, how does an application extend streaming for its own data. 09:41:09 ... A game streaming product may send map location, remote inputs, etc. along with media. 09:41:40 ... If there are APIs for these things, it makes sense to eventually consider them. 09:43:18 Peter: [difference between streaming and remoting in terms of latency/buffering optimizations] 09:43:37 mfoltzgoogle: If there are new APIs coming, we'll have to consider them in the future. 09:43:56 Peter: The solution for texttrack should be more first-class citizen than using generic data frames. 09:44:25 ... So I'll keep data-frame 09:44:42 mfoltzgoogle: I'm not sure that we have to keep the message. Extensions can define their own. 09:45:11 Peter: I think it can be good to define a message for generic data so that extensions don't need to create new ones each time. 09:45:19 mfoltzgoogle: Let's discuss this on the pull request. 09:46:22 ACTION: Peter to update pull request to keep data frames and discuss can be used to extend the streaming protocol. 09:47:09 Kaiser_ has joined #webscreens 09:52:56 Zakim has left #webscreens 09:59:58 Louay has joined #webscreens 10:00:05 present+ Louay_Bassbouss 10:02:21 kaiser has joined #webscreens 10:03:15 Topic: Open Screen Protocol Library 10:03:40 mfoltzgoogle: Goal is to provide update since TPAC 10:04:44 ... Open source implementation of the Open Screen Protocol. We intent to implement everything that is or will be in the spec. We're a bit behind, because spec moves faster. 10:05:39 ... The feature set we support: advertize/listen using mDNS, agent info, connetion to an agent with QUIC, and we support all of the presentation-* messages. 10:05:59 ... We have a demo folder to demonstrate the presentation side of the protocol. 10:06:18 ... There's also Peter's implementation written in Go. 10:06:35 anssik: Is the goal to keep evolving the two implementations in parallel? 10:06:37 Peter: Yes. 10:07:01 anssik: So two independent implementations. That's great and would be very helpful for more formal steps later on. 10:07:34 Peter: It's in the repo. osp/go 10:08:12 mfoltzgoogle: At TPAC, we were figuring out how we were going to implement these CDDL messages. 10:09:08 ... We now use a lot of CDDL features to generate code automatically. 10:09:53 ... Currently, by default, the library uses a copy of what's in Chromium for QUIC. 10:10:34 ... PresentationController, PresentationReceiver, PresentationConnection APIs exposed by the library, so hopefully easy to integrated in user agents. 10:10:53 ... Support for mDND TXT data, to make it easy to handle fingerprint. 10:11:32 Peter: QUIC is still maturing so code is not fully compatible with IETF spec for now. 10:13:05 Eric: We have our own implementation of string_view and optional 10:13:21 mfoltzgoogle: Yes, we have an open question on whether to expose that through the API as well. 10:13:37 ... I don't think we use any of those in the public API for now. 10:14:03 ... We finished adding support for IPv6 socket and multicast. 10:14:15 ... We finished our Mac porting layer. 10:15:08 ... We added support for clang as well as gcc 10:15:49 ... We decided to use std::chrono for our time library as it provides useful abstraction and access to system time in a consistent way. 10:16:15 ... We're still discussing threading. 10:16:35 ... That probably will require another few months. 10:17:52 ... We're also trying to prevent duplicate dependencies with Chromium, and also about reusing Chromium stuff instead of our own when possible (e.g. mDNS). 10:20:26 ... [reviews high level structure] 10:21:55 ... Clients may care about the platform layer (e.g. threads). We currently have platform implementations for Linux, Mac and POSIX. 10:23:29 Eric: One thing that's really painful for us to use WebRTC libraries. Because interfaces are in C++, it isn't possible to load them dynamically. Any client, whether it is going to use WebRTC or not, pays the cost of loading them to start with. 10:23:52 ... There is no standard way to lazy initialize interfaces. 10:25:17 ... If there is some way to load and initialize these things on demand, that would be a big win, and would make it much easier to adopt. That doesn't mean that the interface cannot be in C++, but it would be very helpful to be able to load it and tell it to initialize itself and find the entry points dynamically. 10:25:36 mfoltzgoogle: Got it. 10:28:10 [The group discusses possibility to include/exclude dependencies in build params] 10:30:25 mfoltzgoogle: [shows architecture]. It mostly replicates the folder structure. 10:32:39 cpn: Thinking about an HbbTV implementation where you would discovery and authentication. These would need to happen outside of the browser engine. Done at the TV layer, and then passed on to the browser layer. 10:33:02 mfoltzgoogle: What parts would be done where? 10:34:03 cpn: The browser would be responsible for actually implementing the interface exposed to the application. The TV itself would be doing the discovery part, authentication. 10:34:56 ... We may need cross-process communication 10:35:47 mfoltzgoogle: If you wanted such a model, then we would need to make sure that the embedder could run the library in two places with different starting points. You'd have to figure out how to synchronize the two. 10:36:03 ... Right now, we kind of assumed the code would be run in one place to do the whole thing. 10:36:25 ... If we did want to split the library, it should be possible in principle. 10:36:43 ... Knowing where those splits could be would help that process. 10:37:40 scribenick: cpn 10:38:23 Chris: I suspect an integrator may not be able to just take the Chromium implementation as it, they would have to separate out some parts, e.g., the discovery and authentication 10:38:44 [Some discussion on network sockets and where they are supported] 10:39:39 mfoltzgoogle: [shows PresentationController interface in C++ code] 10:39:54 ... We do request the embedder to provide some callback parameters 10:40:09 RRSAgent, draft minutes v2 10:40:09 I have made the request to generate https://www.w3.org/2019/05/24-webscreens-minutes.html tidoust 10:41:43 i/[Some discussion on network sockets and where they are supported]/scribe: tidoust/ 10:43:06 mfoltzgoogle: Depending on which protocol you support, you're going to inject different services. If you want to use the default implementation, you can just use it. Otherwise, you can inject your own implementation, and then it's up to the library to decide when these services are run. 10:43:39 ... each of these objects has an observer that gets notified when the state changes. 10:44:45 Eric: Back to lazy loading, it's harder to look for these things yourself. Wrapping this functionality in factory functions makes it considerably easier. 10:47:37 RRSAgent, draft minutes v2 10:47:37 I have made the request to generate https://www.w3.org/2019/05/24-webscreens-minutes.html tidoust 11:39:42 scribenick: cpn 11:41:55 Mark: We can generate code to parse messages 11:41:59 Louay has joined #webscreens 11:42:47 ... [shows example of CDDL translated to a C++ struct] 11:43:29 ... There's a platform API, based on C++ interfaces, high level methods to create and bind sockets, join multicast groups, 11:43:35 ... send and receive datagrams, etc 11:44:02 ... For the next few months, we want to implement auth messages and TLS support 11:44:31 ... We'll add an API for remote playback, and messages; also streaming 11:44:59 ... We're integrating with Chrome, and hope to make available in the browser behind a flag 11:45:18 ... By end of year 11:45:53 ... You can contribute to the code. There's a CI system to build and test 11:47:31 Topic: Presentation and Remote Playback APIs: Proposals for enhancements 11:47:42 Peter: There are three ideas for changing the APIs 11:48:15 ... Some things we want to improve. We have two APIs, but if a web app wants to support both, you need two buttons 11:48:25 ... Also synchronized media in the Presentation API 11:48:51 ... Also MSE streaming in remote playback 11:49:44 ... Looking at MSE streaming in remote playback. If the browser is doing the forwarding, what the app puts in the buffer isn't suitable for the remote playback device, you may need to transcode. We want to avoid that 11:50:02 ... Proposal to report capabilities of the remote playback device via the API 11:50:31 ... Add a RemotePlaybackCapabilities interface - for display resolution, codec support 11:50:39 Anssi: Is this Media Capabilities, or something more specific? 11:50:53 Peter: This would give a dynamic set of capabilities that could change at any time 11:51:17 ... This could be problematic from a fingerprinting point of view 11:51:19 Mark: When would this attribute be populated? 11:51:54 Peter: In OSP, it's after you get the response from the remote side. So the API discussion is tied to the protocol discussion 11:55:16 cpn_ has joined #webscreens 11:55:32 scribenick: cpn_ 11:55:50 Peter: Yes 11:55:53 Mark: In Media Capabilities, the application asks the platform what media it can support 11:57:16 Eric: The difference is that the answer tells the controller which is its preferred format, rather than indications of smooth and power efficient, etc 11:57:40 ... If we go with this kind of API, do we want the receiver to be able to change, if conditions change? For example, if the bandwidth available changes 11:58:10 Peter: The controller would know about the bandwith 11:58:56 ... A video in picture in picture, could move to a smaller size rendering to save bandwidth 11:59:43 Eric: It's the size of the rendering layer that's important 12:00:20 Peter: We could follow the Media Capabilities model, but would need extending to give extra information 12:00:25 ... There's nothing there about screen resolution 12:00:37 Mark: It's distinct from media queries 12:01:18 Chris: The explainer talks about the relationship between Media Capabilities and the Screen interface 12:02:20 Peter: You'd want to know about screen capabilities such as HDR when deciding what to request 12:02:33 Eric: You'd need Key system, codec type, size 12:03:53 Peter: Eric, you were interested in the receiver picking, to help with fingerprinting. Is there still a fingerprinting concern if we model it on Media Capabilities? 12:03:54 Eric: Sure, the less information we expose, the better 12:04:47 ... If there's a good reason to make the decision on the controller size, we should discuss it. But it seems the remote device is the best place to do that, it has all the information 12:05:57 Peter: What would app developers actually use? 12:06:04 Eric: The app developer would just have to request the one selected by the receiver 12:06:51 ... It's also more consistent, as the logic for which one to pick is the same from app to app 12:07:40 Anssi: So the controller presents the available options and the receiver selects one? 12:07:40 Eric: Yes 12:08:12 Anssi: Can we use the source element approach? 12:08:40 Eric: The controller knows more about the sources 12:08:59 Peter: It's MSE using applications that will use this 12:09:17 Mark: In this case, it's the application that selects 12:10:06 ... Those applications are now being directed to use Media Capabilities. To extend that to the remote playback case, it would make sense to use the MediaConfiguration 12:10:48 Peter: instead of querying by format one by one, you could pass in a list 12:10:58 Mark: Do we need to extend the protocol to support the MediaConfiguration dictionary? 12:11:19 Eric: Yes, and offer and an answer 12:12:22 ... The message could be a serialization of VideoConfiguration and AudioConfiguration 12:12:25 s/.../Peter:/ 12:16:51 cpn has joined #webscreens 12:16:54 Action: Eric to people at Apple about which of these options makes more sense: offer/answer vs exposing media capabilities 12:17:03 scribenick: cpn 12:18:40 ... [Discussion of when the information is needed, in relation to the prompt to the user] 12:20:21 Peter: What happens during a prompt? 12:20:21 Mark: If the application has the choice of flushing the buffer, rather than playing out the content, it can decide 12:23:12 Peter: Option to look at what's been buffered, or look at duration 12:23:21 Eric: Another option could be to have a poll model where the remote indicates when its ready to receive more data 12:23:47 ... It could look at latency between controller and receiver, and space available in its local buffer 12:25:09 Peter: Perhaps we could have a RemotingState object and an onremotingstatechange event 12:25:20 Mark: The MSE spec describes a HAVE_ENOUGH_DATA state 12:25:30 RRSAgent, draft minutes v2 12:25:30 I have made the request to generate https://www.w3.org/2019/05/24-webscreens-minutes.html anssik 12:25:36 Peter: This is for ensuring smoothness due to buffer underrun 12:27:23 ... With option A, the application needs to diff the buffers to decide what to download. In option C, it's simpler 12:27:42 Eric: It's subtly different, there are two states: how much you have buffered and the state of the receiver 12:27:59 ... A problem with option A is that the controller doesn't have any idea of how much the receiver wants to have buffered 12:28:55 Peter: What is the browser's job in remoting mode? It could copy the local buffer to the remote buffer 12:29:05 Eric: I think we need input from MSE people 12:29:23 Mark: Are the buffering algorithms going to be different between local and remote playback modes? 12:29:44 ... Then there's the congestion control difference between controller and server, and controller and receiver 12:30:26 Eric: If it works as a poll mode, it makes it easier for the app on the controller, which is responsible for making sure it has data when the receiver needs it 12:30:57 Peter: Our Chromecast receivers are either showing a web page, Presentation API, or streaming and then discarding the data 12:31:57 Eric: From a low level implementation perspective, we use AV foundation objects. You give this a lambda and a dispatch queue, and it calls the lambda when it wants more data 12:32:36 ... In the case of a stream coming from a camera, i feed the data into a queue, then the lambda on a different thread, then another thread pushing it down to the display code 12:33:54 ... At the lowest level, it's looking at the clock and at the timestamps. This is the thing that knows how fast it needs data 12:34:04 Peter: This is like "has enough" in MSE. I think we need a third state, "has too much" 12:34:45 Eric: In MSE today, it's the responsibility of the app to do that, using a heuristic that doesn't cause the browser to run out of memory 12:35:04 ... So we need a mechanism to deal with the back pressure 12:35:16 ... Could lead to throwing data away 12:36:42 Peter: So it's not just networking, it's the network and the receiver buffer 12:36:42 Eric: Yes, the receiver is in a place to adjust the frequency at which it makes requests 12:38:01 cpn_ has joined #webscreens 12:38:11 scribenick: cpn_ 12:38:17 ... So options A and B don't work, if we want to allow remote receivers to push back 12:38:20 Eric: There may be other options 12:38:25 Mark: Could use the readyState, e.g., to add "have too much" 12:38:33 Eric: Or does "have enough" mean also "don't send any more"? 12:39:19 ... I think option C is a good starting point 12:41:19 Action: Peter to make the buffer and ready state proposal more concrete, and update the protocol to match 12:41:40 Mark: I think there are open questions about the API. When would the application produce the offer? 12:43:09 Eric: We should think about how this works for a MediaStream as well 12:43:50 ... For a local stream I could make an offer, get a reply, then push frames 12:44:14 Peter: This is like the streaming case 12:44:28 Eric: The simpler we can make the API, the more successful it'll be 12:45:40 Peter: We could provide an API where you give a stream and the browser decides what to do (e.g, encoding) 12:46:00 Eric: Hopefully that would be as similar to this as possible. 12:47:11 Peter: When connected to a stream, the only part of the media element that's used is the rendering 12:47:23 ... Could simplify further by not requiring a media element 12:47:54 Eric: For a MediaStream, the app doesn't have access to the frames, so you just need a mechanism to negotiate the format 12:48:45 ... We have a similar situation as with MSE, where the conditions at the controller (optimal size and frame rate) aren't the same as at the receiver 12:49:22 ... Could configure on the media stream Track, most efficient to do that as it's being captured, so you want the optimum configuration information 12:49:58 Mark: If you want to base the capture constraints on the receiver capabilities, need to do that when initiating capture 12:50:09 RRSAgent, draft minutes v2 12:50:09 I have made the request to generate https://www.w3.org/2019/05/24-webscreens-minutes.html anssik 12:51:10 Peter: This is a cool idea, supporting streaming, maybe for v2 12:52:30 Mark: This could be a new separate API, or an extension to the Presentation API, to be determined 12:53:24 Topic: Issue 173 12:53:57 s/Topic: Issue 173/Topic: Revisit adding support for Remoting in OSP/ 12:54:29 [reviewing https://github.com/webscreens/openscreenprotocol/pull/173 on a big screen] 12:54:55 Peter: This is modelled on the receiver advertising capabilities, but now looking to move to offer based 12:55:10 ... Also add information on buffer state 12:58:58 Mark: we can use readyState 12:58:58 Peter: With an additional "have too much data" value 12:59:40 Resolution: Use the HTMLMediaElement's readyState with an additional HAVE_TOO_MUCH state 13:01:55 Topic: Local Presentation Mode 13:02:52 Mark: There's an explainer. Today, the presentation created is in its own separate context 13:03:26 ... If it needs access to cookies, or want to use parts of the original page (copy DOM content), would need to serialize to a message 13:03:33 ... People want interactive elements across controller and receiver pages 13:03:59 ... For applications like presentations, visualisations, this makes the presentation API harder to use 13:04:20 ... Proposal to create a mode where you create a presentation more like a child of the controlling page, per window.open() 13:04:39 ... That child can then be mirrored to another display 13:04:49 ... This would work for applications like slides, with speaker notes and controls 13:05:06 Eric: Mirroring as in the bits, or as in the live document 13:05:10 Mark: The bits 13:05:23 s/document/document?/ 13:05:39 Mark: Added a feature to the Presentation API to request this mode 13:06:05 ... Feedback was adding a new mode was preferable to adding options 13:06:15 Eric: How does the application know how big to make the window? 13:06:35 Mark: Currently it doesn't, we'd want to size the window accoridng to the size of the target display 13:07:47 ... Adds a dictionary to PresentationRequest with isLocal flag 13:08:12 ... The request has an attribute that reflect the options set on the request 13:08:27 ... The start promise returns a connection or a Window object 13:08:58 ... Developers are asking us for this, may want to run an origin trial to get feedback 13:10:19 Peter: So from a developer perspective it's like window.open() with a display picker 13:11:18 -> https://github.com/mfoltzgoogle/local-presentation-api/blob/gh-pages/explainer.md Local Presentation Mode explainer 13:11:37 Topic: Synced media in Presentation API 13:11:40 mfoltzgoogle has joined #webscreens 13:12:03 -> https://github.com/takumif/presentation_api_synced_media/blob/master/explainer.md Synced media in Presentation API 13:13:34 Topic: One API for starting sessions 13:13:51 Mark: We currently have two APIs, difficult for applications 13:14:30 ... An idea we had was to allow a PresentationRequest to take a sourceMedia attribute to allow section between a presentation and remote playback 13:14:53 ... The user would see a combined list of devices 13:15:12 ... The start() would resolve with a presentation connection or a null for remote playback 13:16:26 ... The remote playback API would work in the same way, the app would get remote playback events, just as if they'd chosen prompt() 13:16:40 ... [shows example sender code] 13:24:46 cpn has joined #webscreens 13:24:51 scribenick: cpn 13:24:54 Eric: In this case, if I pick something that goes to remote playback, what happens with the URL in the PresentationRequest? 13:24:58 Peter: It's not used 13:25:21 Eric: I don't like this, seems we're trying to retrofit onto the existing API 13:27:32 ... Something we were talking about last week was the setSyncId. As written in the current spec, enumerateDevices is supposed to include both input and output devices. By default, with no prompt, the user can get UUIDs for all devices. It's an information leak. We need the capability for WebRTC on the phone where a script can request that the output from a MediaStream goes to a headset instead of the speaker, depending on user choice. 13:28:04 ... We looked at PresentationRequest to choose how to route the request without exposing information. Seems like the same problem, maybe there's a more general purpose API that could be used to do both things 13:29:09 Peter: [Shows example code with remoteDevice prompt] 13:30:04 Eric: If we try to make it also work for picking an audio output device, wouldn't pass array, as those are different to presentation devices 13:30:31 Anssi: This seems to need more thought 13:30:47 Mark: Maybe there's a generic device selection API? Worth talking with Device and Sensors people 13:32:18 Peter: [Shows example with capability based selection] 13:32:49 Resolution: Explore a more generic device selection mechanism 13:33:51 Topic: OSP Explainer 13:34:11 - > https://github.com/webscreens/openscreenprotocol/blob/explainer/explainer.md Explainer for Open Screen Protocol 13:34:36 Mark: Explainer talks about the goals, enourage wider adoption of open protocol 13:35:11 ... It describes alternatives, such as existing protocols like HbbTV and cast, and why they may not be the ideal starting points 13:35:23 ... It's not an API, so how do I convey this to the TAG? 13:35:42 ... I was planning to show an example mapping to the API, without too much detail 13:36:42 Anssi: Could invite TAG members at TPAC 13:36:49 Mark: What's the TAG's turnaround time? 13:37:00 Anssi: Depends on who is assigned 13:37:23 -> https://github.com/w3ctag/design-reviews/issues/240 The web platform needs a service discovery mechanism 13:37:24 Mark: Also related to service discovery 13:37:34 Eric: Tess is interested in that 13:38:24 Mark: For the breakout, we should ask people to familiarise with our work so we can have a productive discussion 13:38:56 ... For planning, should we include streaming and remoting in scope for V1? 13:39:13 ... I'm happy to include it 13:40:45 RESOLUTION: Add streaming and remoting in scope of v1 OSP, conditional to CG charter update 13:42:53 cpn_ has joined #webscreens 13:42:58 scribenick: cpn_ 13:43:21 Anssi: We should have a draft charter ready for TPAC 13:43:58 ... Then in October, have a paper to go for approval 13:44:29 Mark: Two things: standards process for OSP, and what are the main API features we want to include in scope of next charter 13:46:15 Topic: Wrap up 13:46:29 Anssi: Thank you everyone for attending 13:47:11 Mark: We'll update and share the slides 13:47:19 RRSAgent, draft minutes v2 13:47:19 I have made the request to generate https://www.w3.org/2019/05/24-webscreens-minutes.html anssik 13:47:58 [adjourned]