Meeting minutes
Welcome
anssik: Welcome to Second Screen WG F2F!
… We'll do a fast intro round today
Anssi: co-chair.
Louay: co-chair
Francois: Staff contact
Ryo: Work for NHK, public broadcaster.
Brad: Google, mainly working on fullscreen purposes, closely with Mike
Mike: Google Chrome, web rendering topic, editor of Window Management API.
Fritz: Part of the Fraunhofer team with Louay and implemented tests
Hakan: Google. Work with Mike and Brad.
Juha: Intel, same team as Anssi.
Michiel: Interested in following up with topics we discussed in the past.
Sanket: I work on Microsoft Edge
… New incubation proposal.
Sonja: Google, working on Window management [microphone not working]
Mark: Google. Editor of Presentation API, and Open Screen Protocol. Also work on Remote Playback API, and Screen Capture.
anssik: Everyone fine with the agenda? I did some minor adjustments
… goes through the agenda
… For Remote Playback API, we concluded that there is no high priority topic.
anssik: Any comment on the agenda?
Window Management
Repository: w3c/window-management
Creating Fullscreen Popup Windows
Mike: I'll present some slides and Brad and I can go back and forth on some topics.
… Some ongoing projects.
Mike: First topic is most pressing for this group: creating fullscreen popup windows.
Brad: Introduced briefly during last Second Screen WG meeting.
… cumbersome UX to show fullscreen content on a specific display.
… Need a user gesture, move window.
… We're trying to streamline this process.
… We iterated on a few proposals, but ended up on a fullscreen feature gated on window-management permission tied to window.open.
… It's backward compatible.
… You see an example on screen.
… One of the advantages that we found is that, even on single screen contexts, this is useful to open fullscreen windows.
… More accessible.
Brad: I'll show a quick demo.
… Two virtual screens.
… This example launches the window fullscreen on the second screen.
… You have to provide the right screen coordinates and magic happens.
… Origin trial in 119. We've been getting TAG feedback.
… We recently merged the explainer in the GitHub repository.
… We're looking for Second Screen WG/CG feedback.
… Happy for everyone to try the demo.
mfoltzgoogle: How does the application choose which display fullscreen popup will be fullscreened on?
Brad: The application gets details on available screen, to target the right screen, application needs to target a position in that screen to open the window fullscreen on that screen.
… It's really small in the demo, but I'm basically typing the x coordinate.
mfoltzgoogle: It's based on the origin of the popup.
Brad: Yes.
<Zakim> anssik, you wanted to ask about error signalling
anssik: Question about the error part. How would the developer deal with the error condition?
… If popup cannot be opened for some reason.
Brad: Under the hoods, still using the usual Fullscreen path, so same errors.
… The only downside is that, to make things backwards compatible, we didn't want to throw exceptions.
anssik: Can this be polyfilled, even in a very bad way?
Brad: The whole fullscreen transition cannot be fullfilled. If fullscreen is not implemented, the user will still see a popup, it just won't be fullscreen, and site will have to have the user click some fullscreen button.
mfoltzgoogle: Two types of browser fullscreen and element fullscreen.
… Which one is this?
Brad: This is document fullscreen. Always the document that goes fullscreen.
mfoltzgoogle: The popup window does not get to choose, then?
Brad: That's right.
Many windows, many screens
Mike: Generally, cumbersome UX in initiating multi-screen experiences.
… We did explore multi-screen experiences, ending up with a small extension to requesting fullscreen on a specific screen and open a popup elsewhere.
… That's a useful step.
… But goal is to go beyond that.
… And enable to show content on all screens from a single user gesture.
… [goes through use cases in the slide]
… Idea is to generalize the concept through a user activation API.
… This seems to satisfy a number of requests that we've seen from partners and a set of users.
… We've some additional use cases from partners developing SPA that go beyond what we're proposing here.
… Explainer goes into more details
anssik: Have you started prototyping this in code? Or just an explainer?
Mike: The prototype is not quite functional for the time being.
mfoltzgoogle: In the scenario where multiple windows might be created. Are you thinking about using window.open or using different API shape.
Mike: Already leveraging the existing API, relying on delegation to allow another window to request fullscreen.
Additional Windowing Controls
Mike: VDI is acronym for Virtual Desktop Infrastructure.
… Allows to present remote desktop as if they were local.
… The problem here is the inability for web applications presenting this content of making use of the windowing controls of the remote application.
… e.g. the "Maximize" button.
… Goal is to allow that.
… with a few additional points listed in the slide.
… such as the ability to detect that the window has been resized.
… This explainer has gotten some feedback from Webkit, currently addressing that.
… Feedback was largely about only enabling this functionality for a class of windows, the standard ones.
… As not all windows are candidate for control through the operations in this script.
… Other feedback has, I think, been addressed.
… Proposal is in quite early stages.
mfoltzgoogle: native window controls, relationship with Window Control API.
Mike: The premise is that those controls in green will do nothing in the local client.
… The idea is that the virtual desktop application will monitor the state of the window and use that to programmatically match things between local and remote.
mfoltzgoogle: Second question: is there a scenario where the local VDI application need to know whether it can restore/minimize/restore?
Mike: The explainer is focusing on enabling the feature.
… There may be other window states that prevent it from successfully applying the requested operation.
… VDI app devs will need to check the promise to ensure that the operation was carried through.
Borderless mode
Mike: I will proxy Sunja, since her microphone is not working.
… Problem is that the ability for a web app to customize his title bar is insufficient, even with with Window Control Overlay.
… Exemple provided here.
… The goal of this other display overlay is to enable full control of the title area to allow custom bar or remote content.
… The specific proposal is only making itself available to isolated web apps (discussed earlier this week at TPAC), and gated on window-management permission.
… It received feedback from the TAG.
… Once you remove the local client windowing control, the additional feature will make the yellow and red buttons presented there operative.
… Some synergy.
… Currently this is a Web App Manifest WICG incubation.
Brad: Is there anything that requires that web app some window controls itself?
Mike: Great question that was also raised by the TAG review.
<Zakim> anssik, you wanted to ask how this title bar area is styled, what are the constraints?
Mike: One of the purposes of "borderless" is to enable cases without enforcing the presence of window controls.
anssik: How is styling done? Do you use CSS? Sizing? Color?
Mike: It will be entirely up to the application.
… There is a demo application. The first picture above is litteraly a canvas covering the entire space.
anssik: Devs love to have complete control. E.g., <input type="file">.
… Some existing constraints are meant to prevent spoofing of the user agent, or OS.
… Was it raised in the TAG?
Mike: The TAG acknowledged that the proposal is specifically restricted to Isolated Web Apps.
… The broader question is also on the table of the TAG.
anssik: OK, good to hear about the acknowledgment.
Virtual Display Testing
mfoltzgoogle: We wanted to give an update to this as it could be useful.
… To automatically test multi-screen displays in Chrome, we found a gap.
… We've been working on a mechanism to automate that.
mfoltzgoogle: Initially on Mac, then expanding to other platforms.
… ChromeOS and macOS, progress towards support for Windows OS.
… And we're doing some exploratory work for Linux.
… For Mac, we're using some non public APIs to manage virtual displays. We're trying to see whether these APIs could be made public as they seem useful for other purpose.
… If would be useful to have that type of support in Web Platform Tests.
… That would facilitate testing.
anssik: Intel has been doing some work on test automation for sensors, related to WebDriver.
… we might be interested in helping in the future.
Louay: Thanks for the proposal. Will help a lot in automating more tests.
… Is that related to some APIs or abstract away the APIs?
Brad: The work we're doing is pretty much agnostic of any API, we're enabling this at a lower level
anssik: Louay has been working with Fritz on some test automation.
… Any other testing related comments?
Remote Playback API
Repository: w3c/remote-playback
Remote Playback API test automation
<gb> Issue 92 Remote Playback API test automation (by Honry) [F2F]
Louay: Pull request has been merged recently.
… Now able to run the test from the master branch of WPT.
… I can show the current results on my screen.
… In the dashboard, the remote-playback is listed.
… The results lists the automated tests.
… In Chrome and Safari, there is support of most of the API.
… Then we did a run of additional manual tests.
… There are a few manual tests that fail on current implementation.
… Showing an example of a test.
… You can see the "Prompt" button. The UI says that device is available, and yet it's not working.
… You cannot continue the test because the device selection fails.
… This is specific to Safari. On Chrome, we didn't manage to use the Remote Playback API in desktop.
Fritz: I implemented the tests in Chrome but unable to find how to enable the experimental remote playback API.
Louay: The Remote Playback API is deactivated on the platforms that we are using.
… Mac, Android, Linux.
… This means we do not have two implementations passing tests for all of them.
anssik: For Safari behavior, it would be great to have Eric around.
… For Chrome, any comment?
mfoltzgoogle: The current implementation status is that it should be available on Android. That is what Anton shipped some time ago.
… We are working towards shipping a full API on desktops. I don't believe that it's on by default. I'll check whether the flag name changed.
ACTION: mfoltzgoogle to check on the flag to activate the Remote Playback API on desktop
<gb> I created issue #152
mfoltzgoogle: Pending PR
… It needs review and merging.
tidoust: I'll look into it.
anssik: The Media WG would like to remove monkeypatching from Picture-in-Picture, which would go into Remote Playback API. This would be a note.
Presentation API
anssik: No P1 issue
mfoltzgoogle: Yes, no update.
Matter and Open Screen Protocol
Repository: w3c/openscreenprotocol
anssik: In the charter, we mentioned that we want to evaluate features of Matter as they relate to the Open Screen Protocol. To align with it or reuse these features if applicable.
OSP and Matter interop
anssik: Some exploration, proposal, and recommendations.
mfoltzgoogle: I'm going to be presenting on the Open Screen Protocol in relation with another set of protocols developed elsewhere called Matter.
… The most substantial part will be the last part of the presentation. Good news/Bad news situation but we'll get to that.
mfoltzgoogle: Matter is an architecture that has been in development by a number of industry patners. Google Nest, Alexa, Apple is also involved, smart window blinds, etc.
… At a high level, Matter allows all sorts of devices to interoperate by creating an authenticated network I would say.
… All IP based, but they can go through different types of links, including Thread.
… Even if devices are from different manufacturers, they can be paired if they support Matter.
mfoltzgoogle: Several layers.
… On top of the IP layer, there are a number of them.
… Encodes all sorts of commands and authentication mechanisms that can be used in Matter.
… Goal is to avoid requiring custom programming to interact with Matter devices.
mfoltzgoogle: In this group, we develop the Open Screen Protocol, a way to create interoperable protocols that implementers can use to implement some of the APIs developed in the group.
… There is definitly some mapping here.
… In OSP, we chose to standardize around IETF protocols.
… On top of that, we have different protocols that map to the APIs.
… CBOR allows to use flexible schemas to exchange structured messages.
mfoltzgoogle: Previously we discussed two possible approaches to align Matter and OSP.
… Approach number 1 is the "tunneling option".
… We could make OSP messages look like Matter messages.
… If we don't create a schema for every message, we could have a generic OSP message in Matter.
… And assuming that two Matter devices have authenticated and created a communication channel, this could work.
… Based on internal feedback, this turns out to be somewhat controversial from a Matter perspective.
… You are using a transport that Matter has already created, in many cases an IP transport, and specifically for the streaming cases, we would prefer to use QUIC or UDP.
mfoltzgoogle: Second approach is the "bootstrap" option.
… You can use Matter to establish a QUIC connection. Once that is established, the OSP takes over. And we can proceed as we did before.
… Based on feedback, this seems to be better approach to rely on. This is probably the option that I would recomment. I cannot go into details of upcoming things in Matter.
mfoltzgoogle: Some areas of investigation include suitability of the Matter transport for streaming use cases, authentication. No time to investigate the fourth point.
… We need to await progress on Matter on the relevant platform pieces to be able to go further in this group.
tidoust: Wondering if Matter would be good way to bootstrap devices OSP is built on? Should we not worry about the bootstrapping part?
mfoltzgoogle: It looks #2 option is more likely to be successful, work well with Matter and meet our goals for OSP.
tidoust: For liaison, everything is possible in theory, Second Screen WG works in public, so external organizations have requirements in terms of disclosure so they work in Member-only mode until they publish something
… we cannot directly share CSA material without signining MoU, which is doable, but has to go through internal W3C review and pass W3C AC review.
… we have MoUs for CTA Wave project, it does not include working on confidential things
… it is possible to do that, but it is likely to be an issue on the CSA side because they want work-in-progress deliverables to remain confidential.
… I'd need a good point of contact in CSA related to Matter and can look into it internally, it is not going to be very easy, we don't like confidential requirements in W3C typically.
mfoltzgoogle: It sounds like it is process-heavy to get access to CSA confidential materials, I think this is not critical enough to pursue this route, but I can take an action to understand the timelines for when important CSA work would become public and we would be better informed.
… If the timeline is 6 months, liaison investment not worth it perhaps, but if we talk about years then maybe liaison needs to be looked at.
tidoust: Another idea, we may not need this, another thing that might work, CSA is willing the W3C to take on this work, they could use a liaison statement that is sent to W3C, with confidential information.
… We received these from time to time, we remain on a member-level
… We'd ask for feedback from CSA on this idea of working more closely together on this.
mfoltzgoogle: I think that could be more feasible, I can follow up with my internal Matter contacts on this.
… Also on our side there are different groups working in this space, so I can introduce OSP to them to seek feedback. Maybe this is less of a process that formal MoU.
mfoltzgoogle: I can take an action to start looking at this and understand what are the W3C side steps to make this liaison letter possible.
ACTION: mfoltzgoogle to start looking at mechanisms to exchange information between W3C and CSA on Matter
<gb> I created issue #320
OSP updates and next steps
mfoltzgoogle: I've been going through issues, closed some, labeled others.
… Feel free to review and comment.
… A few things are marked as V2 or enhancement because they require help from other specifications.
… Multiple senders, multiple receivers of remote playback, however there are no clear requirements to do that. If folks in the group want to incubate a solution in the group, that's great, but I'm not inclined to incorporate that as long as there is not more support.
… I prepared PR and did assign a few reviewers to pull requests.
… It would be great for reviewers to do the reviews.
… The remaining issues that are probably not fully resolved are related to security.
… There is e.g., an issue on clarifying our use of TLS.
… Some of those require more discussion.
… Closing down the issues with open PR would get us pretty close to calling the v1 finished.
… My ask is for folks to review open pull requests.
… I didn't want to go into details here, but we can if needed.
mfoltzgoogle: On the implementation front, we've been making a lot of progress thanks to contributions from Wei Wang, many thanks!
… mDNS discovery, QUIC support now uses the same implementation as Chrome (QUICHE). For various reasons we had duplicated code.
… That is good.
… We've implemented about half of the messages in the spec.
… And then authentication remains an open piece, to generate certificates using SPAKE, etc.
… We're making progress, still some work to do, but I really appreciate contributions!
anssik: Yes, he started looking into demos and fixed bugs that he found along the way.
mfoltzgoogle: No demo yet, maybe next meeting.
mfoltzgoogle: One last topic on OSP. The spec is pretty complicated because it tackles a lot of aspects, even though we're building on existing specs.
… Two main pieces: discovery and authentication. And separate piece is control protocol to support the relevant APIs.
… It might make sense to split the spec into two.
… First is network side. Second more the application control side.
… Partly motivated by some of the exploration we've done for alignment with Matter.
mfoltzgoogle: Also a realization that the control protocol could be useful in other contexts.
… E.g., if two agents wanted to use WebRTC Data Channel, or WebTransport channels.
… As long as the messages can be exchanged, the API shape just worked.
mfoltzgoogle: I know there's been interest in implementing in Web contexts. CBOR can be parsed in JavaScript.
… In Chrome, this could be useful as well, to avoid having to maintain too much native code.
… Also, WebCodecs has come along since we started work on OSP. It becomes possible to build most of the application level on top of a web application.
mfoltzgoogle: We could split things completely or not. The OSP spec has already gone through initial steps on the Recommendation track.
… We might be able to move the second specification a bit faster if we could get other implementations based on other protocols.
anssik: I think that there is no concern with splitting.
… If they can be useful in standalone documents, the split is totally reasonable to me.
tidoust: In the WG charter we have listed deliverables and we kept the statement "the group can split things in the ways it wants".
… scope remains, no issues splitting.
… If it is two new specs, we have to do FPWD first.
… No need to be worried, only a bit more work on the spec editor.
… I'm hearing the messaging part will be important and the connection part will be done differently and maybe we don't need it even?
mfoltzgoogle: Good question, don't have an answer yet. The WG has discussed moving some of the work to IETF, and that could be the long-term future for some of the work in this area.
… We'll have an idea how to split when we get to do that work.
[[ The Working Group may decide to group the API and protocol suite in one or more specifications. ]]
https://
PROPOSED RESOLUTION: Split the Open Screen Protocol into a Connection spec and a Messaging spec
RESOLUTION: Split the Open Screen Protocol into a Connection spec and a Messaging spec
mfoltzgoogle: Last slide lists recommendations for the Second Screen Working Group.
<gb> Issue 308 Matter protocol similarities (by backkem) [F2F]
Michiel: What has been presented so far aligns with my thinking as well, that's great.
… The one thing I want to ask. In the second proposal when you open a separate connection for OSP, how would devices signal compatibility with OSP in that setting?
mfoltzgoogle: Probably there will have to be a new device type from a Matter perspective or a new capability.
… and then ways to establish a communication channel before things can proceed from there. I don't have a technical answer to that yet.
Michiel: A connection to something around the local-peer-to-peer protocol
New incubations
Local Peer-to-Peer API
Local Peer-to-Peer API Explainer
anssik: Local P2P API allows devices to exchange messages or files through close-range communication in a privacy-preserving manner. The key use case is sharing file from one device to another.
… Or maybe a two-player game.
… The requirements: discover nearby devices, ability to push files from one device, to pull a file, and establish a bi-directional communication channel.
… Most of these, we think, are covered by OSP.
… We're exploring an extension to OSP accordingly.
… The protocol stack is actually quite flexible and reusable.
… Feedback is welcome.
… Michiel has been providing some good comments already, thank you.
Allow target navigation at split tab
Allow target navigation at split tab Explainer
Sanket: we want to allow sites to target a link to open directly into a split tab. The current navigation targets, such as _blank, _self, _parent, _top, are insufficient to solve this problem.
… Example: browsing search engine results, watching videos on Youtube.
… Designed to fallback to _blank.
… Maybe the screen is too small for instance.
… Allow this to be implemented differently across platforms.
… We're looking for feedback on the proposal.
… Or other related proposals in this space or additional use cases.
msw: How does this work for cases where the split tab is already open?
… Have you thought more about if a third party iframe in the opener is able to target the split target window, or navigate the open split target window?
sanket: Yes, basically, it's conservative.
… If it's inside of an iframe, that will respect the usual security constraints of iframe, depending on what iframe attributes you set.
Brad: What's the advantage of having this at the browser level?
… You could implement the split view at the web page level, through an iframe.
Sanket: If the site wanted to split screens, they could do it themselves, for sure. But not everyone wants to do this.
… They might choose an UI if split is supported. Based on the user's UI choice, they may want to proceed differently.
… Better for the user than if each site has a different implementation.
anssik: Let us know where you want to have feedback. Thank you for this proposal.
Hakan: Thomas Steiner asked the question, I think. How this target navigation feature would work if browser supported more than one split tab. See MicrosoftEdge/
Hakan: Someone commented that having multiple split tabs would make product comparison easier.
Sanket: Right now, proposal supports only one split tab because that's what browsers support. That said, we're trying to design the API so that it's extensible and so that this can be done when browsers start supporting more than one split.
… Really open to looking into it.
msw: The screen object itself can be used to specify a screen for fullscreen. It provides a rectangle in multi-screen coordinate space that can be used to position content.
Sanket: We could consider following a similar path.
… That could work.
anssik: Thank you for the discussion!
Back to Picture-in-Picture
cpn: One section in the Picture-in-Picture talks about interaction with the Remote Playback API.
… We'd like to move that information to the Remote Playback API under the definition of the local playback device, as a note.
cpn: There is already a note in there. I'd like to extend that definition.
… Just to give a bit more explanation about what we consider to be local playback.
mfoltzgoogle: That seems fine. If it's an informative note, feel free to open a PR. If it needs to be normative, we probably need to discuss more.
cpn: OK.
<backkem> Thought that popped into my mind: If OSP is split into a OSP-connect and OSP-messaging part. The local-p2p API almost looks like the API surface for the OSP-connect part. What would be the best place to explore this further?
<mfoltzgoogle> bakkem, raise an issue in the OSP github, feel free to elaborate on your idea