Meeting minutes
Welcome
anssik: Welcome on behalf of the chairs to the 2022 Q2 meeting of the Second screen WG/CG.
anssik: CG incubates new ideas. Multiple cases where incubations have advanced to the WG. Thanks for those who pushed multiscreen capabilities through this pipeline!
WG Charter 2022-2024 substantial changes
anssik: This working group was re-chartered a few months ago.
<msw> Woohoo :D ty!
anssik: Multi-screen window placement API was added to the charter.
… We clarified terms
… New groups in coordination section: CSS WG (for multi-screen window placement), Media WG (OSP) and Connectivity Standards Alliance (also for the OSP)
mfoltzgoogle: I was wondering whether there was additional feedback during rechartering process that was received. Any substantive feedback received?
tidoust: Some suggestions to change group process but no substantive feedback received.
CG Charter update 2022
anssik: The community group incubated multi-screen window placement in the previous period. Now migrated to the working group
Patent Advisory Group update
anssik: There is a patent advisory group that is currently running to address patents and claims disclosed and excluded related to the Open Screen Protocol.
… I can share that the PAG has made susbstantive progress recently and planning to publish a final report by the end of the PAG chartering period.
Introductions
anssik: Anssi Kostiainen, Intel, WG/CG co-chair.
Francois: W3C Team, Staff contact for the group
mfoltzgoogle: Mark Foltz, Google, editor for the Presentation API and Open Screen Protocol
btriebw: Brad Tribwasser, also at Google, working with Mark and Mike
cpn: Chris Needham, BBC, co-chair of the Media WG for which this group has a liaison.
Fritz_Heiden: Fritz Heiden, Fraunhofer FOKUS, working in Louay's team, involved in Remote Playback API testing within Web Platform Tests.
Louay: Louay Bassbouss, Fraunhofer FOKUS, co-chair of the Second screen WG/CG
martinthomson: Martin Thomson, Mozilla, just a visitor of this meeting, happy to talk about multi-screen window placement API.
msw: Mike Wasserman, Google, editor of the Multi-screen window placement API.
takumif: Takumy Fujimoto, Google, working on an extension proposal for the Presentation API
Agenda bashing
anssik: [going through the agenda]. Any changes?
[none heard]
Presentation API v1
Presentation API W3C Candidate Recommendation Draft 02 June 2022
anssik: For v1, the working group believes that the spec scope is stable. We're pending open screen protocol implementation feedback
… Last publication status is Candidate Recommendation Draft from a few days ago.
… No substantive changes brought to the spec recently.
mfoltzgoogle: Unless we discover something through implementations, I don't expect there will be significant changes. The one area we may propose some editing is around 1 UA mode.
… Now that Multi-screen window placement has progressed further, there's some overlap, so we may adjust the description of 1 UA mode or somehow update to reflect that.
anssik: Should we have an issue open for that?
mfoltzgoogle: I don't have a specific proposal at this time but happy to discuss.
… I can open an issue as a placeholder.
Presentation API v2 and new features
Site Initiated Mirroring API
anssik: Mark and Takumi prepared an explainer.
<takumif> Slides: https://
<takumif> Chromium bug tracker: https://
<takumif> TAG review: https://
<takumif> Chrome Platform Status: https://
takumif: A number of links related to the API, including slides, TAG review, bug tracker, and implementation status.
… Site-initiated mirroring is a feature to allow a web site to request to mirror the contents of its own tab.
… It is different from the current usage of the Presentation API and Remote Playback API.
… In the Presentation API, the controller requests presentation of another URL. In the Remote Playback API, the website requests to cast a media element.
… With site-initiated mirroring, the request is to cast the entire tab.
… Thisi is currently implemented in Chrome through a Cast SDK.
… Our proposal is to expand the Presentation API.
… We're proposing that, instead of passing a URL, you would pass some special URL to indicate that the web site wants to mirror, along with additional parameters e.g. to indicate that you'd like the capture latency to be low
… and then you would get the connection as usual.
… This would allow a web site to do a mix of mirroring or regular presentation API to support receivers that may not support mirroring.
… Sites would be able to do the fallback within a single presentation request.
… Mark and I wrote an explainer, started working on the implementation in Chromium and I have just requested a TAG review this week.
… That is the status.
anssik: There's no API that allows a MediaStream to be sent to a secondary display. Considering discussions on extending getDisplayMedia(), would it be a possible alternative?
mfoltzgoogle: With the Presentation API, the content of the site is mirrored purely as a browser feature, not as something that the site can have access to. getDisplayMedia is more powerful in that regard.
<martinthomson> The site doesn't necessarily need access to the MediaStreamTrack. I've built isolation capabilities into Gecko for that (building on existing stuff).
mfoltzgoogle: Maybe you do want to capture a tab or canvas, modify the content and then send it, for which getDisplayMedia could be useful.
martinthomson: The whole point of capturing such capabilities in getDisplayMedia was to provide the foundations for all such usage.
… If we had a mechanism to plug that into Presentation API, that would solve things. Some ongoing exploration at Google I think on that, including audio control, etc.
anssik: We have a coordination with the WebRTC WG. Maybe we can look at this approach.
martinthomson: Ultimately, I can only offer my perspective.
… Jan-Ivar would be able to provide a more definitive opinion on WebRTC related topics.
Louay: Regarding the Presentation connection, how do you deal with it? You don't need the communication channel, right?
takumif: That is correct.
… If you're doing mirroring, the connection is going to have some source to allow the controlling side to change settings, and terminate the presentation, but wouldn't be able to communicate with the receiving side, simply because there would be no receiving side as such.
Louay: If we have more integration with media stream APIs, would it be a more natural way to perform Remote Playback API related scenarios?
Extend the Remote Playback API
takumif: We looked at extending the Remote Playback API. We could go either way. For us, the current users are already using the Presentation API. It would be useful if they could do both mirroring or presentation URLs through the same API request call. That's one argument.
… I believe there are other arguments in the explainer.
… Presenting the entire tab is arguably more than pure media, which is the focus of the Remote Playback API.
anssik: I'm hearing some interest to explore MediaStream work for this API. In the TAG review, you may want to surface that aspect as well.
… Good work otherwise, please go to repo and open issues as needed!
<martinthomson> One thing I note about the getDisplayMedia complaint (double-prompting) is that we could avoid a prompt when requesting that if the stream were isolated.
takumif: I'm not that familiar with getDisplayMedia. I wonder whether the embedder origin can get the entire thing.
anssik: Related to capability delegation.
martinthomson: Probably more permissions policy that is the right thing to look at.
… Something else: the potential to get rid of one of the prompts, you could isolate the media from the site so that it wouldn't be able to see the content and it would only be able to show it on the second screen. The second prompt could be dropped in that case.
Local Presentation API to enable child-window presentation documents
<ghurlbot> Issue 347 Local Presentation API to enable child-window presentation documents (mfoltzgoogle) v2, P1
mfoltzgoogle: I thought I had updated the issue with a brief update, but don't see it here.
… Probably just forgot to hit "submit", sorry about that.
… This mode was requested for documents that wish to go in presentation mode, particularly on wired displays and those kinds of use cases.
<martinthomson> https://
mfoltzgoogle: Between Takumi's site mirroring proposal and multi-screen window placement API, that covers pretty much all scenarios that I was looking into.
… If you want to present to a wired display, the multi-screen window placement API may be used. For wireless displays, mirroring could be used.
… I may close this particular issue for the Presentation API as a result.
anssik: We had this marked as explicit feature for v2.
mfoltzgoogle: We can call for consensus but it can be dropped from the CG charter
RESOLUTION: Start CfC to drop "Local Presentation Mode" from the CG Charter scope
anssik: CfC would run for 10 days.
ACTION: mfoltzgoogle to update status for issue #347 (Local Presentation API to enable child-window presentation documents)
<ghurlbot> Created action 504 update status for issue #347 (Local Presentation API to enable child-window presentation documents) (on ) due 14 Jun 2022
Presentation of objects
<ghurlbot> Issue 439 Presentation of objects (mfoltzgoogle) P3
mfoltzgoogle: This never got passed the idea stage. We haven't found necessarily a use case for it yet. The scenarios where the application has the encoded media and wishes to send that to a secondary display could either be handled through the Presentation API, or potentially by plumbing it through a media element and using the Remote Playback API.
… Still the problem of "need two prompts", but that's orthogonal.
… We still need some way to encode the media before we can transmit it. This issue was filed before advances like WebCodecs and streams, which allow sites to create encoded media from their own frames.
… Given those, it may be possible to achieve all of this with current or upcoming technologies.
… Some polyfill or prototype would help clarify what can be achieved. If there are gaps found, we can then try to fill them.
… Right now, there isn't a good solution for rendering to a canvas, capturing that and sending to a secondary display. There is no clear path to do that.
… Until we've done this exploration with WebCodecs, we won't know whether that works. There may still be a need to expand the Presentation API.
anssik: This intersects with our previous discussion on MediaStream.
martinthomson: Fullscreen API, ability to project from same origin to a screen, from different origin, etc., it seems to me that there is a way to create a common mechanism to address the different scenarios.
mfoltzgoogle: I would not support adding bespoke APIs.
martinthomson: Potential avenue through PeerConnection, but requires a different programming model.
mfoltzgoogle: I don't know whether we can expect corresponding capabilities to be supported across receivers, which are typically much more constrained than regular devices.
martinthomson: There's also the assumption that the second screen has access to the same network connection as the controller. Access to public internet, etc.
mfoltzgoogle: More thinking about capabilities such as jitter buffer support, etc.
martinthomson: You're often talking about same-network communication, so maybe it could still work.
mfoltzgoogle: Until we invest in experimentations, it's going to be tricky to understand what is the best solution.
… I'll look into it, but don't know when I'll get to it.
<martinthomson> mfoltzgoogle: maybe drop a note as a comment on https://
ACTION: mfoltzgoogle to describe the prototype opportunity for #439
<ghurlbot> Created action 505 describe the prototype opportunity for #439 (on ) due 14 Jun 2022
Louay: I can take this topic and share with students for next semester.
anssik: That would be fantastic.
anssik: I propose to postpone discussion on other v2 features unless there are v2 features that need to be discussed now.
Remote Playback API v1
Implementation report refresh
Remote Playback API: All Results
w3c/test-results/remote-playback repo
anssik: Louay and Fritz volunteered to refresh the implementation status report.
Louay: It took a while to find students willing to work on it. Fritz is also involved in CTA WAVE testing so an expert on this.
… The way we do this: we develop a test implementation: stimulus and observation.
… For instance, playback capability on a screen, and you want to detect latency, you need some camera to observe automatically.
… For us, the most important thing is to have this implementation report, and then we can explore automation in the future, e.g. using cameras or sound.
Fritz_Heiden: Approach was to look at the specification and pinpoint the main features that the API provides, then trying to provide a test for each of the functionalities. So far, I came up with 5 tests.
… First test was already contained in the repository. Then I tried to implement the test that includes the whole procedure to cast a video.
… This needs to be a manual test so far as user needs to select the display and confirm that the video is actually playing on the second screen.
Fritz_Heiden: In the end, there will likely be more than 5 tests, to handle exceptions and corner cases.
<ghurlbot> Issue 92 Remote Playback API test automation (Honry) F2F
Fritz_Heiden: For each of them, I have pre-conditions, stimulus that the test needs to provide and observations to be made.
anssik: It would be useful if this information were available on the GitHub repo, either in the issue itself or as a separated markdown document.
Louay: Once we have tests available, we can ask someone from WPT to review and merge the PR.
anssik: Yes, it is appropriate if tests are manual to start with. It is of course great to have automated tests but not a requirement for transition to Proposed Recommendation.
Louay: To automate tests, we would need browsers to implement something so that selection of receiver does not require user intervention.
Path to Proposed Recommendation
Louay: Detecting video/audio playback is more doable automatically, but screen selection currently cannot be automated.
anssik: "The Working Group plans to publish the Remote Playback API as a final Recommendation when the Success Criteria are met."
WG Charter > Normative Specifications > Remote Playback API
anssik: "In order to advance to Proposed Recommendation, each normative specification is expected to have at least two independent implementations of every feature defined in the specification."
Multi-Screen Window Placement API
Path to First Public Working Draft
anssik: "To publish the First Public Working Draft of a document, a Working Group must meet the applicable requirements for advancement."
W3C Process > Publishing a First Public Working Draft
anssik: The MVP of the API already ships in Chrome. I think the spec is adequately complete for publication as First Public Working Draft. Once we have group approval, I think we can request publication.
RESOLUTION: Start a CfC to publish Multi-Screen Window Placement API First Public Working Draft
Mozilla's feedback
Mozilla position: Multi-Screen Window Placement
anssik: Mozilla reviewed the API and provided feedback.
Mozilla position: Multi-Screen Window Placement: Fullscreen Companion Window
msw: As I understood it, the biggest point of feedback in the standard position thread was about the privacy implications of exposing screen indications to the site.
… I believe we've had a series of back and forth exchanges on how these implications could be addressed and how implementations can deal with exposing screens.
… There's also a point on X11 style falling out of favor, but not sure what the specific concern was.
… Requiring users to move a window before exposing additional screens does not seem ideal. That can be explored though.
martinthomson: I cannot really speak to Anne's comment on X11. Related to unusual display layouts. Answer may be "well, too bad", or there may be other ways to deal with that. Probably some work to be done there.
… I don't know where Anne's concern will ultimately go there.
… My concern is more around user consent. We're looking of ways to avoid having to rely too heavily on user consent for something that is more or less going to go without problem in most cases here.
… Where we disagree is that we don't have a real recourse in case we're taking different approach. If Mozilla decides to only present a screen when a web site has been rendered onto it, that's fundamentally incompatible with Chrome's approach, and Chrome having shipped this already creates practical history that makes it difficult to envision other paradigms.
msw: It would take some amount of training users to understand that a site can only place a window on a specific display if user has already done that manually in the past.
… It would take some amount of UA instructions to educate users, and it may be that the appropriate time to do something like that would be when a site requests access to detailed screen information.
martinthomson: You're suggesting that the browser deal with this. I was imagining that sites would need to deal with it.
… In the case where we want to hide a screen that has never been used, you're saying that it would be on the user agent to do the education. That's possible.
… What worries me is prompting users with a prompt that does not mean much to people. You're right that there's some education to be done here.
… It's a fairly narrow application domain. Do you have examples?
msw: Spec highlights some of them. Presentation software such as Powerpoint or Google slides.
… Creative software, multi-screen gaming potentially, even video playback.
… Some big ones are around VDI providers who want to engage with multi-screen, display advertising.
… Medical images software providers as well.
martinthomson: It sounds that a lot of these use cases are not casual interactions with web sites.
msw: One of our most engaged partners is Google slides where the flow of starting a presentation and having it work well, e.g. to present your pitch to investors, is important.
… A model that allows them to connect a new display and work with it is needed so that they don't have to move a window across screens.
martinthomson: Makes sense and you're talking about new displays here.
anssik: [mentions OS integrations]. I'm wondering about quality of implementation differentiation.
martinthomson: There's a challenge here. If we have the shape of the API as implemented in Chrome today, we would have the challenge of having to educate users before the Promise can be resolved.
… Maybe this would be ok as sites may tolerate screens appearing over time.
anssik: I'm hearing that we are starting to converge on possible solutions.
… Probably details to look into.
… Thanks for the feedback.
… Tomorrow, we'll explore virtual displays, which should resonate with today's discussion.
<martinthomson> thanks for describing that in a little more detail msw; I don't know that it resolved my concerns completely, but it did help
anssik: We're already overtime today. OK to move the rest of the discussion tomorrow?
msw: Yes, fine with me.
… I wouldn't expect to need more than 5mn per topic.
mfoltzgoogle: No issue labeled with "F2F" for the Open Screen Protocol, so discussions should be short.
… I do have some material prepared on Matter developed by the CSA. Could take 15-25mn depending on time available.
… I want to make sure Mike has enough time for the Multi-screen Window Placement API and save some time to discuss TPAC.
anssik: OK, that sounds good. For TPAC, we need to know whether we plan to meet there and how many participants will be present.
msw: Just wanted to thank Martin for joining today and providing feedback.
anssik: Indeed. Thanks Martin, and thanks all for joining the meeting! Talk to you tomorrow.