Joint meeting between CSS WG and Media WG

14 July 2020


Alan Stearns, Barbara Hochgesang, Becca Hughes, Chris Needham, Daniel Holbert, Florian Rivoal, Gary Katsevman, Greg Freedman, Jer Noble, Jimmy Huang, Kevin Babbitt, Matt Wolenetz, Mounir Lamouri, Nigel Megitt, Peng Liu, Rossen Atanassov, Simon Fraser, Simon Thompson, Theresa O'Connor, Vi Nguyen
Jer, Mounir
Jer, Mounir, Vi

Meeting minutes


Video Media Queries

Related issues:

florian: introduction of topic: some devices have separate planes for video and graphics contents

florian: these planes have separate properties including width, height, and resolution. The existing queries target only the graphics plane

florian: video sites care mostly about the graphics plane, and would like api to choose an appropriate variant

florian: originally the CSS WG accepted this proposal, but found issues and wondered if these were the correct way of solving this problem

florian: Media Queries need to come with a unit; CSS pixels? CSS Pixels can mean an angular measurement that changes size based on the distance from the screen. Physical pixels? anchor a CSS measurement to a physical measurement.

florian: for the video plane, you may want actual literal pixels, which would include a third definition.

florian: One possibility would be to make these properties unitless.

florian: this issue has an analogue for images, and the general path has been not to do this through Media Queries.

florian: images must also be sensitive to changes in resolution as windows move between windows, but also to available bandwidth

florian: for images, the UA will choose the appropriate image variant based on not only resolution, but other factors.

florian: this may not be the same solution available for video, but there could be a .js API in which the page provides the available resolutions and properties, then the UA chooses the correct variant

florian: were there other arguments pro or con for Media Queries?

chcunningham: question: does it not solve the problem by mapping the pixel ratio to a fixed value?

florian: we could make an allowance, but it's not currently allowed.

florian: what does that then mean when people query for the video plane in "em" units?

florian: There's not a number that is based on font sizes that is correct.

chcunningham: if you use the existing mechanism that ratio, you could discover the ratio.

florian: Media Queries don't allow you to query for the size of the screen, it only allows you to query for whether the screen is within a provided value

florian: maybe this is the wrong tool, since you can't query the value directly.

chcunningham: I don't think that's a dealbreaker, since sites will have a list of formats or variants with specific resolutions, so that works with the way Media Queries work

chcunningham: w.r.t. the Picture API; this would be a profound change from how media APIs are used on the web.

chcunningham: in MSE and WebCodecs, the page has low level access to variant selection

chcunningham: clients prefer to make these decisions themselves

dholbert: there are also "inch' and "centimeter" units

dholbert: that would be something difficult to define; argument for unitless value

florian: if we wanted to do this, we don't want to break all of CSS units, so it would still be a fixed ratio between this value and a CSS inch

cpn: There was a proposal to expose these values on the screen object; is there a value to re-visiting that decision?

florian: the query example is only one possibility; we could also just expose those numbers explicitly.

cpn: a proposal which inverts the current model that gives all the variant selection to the client would be a big change

wolenetz: is there already a way for JS apps to determine the resolution where the video will be composited?

florian: I think the Media Queries was proposed as a way to expose that value.

wolenetz: the current way media works on the web is to expose to clients underlying capabilities and allow the site to choose variants.

Simon_T_BBC: set top boxes should be thought about.

Simon_T_BBC: HDMI capabilities must also be taken into account; some may only be capable of a certain frame rate

Simon_T_BBC: There are some non-screen parameters that are important for deciding which variant to choose.

florian: you may want to make a different tradeoff than just available resolution

jernoble: as the person who proposed MediaQueries as a possible solution. Media Queries already provides event mechanism for when that ratio changes when moving windows to different screens. If we did expose something to Screens API, we have to figure out events

chcunningham: I understand CSS WG is concerned that this is not the correct solution. But we ran this by the WG before it was proposed. Are we back to the drawing board, or is this a minor issue?

florian: I don't think the CSS WG has concluded that this was impossible; merely uneasy. On further reflection, the way it is spec'd cannot work. It doesn't work for print media. We would need to fix the unit issue to make this possible.

florian: we could fix this, but the questions this raises makes this solution feel "odd".

florian: there's also a desire in the CSS WG that for devices that do not have a separate video plane, these properties should be the same for the graphics plane.

florian: screen properties were deprecated, because leaking non-important information, and those properties have privacy implications

florian: CSS pixels can and will be used incorrectly

florian: we think this should return to the drawing board.

florian: the current approach is possible, but not ideal.

ving: the Media WG did discuss exposing this to the Screen object; could we get a summary of why Screen is not the ideal proposal?

ving: also, there was a concern about interfaces and cables, but that should be handled through MediaCapabilities API. We're trying to solve the problem of exposing the exact resolution of the video plane on the actual display.

chcunningham: where are you seeing these concerns jernoble listed?

ving: in the notes.

chcunningham: I don't oppose putting this on screen; florian what do you think of having device pixels on the Screen?

florian: my personal feeling is that's ok; we'd need events if that changes

florian: is this a concern for those devices that have a separate video plane?

mounir: I don't think an event is really an issue. The Screen object doesn't have an event. This is an existing problem in that the Screen's properties can change without an event.

chcunningham: if we expose device pixels on the screen object, we're hitting the same reasons why the existing properties are deprecated

florian: if we expose the video device pixel, you'd just get a number that is in device pixels. You wouldn't have a way to ask for the video plane in "em" unit.s

florian: yes, we'd still have the privacy concern.

GregFreedman: if you put device pixels on the Screen object, you'd still have the problem of the difference between the video and graphics plane. Would you have device pixels for both planes?

GregFreedman: I kind of liked where we landed with the Media Queries. what we want to do is say: can you support this resolution on the video plane; and that seems like this works with Media Queries well.

GregFreedman: is the problem that CSS doesn't like the unitless query?

florian: yes, CSS as a language could support it, but if we're on a device that does not have a separate plane, we probably want width: to do the same thing as video-width:

florian: additionally, that definition also affects print media

florian: so what should you do on print media?

<Zakim> nigel, you wanted to ask for a reminder of the use cases that remain if we use MediaCapabilities to choose which video to get

nigel: the use case is originally to detect what variant to display. If MediaCapabilities is the best way to query for that, what's left in CSS ?

GregFreedman: the device may be capable of doing 4k UHD, but if the display is only 1080p SD, there's no reason to pick that variant. so we split it into two APIs: one to query decode, one to query display.

ving: MediaCapabilies also is not screen aware.

nigel: could Media Queries handle that scenario, where there's multiple displays?

hober: we already handle that with the 50%+1 to determine which monitor's properties to return.

chcunningham: that's the implementation in chrome.

cpn: can the UA play this particular content is answered by the combination of screen and decode capabilities.

cpn: asking these two questions separately doesn't give a complete picture of whether a variant can be successfully played

GregFreedman: Media Capabilities (MC) asks "can we decode this" and Media Queries (MQ) asks "should we decode this".

nigel: is the set of things we need to know just the video plane resolution, and the frame rate?

nigel: are there other properties that we need to include?

florian: those do not yet exist?

<Zakim> wolenetz, you wanted to say that decoder capabilities exceeding display capabilities shouldn't limit what is decodable

nigel: just wondering about design and what info needs to be included.

wolenetz: Media Capabilities already gives you the answer of whether you can decode a given piece of media. Do we try to move some of the properties in Media Capabilities into Media Queries?

mounir: oginially there was information in the Screen object.

wolenetz: since Media Capabilities is already being asked about candidates, maybe Media Capabilities is the best place to ask more questions

wolenetz: since I'm already using Media Capabilities, doing that again against a different API is difficult.

chcunningham: I have the exact opposite opinion, respectfully. the Screen API and the Media Queries API already exists.

chcunningham: we never considered putting Media Capabilities queries into the Screen API

mounir: websites already will react to window size changes, e.g., when going into fullscreen.

jernoble: we want to answer the question whether the video should be upscaled or downscaled

jernoble: so maybe we should design an API for that?

smfr: I would be reluctant to add new APIs to the Screen api due to fingerprinting concerns.

<Zakim> florian, you wanted to suggest that the JS API could take the frame rate as input more easily than Media Queries

florian: if the frame rate matters, it would be more amenable to a Screen API, since you can pass in parameters to that API, but that style of query isn't possible with Media Queries.

chcunningham: no option seems to be the clear winner; IMO the least of several evils was the CSS Media Queries options.

<nigel> CSS Media Queries don't remove the fingerprinting vector

chcunningham: I wonder if there is a way to fix this problem

florian: I'd like to ask Simon to clarify this answer about fingerprinting

smfr: it depends on whether it allows you to enumerate all screens. and depends on whether the API allows you to detect whether users have changed their screen resolution or are using non-standard third party screens

smfr: it's still doable with Media Queries, but you have to binary search, so it's not as easy.

mounir: what's the next step here? shall we move this to GitHub issues?

chcunningham: we haven't discussed any options beyond "fix Media Queries" or "move to Screen".

<astearns> lost my webex connection - would there be a way of making a video-playback-specific API that would expose less fingerprinting surface but still be able to answer coarse questions on what can be displayed? How fine-grained do we really need to be for playback?

chcunningham: maybe now that we've had this meeting the CSS WG will come up with an alternate solution that better solves all the use cases

mounir: lets follow up on GitHub; lets try to discuss between MediaWG and CSS WG people there

florian: it's unfortunate that there were fewer voices in the CSS WG than I anticipated; there are conflicting desires in the CSS WG.

florian: I'm unable to represent all those various POV.

florian: but yes, we can explore; either alternative ways of speccing the Media Queries or alternative API entirely.

chcunningham: thank you florian! and thank you for representing the CSS WG.

Minutes manually created (not a transcript), formatted by scribe.perl version 121 (Mon Jun 8 14:50:45 2020 UTC).