[Minutes] Media Working Group Teleconference - 2020-07-14

Hi all,

The minutes of last week's joint meeting between the Media WG and the 
CSS Media WG are available at:
https://www.w3.org/2020/07/14-mediawg-minutes.html

The groups discussed solutions and problems to expose bi-plane 
(video/graphics) capabilities to applications. Minutes are copied as raw 
text below.

Thanks,
Francois.

-----
Joint meeting between CSS WG and Media WG
14 July 2020

    [2]Agenda. [3]IRC log.

       [2] https://github.com/w3c/media-wg/issues/18
       [3] https://www.w3.org/2020/07/14-mediawg-irc

Attendees

    Present
           Alan Stearns, Barbara Hochgesang, Becca Hughes, Chris
           Needham, Daniel Holbert, Florian Rivoal, Gary Katsevman,
           Greg Freedman, Jer Noble, Jimmy Huang, Kevin Babbitt,
           Matt Wolenetz, Mounir Lamouri, Nigel Megitt, Peng Liu,
           Rossen Atanassov, Simon Fraser, Simon Thompson, Theresa
           O'Connor, Vi Nguyen

    Regrets
           -

    Chair
           Jer, Mounir

    Scribe
           Jer, Mounir, Vi

Contents

     1. [4]Introductions
     2. [5]Video Media Queries

Meeting minutes

   Introductions

      * Florian Rivoal, Invited Expert
      * Daniel Holbert, Mozilla
      * Alan Stearns, Adobe
      * Matt Wolenetz, MSE spec co-editor, Google
      * Jer Noble, Apple
      * Greg Freedman, Netflix
      * Simon Fraser, Apple
      * Rossen Atanassov, Microsoft
      * Jimmy Huang, Intel
      * Mounir Lamouri, Google
      * Kevin Babbitt, Microsoft
      * Nigel Megitt, BBC, Chair TTWG
      * Simon Thompson, BBC
      * Theresa O'Connor, Apple
      * Vi Nguyen, Microsoft, Media Capabilities
      * Peng Liu, Apple
      * Chris Needham, BBC, Media & Entertainment IG co-chair
      * Gary Katsevman, Brightcove, Chair TTWG
      * Chris Cunningham, Google

   Video Media Queries

    Related issues:
      * [6]Media WG issue #135 — hdrSupported - Screen.video or
        simply Screen?
       https://github.com/w3c/media-capabilities/issues/135

      * [7]CSS WG issue #4471 —
        [cssom-1][mediaqueries-5][css-color] Dealing with bi-plane
        (video/graphics) when reporting values

       [7] https://github.com/w3c/csswg-drafts/issues/4471

    florian: introduction of topic: some devices have separate
    planes for video and graphics contents

    florian: these planes have separate properties including width,
    height, and resolution. The existing queries target only the
    graphics plane

    florian: video sites care mostly about the graphics plane, and
    would like api to choose an appropriate variant

    florian: originally the CSS WG accepted this proposal, but
    found issues and wondered if these were the correct way of
    solving this problem

    florian: Media Queries need to come with a unit; CSS pixels?
    CSS Pixels can mean an angular measurement that changes size
    based on the distance from the screen. Physical pixels? anchor
    a CSS measurement to a physical measurement.

    florian: for the video plane, you may want actual literal
    pixels, which would include a third definition.

    florian: One possibility would be to make these properties
    unitless.

    florian: this issue has an analogue for images, and the general
    path has been not to do this through Media Queries.

    florian: images must also be sensitive to changes in resolution
    as windows move between windows, but also to available
    bandwidth

    florian: for images, the UA will choose the appropriate image
    variant based on not only resolution, but other factors.

    florian: this may not be the same solution available for video,
    but there could be a .js API in which the page provides the
    available resolutions and properties, then the UA chooses the
    correct variant

    florian: were there other arguments pro or con for Media
    Queries?

    chcunningham: question: does it not solve the problem by
    mapping the pixel ratio to a fixed value?

    florian: we could make an allowance, but it's not currently
    allowed.

    florian: what does that then mean when people query for the
    video plane in "em" units?

    florian: There's not a number that is based on font sizes that
    is correct.

    chcunningham: if you use the existing mechanism that ratio, you
    could discover the ratio.

    florian: Media Queries don't allow you to query for the size of
    the screen, it only allows you to query for whether the screen
    is within a provided value

    florian: maybe this is the wrong tool, since you can't query
    the value directly.

    chcunningham: I don't think that's a dealbreaker, since sites
    will have a list of formats or variants with specific
    resolutions, so that works with the way Media Queries work

    chcunningham: w.r.t. the Picture API; this would be a profound
    change from how media APIs are used on the web.

    chcunningham: in MSE and WebCodecs, the page has low level
    access to variant selection

    chcunningham: clients prefer to make these decisions themselves

    dholbert: there are also "inch' and "centimeter" units

    dholbert: that would be something difficult to define; argument
    for unitless value

    florian: if we wanted to do this, we don't want to break all of
    CSS units, so it would still be a fixed ratio between this
    value and a CSS inch

    cpn: There was a proposal to expose these values on the screen
    object; is there a value to re-visiting that decision?

    florian: the query example is only one possibility; we could
    also just expose those numbers explicitly.

    cpn: a proposal which inverts the current model that gives all
    the variant selection to the client would be a big change

    wolenetz: is there already a way for JS apps to determine the
    resolution where the video will be composited?

    florian: I think the Media Queries was proposed as a way to
    expose that value.

    wolenetz: the current way media works on the web is to expose
    to clients underlying capabilities and allow the site to choose
    variants.

    Simon_T_BBC: set top boxes should be thought about.

    Simon_T_BBC: HDMI capabilities must also be taken into account;
    some may only be capable of a certain frame rate

    Simon_T_BBC: There are some non-screen parameters that are
    important for deciding which variant to choose.

    florian: you may want to make a different tradeoff than just
    available resolution

    jernoble: as the person who proposed MediaQueries as a possible
    solution. Media Queries already provides event mechanism for
    when that ratio changes when moving windows to different
    screens. If we did expose something to Screens API, we have to
    figure out events

    chcunningham: I understand CSS WG is concerned that this is not
    the correct solution. But we ran this by the WG before it was
    proposed. Are we back to the drawing board, or is this a minor
    issue?

    florian: I don't think the CSS WG has concluded that this was
    impossible; merely uneasy. On further reflection, the way it is
    spec'd cannot work. It doesn't work for print media. We would
    need to fix the unit issue to make this possible.

    florian: we could fix this, but the questions this raises makes
    this solution feel "odd".

    florian: there's also a desire in the CSS WG that for devices
    that do not have a separate video plane, these properties
    should be the same for the graphics plane.

    florian: screen properties were deprecated, because leaking
    non-important information, and those properties have privacy
    implications

    florian: CSS pixels can and will be used incorrectly

    florian: we think this should return to the drawing board.

    florian: the current approach is possible, but not ideal.

    ving: the Media WG did discuss exposing this to the Screen
    object; could we get a summary of why Screen is not the ideal
    proposal?

    ving: also, there was a concern about interfaces and cables,
    but that should be handled through MediaCapabilities API. We're
    trying to solve the problem of exposing the exact resolution of
    the video plane on the actual display.

    chcunningham: where are you seeing these concerns jernoble
    listed?

    ving: in the notes.

    chcunningham: I don't oppose putting this on screen; florian
    what do you think of having device pixels on the Screen?

    florian: my personal feeling is that's ok; we'd need events if
    that changes

    florian: is this a concern for those devices that have a
    separate video plane?

    mounir: I don't think an event is really an issue. The Screen
    object doesn't have an event. This is an existing problem in
    that the Screen's properties can change without an event.

    chcunningham: if we expose device pixels on the screen object,
    we're hitting the same reasons why the existing properties are
    deprecated

    florian: if we expose the video device pixel, you'd just get a
    number that is in device pixels. You wouldn't have a way to ask
    for the video plane in "em" unit.s

    florian: yes, we'd still have the privacy concern.

    GregFreedman: if you put device pixels on the Screen object,
    you'd still have the problem of the difference between the
    video and graphics plane. Would you have device pixels for both
    planes?

    GregFreedman: I kind of liked where we landed with the Media
    Queries. what we want to do is say: can you support this
    resolution on the video plane; and that seems like this works
    with Media Queries well.

    GregFreedman: is the problem that CSS doesn't like the unitless
    query?

    florian: yes, CSS as a language could support it, but if we're
    on a device that does not have a separate plane, we probably
    want width: to do the same thing as video-width:

    florian: additionally, that definition also affects print media

    florian: so what should you do on print media?

    <Zakim> nigel, you wanted to ask for a reminder of the use
    cases that remain if we use MediaCapabilities to choose which
    video to get

    nigel: the use case is originally to detect what variant to
    display. If MediaCapabilities is the best way to query for
    that, what's left in CSS ?

    GregFreedman: the device may be capable of doing 4k UHD, but if
    the display is only 1080p SD, there's no reason to pick that
    variant. so we split it into two APIs: one to query decode, one
    to query display.

    ving: MediaCapabilies also is not screen aware.

    nigel: could Media Queries handle that scenario, where there's
    multiple displays?

    hober: we already handle that with the 50%+1 to determine which
    monitor's properties to return.

    chcunningham: that's the implementation in chrome.

    cpn: can the UA play this particular content is answered by the
    combination of screen and decode capabilities.

    cpn: asking these two questions separately doesn't give a
    complete picture of whether a variant can be successfully
    played

    GregFreedman: Media Capabilities (MC) asks "can we decode this"
    and Media Queries (MQ) asks "should we decode this".

    nigel: is the set of things we need to know just the video
    plane resolution, and the frame rate?

    nigel: are there other properties that we need to include?

    florian: those do not yet exist?

    <Zakim> wolenetz, you wanted to say that decoder capabilities
    exceeding display capabilities shouldn't limit what is
    decodable

    nigel: just wondering about design and what info needs to be
    included.

    wolenetz: Media Capabilities already gives you the answer of
    whether you can decode a given piece of media. Do we try to
    move some of the properties in Media Capabilities into Media
    Queries?

    mounir: oginially there was information in the Screen object.

    wolenetz: since Media Capabilities is already being asked about
    candidates, maybe Media Capabilities is the best place to ask
    more questions

    wolenetz: since I'm already using Media Capabilities, doing
    that again against a different API is difficult.

    chcunningham: I have the exact opposite opinion, respectfully.
    the Screen API and the Media Queries API already exists.

    chcunningham: we never considered putting Media Capabilities
    queries into the Screen API

    mounir: websites already will react to window size changes,
    e.g., when going into fullscreen.

    jernoble: we want to answer the question whether the video
    should be upscaled or downscaled

    jernoble: so maybe we should design an API for that?

    smfr: I would be reluctant to add new APIs to the Screen api
    due to fingerprinting concerns.

    <Zakim> florian, you wanted to suggest that the JS API could
    take the frame rate as input more easily than Media Queries

    florian: if the frame rate matters, it would be more amenable
    to a Screen API, since you can pass in parameters to that API,
    but that style of query isn't possible with Media Queries.

    chcunningham: no option seems to be the clear winner; IMO the
    least of several evils was the CSS Media Queries options.

    <nigel> CSS Media Queries don't remove the fingerprinting
    vector

    chcunningham: I wonder if there is a way to fix this problem

    florian: I'd like to ask Simon to clarify this answer about
    fingerprinting

    smfr: it depends on whether it allows you to enumerate all
    screens. and depends on whether the API allows you to detect
    whether users have changed their screen resolution or are using
    non-standard third party screens

    smfr: it's still doable with Media Queries, but you have to
    binary search, so it's not as easy.

    mounir: what's the next step here? shall we move this to GitHub
    issues?

    chcunningham: we haven't discussed any options beyond "fix
    Media Queries" or "move to Screen".

    <astearns> lost my webex connection - would there be a way of
    making a video-playback-specific API that would expose less
    fingerprinting surface but still be able to answer coarse
    questions on what can be displayed? How fine-grained do we
    really need to be for playback?

    chcunningham: maybe now that we've had this meeting the CSS WG
    will come up with an alternate solution that better solves all
    the use cases

    mounir: lets follow up on GitHub; lets try to discuss between
    MediaWG and CSS WG people there

    florian: it's unfortunate that there were fewer voices in the
    CSS WG than I anticipated; there are conflicting desires in the
    CSS WG.

    florian: I'm unable to represent all those various POV.

    florian: but yes, we can explore; either alternative ways of
    speccing the Media Queries or alternative API entirely.

    chcunningham: thank you florian! and thank you for representing
    the CSS WG.

Received on Monday, 20 July 2020 12:25:39 UTC