Meeting minutes
Welcome
cpn: Hello everyone, welcome, thank you for joining.
… We have some new participants
… As an interest group we do pre-standardisation work,
… looking at use cases or requirements, potential improvements to the web platform or
related web technologies,
… where we might want to initiate incubations or affect work happening in WGs.
… That's our scope and remit.
… Today, we'll look at timed data and locations, cartography, maps etc.
… Especially the HEIF format.
… Before then, updates from the group.
Next Generation Audio codec support, and draft explainer
cpn: First, thinking back to TPAC, we discussed next generation
audio codec support,
… a use case we've been discussing for a while.
<cpn> https://
cpn: As a follow up, Wolfgang and Bernd wrote an explainer document
that goes into the use cases.
… Wolfgang, what's next? We have a draft but haven't discussed it in a meeting.
wschildbach: We'll want to call for consensus at some point.
… On this document I've not seen any feedback yet.
… It tries to pull together all the questions and feedback from the meetings so
far.
… For example the gap analysis is much expanded, to capture some contenders for this
functionality,
… and why we think they're not quite sufficient.
… There's a new section called Privacy Considerations.
… I think Bernd pointed out there could be some interesting privacy challenges.
… Other than that, I'm not aware of other content that is needed, but I'm happy to hear from
the group.
… If the group is happy we should go for Call for Consensus.
cpn: Please do take a look and raise issues there, or contact the
Editors directly.
… Perhaps we'll arrange a future call to walk through it in more detail,
… and at that point we can get the consensus.
… Please do review.
… Related to that, there's an incoming liaison from 3GPP to Media Working Group, about
a
… format they've been developing called EVAS.
<cpn> Nigel: You mentioned a CfC, what would that involve? What would be the plan, create an IG Note?
<cpn> Wolfgang: That's exactly it, it would be to publish as an IG Note
Nigel: I also support the idea Chris mentioned for a walk through.
wschildbach: We can talk about what to do next when the walkthrough comes around
cpn: Let's do that, there's limited time today.
New media API proposal: Decouple audio playback from document lifecycle
cpn: The other topic I wanted to mention today, a use case that might be interesting to people here,
<cpn> w3c/
cpn: again in the Media WG GitHub repo, someone raised the case of
decoupling audio playback from
… document lifecycle, so that if you're building a website and you want continuous audio
playback
… while people navigate around your site, how would you do that?
… You might create a popout window, which has limited usefulness in mobile
devices.
… The person is proposing a particular solution. It's not the first time the use case has
come up.
… Other solutions have been proposed.
<song> might need to correct EVAS to IVAS, Immersive Voice and Audio Services
cpn: Flagging in case this interests you for your own website, as a
potential new capability.
… Next step would be to pull together a similar use case document and evaluation of the
potential approaches.
… Again I wanted to flag, not for detailed discussion today.
… If the use case interests you please leave a comment on the issue, to support.
<cpn> Nigel: The BBC has done this not so long ago, for BBC Sounds, they changed their UI to navigate the page while listening
cpn: That's it for general updates.
High Efficiency Image File (HEIF)
RobSmith: I've been involved with OGC Testbed.
… My interest timed video metadata.
… Last couple of years: HEIF for still and motion imagery.
… Timed metadata is part of that, which is my interest.
… Group perspective: new features for tiled images for online maps.
… To allow zooming in to get more detail, and storing and accessing it efficiently
online
… so that those tiles are presented quickly and its a smooth experience.
… Also been thinking about what else could be done with that.
… Feature requests for HEIF support in all the browsers.
… Mozilla, Chromium and Webkit.
… We've been thinking about the benefits for the web,
… if we can build on the existing functionality and add our innovations.
… I'll let Joe explain what those ideas are.
… Interesting that benefits for still images come to motion, and vice versa.
Joe: [shares slides]
<cpn> https://
Joe: I'm a member of the Geospatial-Intelligence Media Standards
Board in the US,
… to promote worldwide standards for all kinds of applications.
… I work with Dmitri Podborski who is in MPEG too, who is on this call
… [ISOBMFF slide]
… ISO BMFF is a foundational architecture for still and moving media
… In the browser space there's an opportunity for the MPEG and JPEG standards for a cohesive
structure
… for delivering efficiencies if you use all the capabilities within the family.
… Today, HEIF standard is the focus.
… Additional capabilities: different formats that do different things.
… CMAF allows us to do streaming video.
… OMAF is an emerging capability for streaming immersive media, 360º etc.
… HEIF is a component of that overall architecture.
… [GIMI slide]
… We've made an extension to integrate structured semantic metadata with our
media
… so we can do reasoning and inference across media sets.
… We participated in the OGC's gathering last year and this year.
cpn: With the GIMI format, is it embedded metadata within the
ISOBMFF box structure or something
… outside the image file?
Joe: You can choose either. With respect to RDF you can carry it
inside the file but that limits
… your ability to interact with it. It's more a wrap and carry mechanism.
… Sidecar RDF files would let you query and SPARKL the data sets
… [Overview of HEIF slide]
… A lot of people viewed it early on as a JPEG replacement.
… It's not just a compression format, also the container format.
… Leverages ISOBMFF.
… Can share underlying software that's common across the formats.
… It's agnostic to the image codec, uses a plugin module.
… Supports JPEG 2000, AVC, AV1/2, HEVS etc
… With the GIMI format, we're intending to heavily leverage uncompressed JPEG
2000
… When you look at adopting HEIF compared with JPEG, PNG, TIFF etc there's a lot more
capability
… with HEIF and you get to leverage other codecs when they come out.
… HEIF itself is independent of them.
… [Key Features slide]
… Focused on still imagery.
… Support for image collections, like collages
… Video like features - image sequences, intraframe and predictive frame modes,
… burst images.
… Broad metadata support - high precision timing - nanosecond TAI timestamps.
… May be critical for some applications.
… Derived image types - grids (tiled images), image collections/overlays
… Transformative Item Properties - instructions for rotating, mirroring, scaling
transformations without having to store modified images
… Descriptive Item Properties - metadata tags
… Image Tiling - image grids, pyramids etc like Google or Apple Maps uses.
… Took advantages from GeoTIFF and built capabilities into HEIF and extended them.
<cpn> https://
Joe: Pyramids allow zoom in/out and pan around efficiently over the network.
Joe: Caching tiles
… Region items and annotations, labelling features in images.
… Bit efficient mode, supporting "tiny" web images, icons etc. minimising the footprint of a
single file.
… Supports media encryption like MPEG's common encryption standard.
… [use cases slide]
… Geospatial applications - satellite imaging, weather and climate research, disaster
recovery.
… Efficient access over the web is important in timely situations.
… Basic web applications - custom image and image sequence downloads to match a
device.
… Access to the same media at different resolutions, bit depths etc.
… Download only appropriate media via an http range request
… 3D models
… Frame accurate controls and nanosecond timestamps
… Ability to combine still and video imagery in a shared file supporting a web
page
… Bit efficient carriage.
… [use cases cont. slide]
… High Dynamic Range (HDR) imagery
… Native support of 14 and 16 bit pixels.
… AV2 and 16 bit video has tremendous benefits in search and rescue, in visible or infra
red.
… May have low signal to noise ratio, going from 8 bits to 16 bits can mean the ability to
see someone or not.
… Medical imaging - MRIs, CT scans, X-rays. Ability to compress at full dynamic range and
provide to a
… specialist in a different location can be a life saving thing.
… Virtual pan & tilt streaming in a 360 degree camera system dataset.
… [Metadata Support slide]
… Metadata is significant. Plug-in model for HEIF and ISOBMFF in general.
… Asynchronous and synchronous timed metadata within linkage to timed imagery.
… Metadata can be attached to individual samples in a video track,
… supports a broad range of use cases.
… And sidecar files for flexibility.
… The coupling between media & ontology structured metadata will help benefit from
merging AI/ML with media.
… Example: big events like Olympics and World Cup, the ability to do reasoning and inference
on that
… data in near real time, and pull up data related to events that have just occurred is
important,
… and HEIF supports those capabilities.
… Looking to harmonise semantic metadata based on what W3C already has in sensor ontology
documentation.
… [HEIF Image Pyramids slide]
… A lot of complexity under the hood. Image pyramid allows byte range access to individual
tiles.
… Very efficient navigation. Images can be uncompressed or compressed.
… Could keep base layer as uncompressed, highest resolution, and then upper layers encoded
for
… easy navigation with minimal hit on storage space.
… [Tiled items slide]
… Dough Farin (libheif developer) made a generalised form of image tiling.
… n-dimensional tiles that can include a time component.
… Submitted by GMSB to ISO/IEC 23008-12
… Currently in final draft of HEIG Ed 3, with functioning software available.
… [HEIG Tiled Image item structure slide]
… ftyp box containing a meta box, containing a location box. Can store files inside the
HEIF
… or externally with defined byte range requests on some URL.
… Can create tags of metadata that attach to each file.
… Might want a different caching scheme, so pointing to separate tiny files is a flexible
alternative.
… [Ontology Structured Metadata slide]
… HEIF file with semantic metadata.
… Leverage the standard RDF structures used in the ontology world - a triple store for data
graphs.
… How to associate a triple with a piece of media - link with a "Content ID".
… Can tag a piece of media with a Content ID and a TAI timestamp, so can use that to
associate the
… metadata to that piece of media.
… [Making the case for HEIF in the browser slide]
… Use cases currently addressed by multiple, stove-piped formats can be harmonized into a
single
… standards and software architecture. Adoption of HEIF is future-proofing.
… It's a container not a codec, so doesn't go obsolete.
… Allows you to take advantage of codecs without difficult licensing challenges.
… Similar case for us as for browsers.
… Opportunity to create new capabilities over what exists together because of the new
features
… that the MPEG committee has baked in.
… It's not a brand new capability. Some browsers have some level of support.
… Small evolution over existing ISOBMFF implementations.
… Are there other groups in W3C we should be engaging with?
… [Recommendations slide]
… We would love to see adoption of HEIF as a default browser format.
… With some specified codecs as a starting point.
… [Contact details slide]
cpn: Thank you for presenting this, fascinating.
… Any questions from anybody?
wschildbach: I understand you are asking for adoption of HEIF as a
format.
… I'm not clear what that would actually mean for W3C or this group to "adopt" the
format?
joe: One of the issues is, we'd like seamless interop across
browsers.
… It's a question of interop with all of these use cases.
… You can drop a JPEG into a browser and it shows up.
… If you do it with a HEIF file it may or may not work.
… As applications get more complex, we want plug and play capability with many of the
features.
<kaz> caniuse for HEIF
cpn: Interesting, I looked at caniuse for HEIF and it says Yes in
Webkit and No elsewhere.
… My guess is that it's a low level of support, not more advanced functionality. Is that
correct?
Joe: Yes, some level of support that will vary across different
browsers, but we're really beginning
… to talk about advanced features, without needing to build your own page.
Nigel: With a wrapper format like this, and flexibility with codecs, you end up pushing the compatibility issue down the road, e.g., if the browser doesn't understand the codec. How to deal with that?
Joe: You have to look at the combination of HEIF and codecs
… Some codecs have desirable capabilities that can serve broad needs
… Whether it's uncompressed, which has no licensing issues or something like AV1/AV2, more
open, or JPEG2000
Alicia: What features would you expect from browsers in HEIF?
… You describe the format as powerful, and there are indeed many features. Maybe not all are
equally important. An advantage of mainstream formats is that they're simpler. Do some
subsetting?
Joe: Having the browser understand how to navigate pulling the content in, byte range addressing, for the pyramids and tiling
Alicia: I'd like that for other use cases, photography
Dmitri: On caniuse, if I search there for AVIF, I see lots of
browser support, but only Safari for HEIF/HEIC. But on the container level it's the same
… Straightfoward thing to support the fourcc, don't treat the container as a separate
thing
Joe: Browsers today support h.264, HEVC possibly. A separation
between the container and codec. If the browser can navigate the container, the codec support is a different
story
… Desire for the browser to navigate the format. Then it's a question of whether it
understands the codec or not
RobSmith: What are the next steps, capture comments or issues in the GitHub repo for this group?
cpn: Yes, I think that's a worthwhile activity en route to writing
and publishing a document.
… You asked before, other groups - I'm thinking about the PNG working group - image format
people are active there.
… They may have an interest in this. One of the co-chairs used to work at Google on chrome
and might
… have some insights on their perspective.
… My question is: given the caniuse chart shows no support in other browsers, what is
holding that back?
… I'd like to know what Mozilla and Chromium think about this, even at a basic HEIF level
before more
… advanced metadata use cases. How do they see the format in comparison with
others?
… Then, if we start a document, we can look at other existing web platform features like the
WebCodec
… image decoding API.
… And MSE for video.
… There's something analogous in still image [pyramids] with random access into
videos.
… Instead of demanding a lot in the browser, could handle in the web application layer.
There are
… architectural choices to consider and analyse.
RobSmith: I think Dirk is using WebASM to integrate some features into the browser.
cpn: That's interesting.
… Seems to me that, assuming you want to go ahead and do this investigation, we'll be happy
to support you
… in doing that (co-chair views dependent)
Joe: We want interop in the end, that's the primary goal, whatever makes sense in the approach.
cpn: Next step is to pull in Chris Blume from the PNG group.
… If not him, he might point us in the right direction.
cpn: We're over time, let's wrap up here and follow up separately
offline to plan next steps, Rob and Joe.
… Thank you for presenting! Thank you everybody, we'll be back at the beginning of March if
not sooner [adjourns]