<kaz> scribenick: kaz

Chris: For this meeting we wanted to have a discussion that we had planned for the TPAC F2F meeting this year, but we weren't able to cover at the time.
... The goal is to look at the direction that the industry is going in, and future requirements. So I will invite Francois to give a presentation and lead that discussion..
... But before that, I have 2 other AOB topics.

Media Timed Events TF Update

Chris: A few of us have been working in the TF, preparing a use case and requirements document for events synchronised alongside audio or video streams,
... things like emsg events, in-band and out-of-band events. We have produced a document that I would like to open up to wider review in the IG to get your feedback.
... We'll circulate this via email with a CfC to publish as an IG Note.
... No news yet regarding starting incubation in WICG, I'm contacting the WICG co-chairs.

<tidoust> Media Timed Events

Media WG Charter

Chris: Is there any update?

Francois: I don't have much news to share. We are trying to organize discussions around EME with potential objectors.
... We're trying to test whether there are objections around chartering a group to work on minor features for EME.
... We want to test that with a few people first before widening the discussion.
... We are looking for someone who could organise that, to reach out to people and have that discussion.

<cpn> scribenick: cpn

Kaz: Maybe we can have some smaller group discussion between the IG chairs and team contacts.
... I thought that was approved on the previous IG call, so let's start that discussion.

Chris: Yes, let's follow that up.
... Any other AOB?
... [none]

A perspective for Media and Entertainment on the Web

<tidoust> A perspective for M&E on the Web (slides)

Francois: (Slide 2) The context for this document was that I was trying to take a step back, trying to understand what media and entertainment is, what the industry is doing and where it is going, and what the future might bring.
... It's more of a general discussion that I'm inviting than a specific discussion on a particular topic.
... The objective is to present these perspectives, discuss long term evolution.
... I'll ask a few questions, to see if you're interested in pursuing the discussion to come up with a common vision, and maybe publish as an IG Note, for instance, or whether you think it should remain as a one-time exercise.
... (Slide 3) At TPAC 2017, I shared the roadmap document, "An overview of media technologies on the web"
... The goal of this document was to list different technologies for media,
... categorised as media rendering, processing, distribution, capture, orchestration, etc.
... (Slide 4) The goal was to understand what's been done already, in terms of technical standards, what's currently ongoing, and which are upcoming, being incubated, and also to identify gaps.
... We've maintained this document since last year, so it should still be accurate, it's a good document.
... It hasn't had the impact that I wished it could have, but it's still useful.
... It doesn't tell a business story, that's a drawback. It's was easier to maintain if authored that way.
... It focuses on media, but what about the entertainment part of M&E?
... It was focused on a mid-term, 1-2 year vision, but what if we look further ahead?
... (Slide 5) That's how things started on the "Perspectives on media and entertainment on the Web" document that I shared with the IG some time ago and want to present here.
... I wanted to understand the industry. I started by creating a map, but it turned into a huge graph that became unreadable.
... How to make sense of things? That was the starting point for the document.
... It defines useful terms, provides an overview of the media and entertainment industry,
... It describes the current status of media on the web, lists some trends, tries to imagine the next big thing, and from that looks at high level requirements.
... I wanted to share it with you, I have talked to some of you already about it.
... I may have misunderstood some of what people said, or added my own bias, so I'd like to get your feedback.
... (Slide 6) Regarding scope, looking at the M&E IG charter, the group is chartered to work on continuous media, defined as audio, video, and associated technologies such as timed text.
... It's very media focused, of course, but it seems to exclude some of the experiences I think would be worth exploring, e.g., games and immersive experiences.
... Looking at it, I tried to refine the scope.
... (Slide 7) One thing you could say is excluded from the scope is this kind of experiment that some of you are doing. An example is BBC "we wait", a VR experience, a way to tell a narrative where you are following a story about migrants.
... It gives you another perspective, tells a narrative. It's not a video, it's a 3D environment, so it doesn't necessarily fit our definition of continuous media, but it's still a media experience.

Mark: As I remember, we used the term "continuous media" to the charter to be able to distinguish video oriented things from other kinds of media, book publishing, for example.
... Someone suggested that the term continuous media means continuously updated pictures, like video. That's why we used it.
... It never occurred to me that this would exclude games, because they are also continuously updated, and virtual reality, etc.
... I don't think there's any implication that continuous media means pre-recorded, I think it just means a way of distinguishing images, like with M-JPEG, but more specifically i-books.

Francois: That matches my understanding. I wanted to make sure we're all on the same page, I worried that I was going out of scope for the IG, so I'm glad that you've confirmed that.
... (Slide 8) In the document I came up with another term I felt we needed. I called it "continuous experiences", maybe not the best term, but it's meant to convey experiences where the user is engaged, and may interact with the content.
... So the timeline may depend on what the user does. If the user is not engaged, then the content will continue nevertheless. So there's a notion of an internal timeline.
... I'm not proposing to change the scope of the IG, by the way, it's more about refining the scope of the document.
... (Slide 9) I excluded digital publishing, purely interactive games, e.g., crosswords, also exploration content such as maps or scientific data, where you're exploring graphs, because there's no internal timeline.
... I wondered if it would be better to talk about content with narrative, but that would then include books...
... (Slide 10) We went through the media pipeline at TPAC 2017, so this is meant to mean to portray the current M&E industry, different phases of the pipeline. Then you have different content: pre-recorded and live as two main categories.
... (Slide 11) For this exercise, I thought it would be useful to add a third category: interactive content, linked to the notion of continuous experiences.
... This is where games fit. Not pre-recorded, not live, somewhere in between.

Kaz: I think the proposal to include gaming would be reasonable, maybe.
... After the comic publishing workshop in Tokyo, some of the publishing companies have started thinking about moving picture books, with animations inside the electronic books.
... In this case, the border between publishing and e-books and media streaming could be more vague these days.

Francois: I agree. It's also more fuzzy when you look at the Audio group, who are looking at an audio profile for TTML, for example.

Kaz: Given this presentation and proposal, maybe we might want to think about this as a possible input for the next charter period for the IG, as the current charter ends at April next year.

Francois: Yes, we might want to present the scope differently, to be reviewed by the IG participants.
... (Slide 12) The three types of content are linked to three different consumption mechanisms: on-demand viewing (for pre-recorded content), linear (TV, select a channel), and immersive viewing, not in the sense of AR/VR but having an interactive content experience.
... (Slide 13, 14) I also looked at different money flows. Money is spent on production and distribution.
... There's a question about whether to add device manufacturing. This part may not impact the trends I identified, similar for how money is made.
... (Slide 15) I looked at current trends. The first one is about reducing device fragmentation. The media and entertainment industry has embraced web technologies. You see this in interactive TV standards, ATSC, HbbTV, Hybridcast.
... At the same time, the web has moved to an evergreen model, but there's still a clash in 2018 between consumer electronics devices that can't be easily updated due to long production cycles, you can't really update the firmware after the device is sold, so a conflict between that and the web, which changes every day.
... There's no clear definition of the web platform, and that has led to the definition of different profiles of web technologies. Hopefully moving to an annually updated baseline, being done by the CTA WAVE project and Web Media API CG.
... Web Platform Tests come to the rescue here, they provide a better definition than listing specs, by defining test that need to pass. This is an interesting trend.
... (Slide 16) Another trend is improving the technical quality of content. HDR, wide gamut color, 4K, 8K, audio spatialization.
... These aspects are usually a lower level than W3C, more on codecs.
... (Slide 17) Another trend is a general move to IP, a long running trend, first for distribution. This has changed the architecture of the web and networks with the promotion of CDNs for content caching close to the user.
... It triggers work on common formats and distribution technologies. For IP distribution, you want to encode once.
... Also about production over IP, replacing SDI cables in the production studios. An interesting space to look at. It could help with remote production from a central place. It triggers a need for REST based APIs for interacting with components.
... It's also linked to the improvements in networks, fibre and 5G.
... Cloud based processing. Several years ago, at IBC I used to see lots of dedicated hardware, which seems now to be a commidity. These days sees it seems to be dedicated software on common hardware, SaaS, moving to the cloud.

<Zakim> kaz, you wanted to ask who call-in user 6 is

Barbara: I wanted to note that for AI and ML, these are starting to ramp up. There are some media areas that would have relationship to AI and ML.
... Some areas that Intel are tracking for AI and ML in relation to media: content creation/translation, image/photo editing, detection/recognition, style transfer, super resolution, generative: image, video, movie, music, tagging.

Francois: Thanks Barbara, that's useful feedback.
... (Slide 18) The fourth trend I see is towards personalised content.
... There's already a convergence between on-demand viewing and linear viewing.
... Here, when you watch a video, you click a link from social media and then watch the video. But the experience doesn't stop there, the website could propose another video, and build a linear experience from there.
... Similarly with on-demand viewing, you have the binge watching trend, where you watch episodes one after another.
... Broadcasters want to create dedicated user experiences, mixed linear content that's customised and personalised to the user. It's a way to monetization.
... We have a Business Group looking at improving web advertising, personalised ads. It's about splicing media, introducing some hooks for users to interact with the media to create more interactive media experiences.
... That may require ways to extract meaningful information from the media, so ways to understand what the media is displaying so you can understand what the user is interested in. This is where it touches on AI and ML.
... (Slide 19) The final trend is XR and 360 scenarios. In pure media companies it seems more a research space than something in products.
... One reason is that you won't go to movie theatres to see VR content, at least not yet. It's an individual experience, at home.
... The push here comes from interactive content, VR games, and then AR experiences.
... There are some interesting trends around immersive live experiences, that we'll come back to.
... (Slide 20) Are there other trends I've missed?
... Having identified the trends, I want to ask what is the next big thing?
... (Slide 21) To answer that I looked at the previous "next big thing".
... The web already delivered on its promise to become the platform for continuous media experiences.
... The HTMLMediaElement interface, MSE, EME, with WebVTT and TTML for captioning.
... (Slide 22) So what's coming? Combining the trends together, I see there's convergence between on-demand and linear, to keep viewers engaged.
... Immersive viewing and interactive viewing, with more powerful devices with better performance.
... More and more we're interacting with device using natural interaction mechanisms. Voice interaction is making a strong entrance.
... Gestural interfaces, and natural in VR environments. This is being made possible by AI. The democratisation of AI allows real time analysis, a growing area, will shake things up.
... The web allows user interactions to be handled securely.
... On top of that it's an excellent platform for social and sharing.
... The web would be at the heart of the three main consumption patterns I identified earlier, converging on the web.
... (Slide 23) Two main scenarios: immersive live experiences: a stadium, interact with players.
... Choose your own adventure books, movies you can play, where users can influence the scenarios in real time.
... (Slide 24) An example: Red Bull air race in VR.
... (Slide 25) World rally championship. It's no longer a linear experience, you can track the car positions, select camera angles, see stats in real time, immersive experience.
... (Slide 26) Late Shift movie, where you can vote at different scenes during the movie, and it changes the ending based on the results of the vote.
... (Slide 27) In games, Detroit: Become Human is like a movie except you control the characters.
... Also Red Dead Redeption 2, where you feel you're in a movie, but it's a game.
... (Slide 29) High level requirements. One goal is to capture and immerse the user in a virtual world.
... The second is to render immersive content, the frontier between the virtual world and the real world becomes blurry.
... This is what the GPU for the Web CG is working on. Processing content in real time, ML and WASM for performance.
... More natural user interaction: voice control and gestures. They don't exist on the web right now, there's no voice control API, there's speech recognition and speech synthesis, something to explore there.
... Interaction with the physical world (AR), how to detect the floor and walls, other objects.
... (Slide 30) I wanted to see where this could lead. Is this the right level to explore? Are the trends correct? Is this a useful exercise?
... Is it work you'd be happy to discuss in the IG, adopt as an IG document? Or leave it as a one time exercise?

Kaz: A comment on voice control. We can refer to existing W3C related work on multimodal interaction from 10 years ago, and device interaction using WoT currently.

Francois: Yes, also SSML. When I said it doesn't exist on the web right now, I mean that a web application cannot leverage voice interaction today. Speech synthesis is well supported, but speech recognition is not.

<kaz> scribenick: kaz

Chris: Thank you, your presentation is very thought provoking.
... Some this related to work we're doing in BBC R&D.
... We are very interested in the media personalization aspect,
... we have a project that's like the rally car demo, an event with multiple camera angles, lots of additional information, which could be a multi-screen experience.
... It's not just about sport but also concerts and in general.
... Sport is very rich with additional information, but the same can be applied to other domains as well.
... We're interested in personalising content to individual preferences, so adapting the media itself.
... For example, adapting or editing the content to fit the user's time available.
... The system could use what it knows about you, such as your interests or location, etc.
... We have a strand of work called Object Based Media, which is investigating these aspects, and it's something that we'd like to explore in terms of the web platform, what gaps may exist that would need to be filled to support these use cases.
... So, yes, this has been a useful exercise, at least from a BBC perspective.

Francois: That's good to know.
... It could be a good topic for a future call, to present your object-based media research, as a way to perhaps make progress on this front.

<Barbara> Use case key... What and when.

Chris: Thank you, I will follow up with my colleagues.

<cpn> scribenick: cpn

Mark: This is excellent, very valuable, and on target.
... I think this would be great to publish as an IG Note, if others agree.

<Barbara> Roadmap ?

Mark: Regarding how often to do it, we don't have a specific schedule, as a long term view it may not need updating every 6 months. We could update it as things happen.
... As far as what we can do with it, as we have rechartering coming up, we can review this and ensure the scope of the charter is appropriate.
... I think we could go through this in more detail, doing a gap analysis, see if there are things to start work on in the IG or WICG.
... It would be worth another detailed pass over this. I think we'd get a lot out of that, so I'd encourage doing that.
... Regarding the switch to IP, a big part of that transition is also about moving from MPEG-2 transport to IP transport for creation and distribution.
... When you talk about interactivity of media, the choose your own story is an example that comes up, but more widespread is also audience voting where people get to interact with the screen.
... And the issue of enhanced TV, red button in the UK, where there are explicitly engineered parts of the show where you can interact with related content. Voting and interacting with associated content is a growing area, much more than choosing your own story, although all are interesting.

Francois: I think we could run a CfC to adopt the doucment in the IG.
... I don't have in mind redoing this every 6 months or year.
... For me, what's interesting is that if the IG publishes it, it becomes a shared vision, which is what I'd like to end up with, and something we can point people to.

<Barbara> Agree - Trends - What, when and the requirements.

Chris: About how to approach this next, Mark mentioned doing a more detailed gap analysis.
... We could usefully look at what's supported today, and where are the gaps.
... If we're to take the document into the IG, I'd want us to review the content to reflect the membership's view in general.
... You shared this a while ago, but we haven't discussed until now.

Francois: I'm not suggesting we publish it as is. I want to improve on the document before it gets published as an IG note.
... We could do a first pass in the IG, publish as FPWD, get feedback also from the community at large and then public.

Chris: We benefit from talking in person. I wonder if we should revisit on this on a future call.
... Could maybe do the gap analysis, requires preparation, but it could benefit from more conversation.

Francois: That's sounds good, I'm happy to edit the document and shape the next phase of discussion.

Chris: We could do that together between the co-chairs.
... As a concrete next action, let's follow up among the co-chairs, then decide on next stops

<kaz> scribenick: kaz

Song: The discussion here focused on high-tech media production and transportation.
... An alternate view could be on video in emerging markets. I could give a presentation on that.

Chris: That would be very interesting for this group. Let's continue discussion about that offline since we're out of time.

Mark: Please also feed back on the presentation today via the mailing list or our GitHub issue tracker.

<cpn> https://github.com/w3c/media-and-entertainment

[adjourned]