Interactive Media

09 February 2022


Adam_Sobieski, Bill_Kasdorf, Xabier_Rodriguez_Calvar, Charles_LaPierre, Chris_Needham, Craig_Wright, Francois_Daoust, Gregorio_Pellegrino, John_Riviello, Karen_Myers, Kaz_Ashimura, Leonard_Rosenthol, Louay_Bassbouss, Marisa_DeMeglio, Wendy_Reid, Wolfgang_Schindler, Zheng_Xu
Chris_Needham, Wendy_Reid
cpn, wendyreid

Meeting minutes



<kaz> (Advanced Multimedia Processing Language)

Adam: It's under development, it's based on HTML, JS. Related to SMIL
… It supports sequential content, supports multiple video streams combined into one output stream
… Place logo onto a video
… Choice points include switch/case and goto are in the markup language
… Example of an intro video with menu
… Paths converge on an outro video
… Choice points may not use a menu, but instead be JS based
… Education scenarios, different application logic and user context
… Metadata to provide context for users
… Metadata can describe the context and framing of the choices presented

Attaching semantics to video

Adam: Spatial bounding boxes, auxiliary video tracks containing output of computer vision algorithms
… Animated silhouettes. Can be referred to by URI

Multipart Data Markup

Adam: Can enhance i18n
… We'll consider those approaches on the menu markup language
… Accessibility, multimodality will be considered

Web Remote Protocol

Adam: AMPL servers could render to one or more video streams for HTML5 video or WebRTC
… Stream interactive multimedia content
… More efficient encoding than traditional video
… Instructions for layout boxes, stream back input events for mouse and touchscreen
… More advance than existing remote desktop software
… Can be used for optimising WebRTC
… 3D game engines and CAD
… Can stream, modify, delete layout boxes
… Layout boxes on top of the video could contain hyperlinks
… Layout tracks can use new protocols, style and position, animate on the screen


Adam: Interactive fiction is a use case


Adam: Could focus on hypertext overlays, e.g., lower thirds
… Example of two video streams, replace overlay video with hypertext

Live streaming

Adam: Important use case, e.g., live news ticker


Adam: Speech recognition, could ensure interop with simple or advanced dialog systems
… Synchronisation between resources enable chatbots
… Synchronsation with dialog systems involve time-based metadata
… The CG is interested in all forms of interactive media. The group discusses how standardisation can help
… I'd like to invite your participation

<AdamSobieski> https://docs.google.com/presentation/d/1NwIJaz-Lin8WNmop_fjkvjaJr7L_P2-iSgrDUSsCO00/edit#slide=id.g1135e4528bf_12_14


Zheng: It's interesting to hear about AMPL. It's trying to solve a lot of problems. What are the issues with current HTML5 or digital publishing that are solved?

Adam: AMPL is a general purpose tool. It uses HTML and CSS
… I've been focusing on synchronising clients and servers, animations on top of videos for clients. Hypervideo

Zheng: Any existing prototypes?

Adam: Not yet, it's quite new

Kaz: It seems to me that this possible too generic. There were several generic frameworks in W3C such as MMI Architecture and SCXML.
… I'd like to see the typical target use cases and a gap analysis with existing frameworks, to identify what's expected and required
… And then see what kind of concrete data model is needed

Adam: Interactive video is a first use case

Kaz: You've also input to the WoT group, there's potential integration with video content and hardware for playback or recording. Could think of that kind of use case to identify requirements

Leonard: Interesting presentation. Have you thought about how the group might try to choose between potential directions?
… Any one could be a significant investment. What are your thoughts on the process?

Adam: We'd use prefixes on the mailing list to distinguish topics. I'm hoping to work on the menu markup language

Leonard: This assumes AMPL is a foundational technology

Adam: Optimistically yes, needs a markup for menus and multimodality. I'm putting the technology forward as an option

Marisa: I'm working on accessbility in multimedia. Could you switch on rich content such as caption tracks. Introduces authoring complexities. Where does that sit in the plan?

Adam: Focus on menus

Wendy: Does the CG have a use cases document yet?

Adam: Not yet, that would be something to do early on

<Karen> +1 use cases document

Wendy: I think that would help to help alignment

Zheng: Interactive video is interesting. Netflix and YouTube. Users can choose a story, is that a vision here?

Adam: Yes, that's a use case we can include

cpn: In my org we've been working on a technology called object based media
… I think there is some overlap, branching narratives, responsive overlays on videos
… more adaptive to devices
… have you established any other connections to communities interested in these topics?

AdamSobieski: I've reached out to Twine, an interactive fiction group
… at BBC there's a tool called Story Former, interactive video and film, I'm new to that community as well

Craig: Use case of accessible menus and choice points
… have you played around with implementing that in modern browsers
… what limitations did you encounter?

AdamSobieski: I haven't started yet aside from visualziing
… it's just getting started, no foreseeable obstacles
… just a standard way to make a menu

zheng_xu: In the presentation, in some parts it looks like there is overlap with SMIL
… how to map text or other media to audio to read to the user
… have you looked at this?

AdamSobieski: We would definitely be referencing the SMIL standard in our documents
… some of the differences include using JS and CSS

cpn: Another question, where we see needs for interoperability
… one aspect to consider, limitations in existing browsers for playback and rendering
… for things like SMIL, are you thinking we end up with a standardized runtime environment for interactive experiences

AdamSobieski: I was considering an AMPL player that uses all of the methods I presented, reference implementation for an AMPL player

leonardr: You mentioned before, one of the big issues in publishing land is around security
… one of the concerns there is presence of JS in EPUB
… presence of scripting in EPUB is considered problematic
… AMPL would use scripting, do you see it as required or optional?
… or declarative

AdamSobieski: In the example with a switch case, you could use declarative markup for the spine in an EPUB
… a non-linear graph
… Twine makes interactive fiction, I would be advocating for some JS, but I understand your points

leonardr: I look forward to those conversations

Craig: It feels like the logical first step is bridging the gap between the current standards and what we need
… review the current browser implementations and then use this as a use case to push some initiatives ahead
… a layer on top to address some of the security concerns

AdamSobieski: Looking forward to feedback from others

cpn: Where do we go next? What is the status of the CG?

AdamSobieski: The group is new and ready to join
… it's currently growing and we'll be working on structure and chairs and use case documents next

cpn: Could you share a link?

<AdamSobieski> https://www.w3.org/community/interactive-media/

cpn: one thing I might recommend is to set a charter to establish a scope and goals for the group
… it seems broad, and drafting a charter can help scope the work
… there is a template somewhere


AdamSobieski: I think a charter will be a useful first step
… in there we can describe the process for chairs, co-chairs, work items
… use case documents, AMPL, menus, and other interesting projects for standardization

zheng_xu: If there is anything the Publishing CG can help with, please let us know
… it's more focused on work with the EPUB3 WG and incubation
… use cases and documentation for digital publishing

AdamSobieski: Thanks

cpn: I'd like to echo that for the Media and Entertainment Interest Group, we are interested in media related use cases
… and also from the Media WG point of vieew
… if there is a gap analysis in the browser APIs, we can take that forward as well
… we welcome this new group and want to keep in contact

gpellegrino: I joined a little late, so you might have answered, I work in accessibility
… do you see use cases for accessible interactive media, aside from media overlays
… do you see possible accessibility issues
… anything that is not just text poses problems

AdamSobieski: My own interest pertains to education and educational technology, so accessibility is important
… speech synthesis and speech recognition, accessibility is key to our work

Charles: I'm curious about interactive video, having a live transcript that you can click on to select words or phrases and go to that part of the book and highlight it
… Work that Kim Patch has donw with WordPress plugins

Adam: Interesting, I can see how that would help with multimedia search. Something to consider
… We talked about dialog systems and synchronise to the video. There might be some overlap there

Wendy: What are the next steps? Sounds like you have a good idea of what to do with the CG
… I can share the link with the publishing community

cpn: Craig, any communities we can connect Adam with?

Craig: There's the BBC MakerBox community. This CG could be a helpful point of contact for us

Adam: Thank you. Everyone from MakerBox and StoryFormer is welcome

We've/Craig: We've We've published some things, got people to look at it, but need to look at the tech to support it


Minutes manually created (not a transcript), formatted by scribe.perl version 185 (Thu Dec 2 18:51:55 2021 UTC).