Meeting minutes
Foldable use cases and requirements
Web support for new device features
anssik: Today's meeting will be focused on exploratory work in the SSCG and in related groups.
… I have asked Srikant to talk to us about new form factors in upcoming devices,
… how they should be considered in new Web APIs.
Srikanth: Presentating on Web Support for New Device Features
… Long history in VESA, HDMI, Wi-Fi display, other bodies.
… There are multiple new form factors. Foldable displays with multiple independent display panels.
… Ranging from laptop size to smaller sizes, like a mobile phone.
[ Srikanth presenting slides ]
[ MID = Multiple Internal Displays ]
Srikanth: Intel Horsehoe Bend, at CES earlier this year. Supports multiple configurations: laptop, table-top, flat, canvas or book
… Depending on the amount of incline, you can lay it all the way flat, for input or drawing.
… Your user experience will differ, and the camera and speaker are reconfigured accordingly.
Srikanth: shows a video demonstrating capabilities of the foldable laptop.
… This is not a new concept. [Shows a slide of the Lenovo X1 Fold on the MID platform]
… Even if you have a MID, the emphasis is on fold. Both are posture-based. User experience and resources will differ.
… Many of these are supported in W3C today, but how to provide a seamless, consistent experience.
… Consistency in how Web applications determine what the posture is. Posture enforcement as necessary.
… Want this regardless of foldable vs. MID.
… [Shows a device with a physical hinge]. This device can with a sharper fold; can have a 360 degree bend.
… Devices without hinge, there is a less acute fold because of the physical layers of the display. Can't completely fold (0 or 360)
… Number of displays, we are seeing two, but there could be more. Could be panels of unequal size, or three different panels.
… Do the interface once, have these webapps work and be foreward compatible.
… Foldables, there is typically one display. It depends on the display configuration, could be one or more depending on the posture.
… Seam is always present on a MID. In a foldable, there is a virtual hinge, it's a user experience decision.
… The beauty of a virtual hinge is it's a software abstraction.
… Touch controller is associated with each panel. Is configurable to generate various HID reports.
Srikanth: [Table showing attributes of MID vs. Foldables]
… If you have an enlightened app, aware of the seam, it can adjust layout according to the seams in the display.
… This is an example of how a seam can help the web app to choose.
… [Slide shows how apps could customize themselves for different foldable postures]
… Key point is, apps can benefit from other information, posture, occlusion, etc.
Alexis: New UX patterns as you use different postures on the device.
… Show you kinds of problems you might get on web applications. We designed a gallery to respond to multiple displays.
… Not to just adapt their app, but based on posture, provide a better experience.
… Don't want to position across the hinge.
… The current thing that we see, is covering the new form factors.I need the screen fold API to determine
… If I am in tent mode. They are complementary, but I can't use them in a consistent manner, across the OS.
… Next steps, MID are different from foldables. Different from an externally connected displays.
… Whatever specification we put up, should cover these types of devices.
anssik: The future looks quite exciting, when it comes to new form factors. We should be designing new APIs in a future-proof manner.
… This is the beginning of the discussion, but want it to be a kick-off for this type of exercise.
Mark: certain type of single displays can do virtual multiple displays, how does this work out with that, has that been considered a use case?
Srikanth: That is a use case as well. For example, you can use docked devices.
… You can have software abstractions that virtualize the hardware. I fully expect the platforms to have multiple docked scenarios.
zouhir_: We have an API to get the geometry of each physical display, an API with an enum of postures.
… Developer has to use two APIs to get geometry and posture.
Alexis: Edge doesn't allow me to get the posture, that's the inconstency.
… : Incomplete implementations, by having them separate, some browsers may not implement one or the other.
Srikanth: Many low-level details are exposed to the application, applications have to put them together to get the big picture.
… Different applications may interpret them differently. What sort of things are okay to allow some inconsistency.
msw: Do you wish to generalize the foldables API to handle additional use cases? Like MID / 360 hinges?
Srikanth: Given all the possibilities in the underlying hardware. What is the right abstraction to expose to webapps?
… Tomorrow, if there are 4 internal panels, foldable in different ways, do we want to re-do some APIs, or create full abstractions?
kenchris: It has to be simple as possible for the common use cases, or developers won't adopt it.
anssik: Try to understand what are the experiences they want to build, and optimize for the common scenarios.
… Make the less common thing possible, but maybe not as easy.
Multi-Screen Window Placement
Multi-Screen Window Placement TPAC 2020 update (also available in PDF)
msw: [Shares slides]
msw: This proposal is related to foldables. Addresses a generalized set of use cases.
… Generalize Web Platform for multi-screen displays.
… [Slide comparing existing Web APIs and goals of MSWP]
… [Slides showing examples of multiple-display devices, projectors, signage, lots of use cases]
Goals: Increasing initial information about additional displays; adding a change event when they change;
… Request additional per-screen info; sites can request fullscreen on another window
… Slide shows current properties exposed through CSSOM
… Proposal is to allow sites to request additional information about connected screens.
… Currently you can use window.open to create new windows, with left/top coordinates
… Can move to/by (x,y) coordinates
… Crux of the proposal is to allow cross-screen coordinates for these operations.
msw: One set of updates is that we have an ongoing origin trial with prototype code.
… Addressing some good feedback, including some API shape changes.
… There is a demo: https://window-placement.glitch.me
… More info: https://web.dev/multi-screen-window-placement
… Github: https://michaelwasserman.github.io/window-placement-demo
msw: We have interested partners in this, they want to explore a new space.
… We have Chrome apps APIs and OS APIs, we want to offer the same affordances to the Web.
anssik: You are catering for the established usages. Your use cases are more clearly defined.
… For foldables, it's not as clear.
… How much we can unify MSWP, and some of these new proposals. It might not be 100% overlap.
Eric: What you've done to mitigate the many bits of entropy about users with non-standard configurations?
msw: Similar information is exposed if windows on placed on multiple screens. We are also proposing a permission that user agents can use to gate access.
Eric: What is exposed without a prompt?
msw: A single bit whether there is an external screen, so sites know whether to prompt multiscreen users.
<anssik> Mark: Presentation API is slightly different in terms of availability detection, the Presentation API does not expose any info on external display, Remote Playback is similar
Zouhir: Want to work to create an API surface that is minimal, can talk again about how make these APIs work together.
… Regarding privacy, if permissions is the way to protect customers, and I want to make sure my YouTube controls are laid out right.
… Don't want to show a bad experience, then require a permissions prompt to show a good experience.
msw: Good opportunity to collaborate on an API surface.
… Permissions are tricky, Web platform was built so that users can see screen info. If you install an app, it can access screens and system info.
… Possibly we could expose screens that a application is spread across, without requesting permissions.
anssik: Let's create follow-up actions through GitHub
Window Segments
Window Segment Enumeration Script API (PDF)
Zouhir: [Presents slides]
… Window Segment Enumeration Script API
… Targeting dual screen and foldable devices.
… Surface Duo/Neo. 2 physical screens, 1 fold.
… Samsung Galaxy, 1 screen, 1 fold, 2 logical screens.
… Email app example, responsive on a dual-screen, the content is not laid out in a user-friendly way. Snap content to fold.
… CSS API to help developer lay out. In case CSS is not available for games and canvas, the JS API is handy.
… Simple, one method. getWindowSegments() returns a list of segments.
… API is an immutable snapshot, need to listen to resize events to know if it's invalid.
… Difference between dual monitors and foldable. App launches is not consuming the full screen on desktop.
… Maximizes, does not span dual monitors, but does on foldable.
… Users want to avoid alerts, dialogs on fold. Want to move it to one side or the other.
… Customers want to focus on portable form factors.
Zouhir: Don't want to create new, redundant APIs
… Don't want a lot of API surface for developers.
… Outlook, e.g. not requesting multi-monitor spanning behavior.
… Want the Web to be prepared for new form factors, versus sniffing around to optimize for them.
anssik: Start with the core use cases, and incrementally ship new features. "When in doubt, leave it out"
Zouhir: If use cases are different, e.g. multi-window applications versus app layout. Think about naming it carefully if we decide to keep APIs separate.
msw: The premise of segments, is potentially polyfillable given the intersections of screens and windows.
… Are foldables doing something different than the desktop display paradigm?
… There is a more general set of primitives, that could provide more information, that people could build libraries for different use cases.
Zouhir: We can use one API with more complex information, developers can optimize it. If I have to do calculations, developers will build a support library.
… What if I am on desktop, and I span my browser. I feel like users don't expect layouts to change. Would there be surprises for users?
… Looking into how we can make information available, without requiring permissions.
kenchris: It is hard to polyfill CSS. You might want to use an iPad as a secondary screen, and not be cut by a seam.
Eric: What do you do to mitigate the entropy that this API exposes about a non-standard configuration.
Zouhir: Right now we don't have permissions, developers can know a lot about the device form factor already. We have a policy to prevent iframes from getting info.
… Gating it by a permission is providing a bad user experience at the end.
Eric: Agree that a user prompt isn't useful, but for perspective, we have device enumeration for getUserMedia.
… When we looked at how they were being used, we found that enumerateDevices was used more than 2 orders of magnitude times more often than getUserMedia.
… It was being used solely as a means of fingerprinting. Now, enumerateDevices returns a lie, until the user has given consent to capture.
… We got feedback that it was a change that had to be made. You'll have to have a compelling argument that the bits of entropy are worth the cost.
Zouhir: We don't want developers to be able to achieve something natural for this class of device. Exposing the device hinge is not a big risk.
… We can't lie to developers about the hinge position. We have to tell them something useful, position and dimension of hinge.
anssik: What extra info does surface duo share, versus e.g. iPhone max?
Zouhir: You can infer device from geometry, because there are few foldables on market. For iPhones, developers need to know size e.g. of notch, to avoid rendering text on it.
anssik: We need to differentiate between multi-monitor setups, and other use cases.
… (regarding fingerprinting possibilities)
Zouhir: Foldables are another responsive Web design target, I think it's something developers need to be aware of.
… As a website developer, I should be able to make it look nice. But if I have an ad iframe, it does not need this information.
Alexis: In the case of a dual screen, you have only one screen, so the viewport size is the sum, so without this API you can get the viewport size and aspect ratio.
… We are not exposing anything, because it's viewport / 2.
… We are already exposing much of this information in other ways.
Action: zouhir_ to document fingerprinting concerns and mitigations in the spec
Zouhir: Want to respond to the discussion about libraries. Whether to have something simple, or require a library/NPM.
Action: zouhir_ to open a GitHub issue to discuss use of libraries to provide window segments information
Screen Fold API
kenchris: Some of these devices have a physical hinge, with dual screens.
… Want to change behavior based on posture. Or animate based on fold angle.
kenchris: Expose as a ScreenFold object with a change event.
… Look at different postures that might exist.
… Some devices might have a second fold, or different screen sizes.
… The hinge is not in the middle. Or maybe there are two hinges.
… Looking at what OEMs are thinking of, dual fold devices.
… Want this to be integrated into other specs. Could be used as a secondary screen.
… Want to use the screen on the table for video editing.
kenchris: Important to standardize when things are in certain postures. What angles are they using? Talked to Samsung, Intel.
kenchris: Can be used in combination with other APIs.
anssik: Are we able to handle multiple, external monitors, which might be foldable?
… To test how prepared we are for the future.
kenchris: We need to integrate with Window Placement, and handle multiple folds.
… I could imagine having the Intel Horseshoe, and multiple external monitors.
anssik: We may be able to handle this API in the future, how does the future look like? Don't paint ourselves into a corner.
msw: When there are multiple integrated displays, if they are separate Screen objects, how would a single fold between those screens be supported?
kenchris: We have seen this as one virtual screen. Are there any devices where the hinge separates the screen?
… See if there is a hinge, but not screens right next to each other.
Alexis: On the surface Duo, it will be two physical screens, two logical screens.
anssik: Does the screen fold API address traditional form factors, that might have a fold sensor?
… Does this API report the angle?
kenchris: Not sure if there is a sensor for these devices.
kenchris: Are there use cases for supporting this?
msw: Anecdotally, laptops and Chromebooks transition into tablet mode, when they fold beyond 180?
Zouhir: Window Segment / Screen Enumeration have a similar concept, but different use case. Foldables API, complements these use cases.
… Which way do we want to think about merging Window Segment.
kenchris: If we have a laptop with a hinge, that knows when it is converted. It's a different screen size, is there a use case for traditional laptops.
… If so, it doesn't make sense for foldables.
anssik: Do native APIs expose hinge angle? If no, then Web content doesn't require it either.
Alexis: There is a sensor for the Pixelbook. It it's used to resolve the media query for hover/touch.
… Web application can adapt to be touch friendly.
kenchris: That seems like a way to optimize for primary input. Also, on some devices, the granularity of the sensor varies.
Alexis: Some devices with a stylus always report that a stylus is available, so input signals are not a good way to detect tablet mode.
Aligning Window Segments, Screen Fold, Multi-Screen Window Placement APIs
Zouhir: We have three APIs, two similar in concepts, two similar in use cases.
… Don't rush and commit, open to ideas.
anssik: Discuss opportunities for aligning. Make sure that these people are watching each others' work.
… Useful to cross-link specs, that Screenfold and Window Segments are addressing similar applications.
Zouhir: Agree. We should add the foldables APIs to keep them in those use cases.
Action: zouhir_ to cross-link foldable-related specs together
Zouhir: When I am dragging my window across two windows, how can web developers decide not to opt-in the foldable layout?
<kenchris> https://github.com/SamsungInternet/Explainers/blob/master/Foldables/FoldState.md
<kenchris> https://w3c.github.io/screen-fold/
msw: There are currently no window events on move. It would probably be valuable to have a move events on window. Maybe expose segments after the move loop is complete.
… Maybe only surface segment information if the window is maximized and spanning both monitors.
Zoushir: Different devices have different granularity, with respect to resizing. Also only maximizing, may exclude Samsung use cases.
… Desktop always be opting in, unless you opt out with media queries, viewport size.
msw: Part of the screen information exposed, could be whether the screens are physically adjacent.
Action: msw to reference complementary proposals of Window Segments and Screen Fold, Visual Viewport, open appropriate GH issues
<tidoust> [As part of cross-referencing specs, alignment on terms could be useful (and easy) too. I heard Alexis talk in terms of logical/physical displays. Screen Fold uses "screen", Window Segment uses "display" and "screen", Window placement uses "connected screens"]
msw: A breakout session could be the best way that we have something concrete to look forward to.
Action: msw to (with help from kenneth, zouhir, dlibby) organize a TPAC breakout session on aligning Window Segments, Screen Fold, Multi-Screen Window Placement APIs