This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.

Bug 9557 - Mouse Lock via Javascript
Summary: Mouse Lock via Javascript
Status: RESOLVED LATER
Alias: None
Product: WebAppsWG
Classification: Unclassified
Component: HISTORICAL - DOM3 Events (show other bugs)
Version: unspecified
Hardware: All All
: P2 enhancement
Target Milestone: ---
Assignee: Travis Leithead [MSFT]
QA Contact: public-webapps-bugzilla
URL:
Whiteboard:
Keywords: needsReview
Depends on:
Blocks:
 
Reported: 2010-04-20 07:20 UTC by Sirisian
Modified: 2012-03-12 23:55 UTC (History)
14 users (show)

See Also:


Attachments

Description Sirisian 2010-04-20 07:20:40 UTC
With the new technologies like Canvas and WebGL it would be nice to get the ability to capture the mouse much like Java does. A user-agent warning could be displayed asking for permission or something with a key to release control.

I'm thinking of this mostly for games like first person shooters and RTS games in the browser or other programs where moving the mouse off the screen is a problem. 

So the basic functionality would be the ability to turn on mouse capture (might need another name, but whatever) then the ability to detect when a user has activated the mouse capture by agreeing to allow their mouse to be captured, a release event if the user-agent has a key to exit the mouse capture mode, and a release mouse capture event that could be triggered to allow javascript to release the mouse if it's captured.

I've seen multiple WebGL people talk about needing it for their projects but no one could find where to submit suggestions relating to this. I'm hoping this is the correct place.
Comment 1 Doug Schepers 2010-04-21 17:52:45 UTC
This seems to be related to an outstanding issue regarding the setCapture/releaseCapture method pair as first implemented in IE, and recently added to Firefox.

I've added these methods to the DOM3 Events spec, though there are some outstanding issues we hope to resolve quickly, regarding which event types this can apply to.

Please review the spec wording here, and let us know if this satisfies your use case:
 http://dev.w3.org/2006/webapi/DOM-Level-3-Events/html/DOM3-Events.html#events-event-type-setCapture
Comment 2 Doug Schepers 2010-04-22 03:25:58 UTC
After further conversation, it seems we were talking about different ideas, so I retract my earlier comment.  Maybe someone else can pick this one up for a different spec.

Also, setCapture() and releaseCapture() were removed from DOM3 Events, on request of Mozilla and Microsoft, to be revisited in the pen-tablet/multi-touch/gesture event spec.
Comment 3 Charles Pritchard 2011-02-07 21:18:44 UTC
Mouse capture is handy for programming "zoom" interfaces with CSS 2D Transforms.
Comment 4 Sirisian 2011-02-08 06:15:36 UTC
If it wasn't for that email from the last message I would have completely forgot about this.

Anyone shown any interest in getting this added?

I typed it up rather fast back in 2010, but to reiterate the idea would be to allow the browser to capture, as in lock the mouse to a certain DOM region such as a canvas tag. Thinking back on this I realized a few things were missing from the initial idea so I'll explain the whole concept in more detail.

The MouseEvent would need two new members:
deltaX
deltaY
Which in the mousemove event would be set to the change in the mouse position. So if you move your mouse left 5 pixels deltaX would be set to -5 and if you moved the mouse up 5 pixels deltaY would be set to -5.

To keep this as simple as possible a developer would create a <canvas> tag on their web page then add a mousecapture, mousemove, mouseup, and mousedown event to it. The new mousecapture event would be fired when a user clicked on the element. Unless anyone can think of a reason I don't see why any other tag should be able to use that event. mousecapture would run before click:
http://www.w3.org/TR/2010/WD-DOM-Level-3-Events-20100907/#events-mouseevent-event-order
In the mousecapture event a user-agent can do whatever it wants to help the user. It could make a small yellow bar like in chrome or firefox and say "This website is requesting to capture your mouse. [Allow] [Deny]". If denied the event would never be called. If allowed the event wouldn't be called until the user clicked a second time. The user-agent could handle this how they wanted.

A user-agent would also define a key to release mouse capture. (Probably Esc, but there's no need to define it in the standard except as a suggestion). Implementers would be encouraged to display something over the page that says "Press <key> to release mouse capture".

With the mousecapture is a mouserelease event which when registered on the same canvas tag responds when the user presses the release key. The event actually does the unregistering. The mouserelease would need to be callable to allow a web page to release control when it wanted, but the mousecapture can't be callable since that would allow someone to capture a mouse without a click.

A situation that needs to be handled is alt-tabbing or general loss of focus from a tab or the browser. During these situations the simplest thing to do is to call the mouserelease event. The user would need to click to regain mouse capture.

That sums up everything. Obviously it could be implemented differently, but I think I outlined what's needed.

--

Oddly enough since I made this post only one other person has asked how to do this. It was for a Canvas real-time strategy (RTS) game with WebSockets. (It was so users could scroll to the edges of the map and move around like in a normal RTS games).

I know this is for the few people that need it and not the majority, but I think it's one step closer to making web apps indistinguishable from regular applications.
Comment 5 Kenneth Russell 2011-02-08 18:10:25 UTC
(In reply to comment #4)
> If it wasn't for that email from the last message I would have completely
> forgot about this.
> 
> Anyone shown any interest in getting this added?

Personally I would love to see this functionality added. It would let apps like the Quake II port to GWT and WebGL (http://code.google.com/p/quake2-gwt-port/) behave just like desktop games, at least when using the mouse and keyboard for control.

The best path forward might be to pick a browser, build a vendor-specific prototype ("mozmousecapture", "webkitmousecapture") and show off how it works in the context of a particular app. The Quake II port might be a good use case.
Comment 6 Charles Pritchard 2011-02-08 19:04:45 UTC
My sole issue with mouse capture is getting out of the Capture state: I don't want to require a keyboard to break out. Full screen modes solve this by creating a hot zone on the top edge of the screen. That's not quite what we'd want here. Any ideas?
Comment 7 Sirisian 2011-02-09 00:29:23 UTC
(In reply to comment #6)
> My sole issue with mouse capture is getting out of the Capture state: I don't
> want to require a keyboard to break out. Full screen modes solve this by
> creating a hot zone on the top edge of the screen. That's not quite what we'd
> want here. Any ideas?

That would ideally be defined by the user-agent. Also why don't you want it to be keyboard dependent? I thought user-agents assumed a keyboard. The escape key is a logical choice since no web application would use it and every keyboard has it making it a fine suggested release option. I believe Java uses escape as the key to release the mouse and everyone is already familiar with exiting fullscreen from flash by pressing the escape key. Anyway this decision is already set in stone. It can't be mouse dependent since the mouse could be made invisible leaving the only choice to use the keyboard.
Comment 8 Charles Pritchard 2011-02-09 02:54:27 UTC
(In reply to comment #7)

http://www.w3.org/TR/WCAG20/
2.1 Make all functionality available from a keyboard.

http://forums.overclockers.co.uk/showthread.php?t=18236540
http://www.joystiq.com/2011/02/07/visceral-games-hears-disabled-gamer-adding-customizable-control/

Yes the mouse can be made invisible: and that could be part of the game.
In full screen the mouse could be invisible, but you still know that if you move up to the top of the screen, you're going to get back to the UA.

Context menu is still a viable option for getting out of a capture state.
Comment 9 Sirisian 2011-02-09 03:01:30 UTC
(In reply to comment #8)
> (In reply to comment #7)
> 
> http://www.w3.org/TR/WCAG20/
> 2.1 Make all functionality available from a keyboard.
> 
> http://forums.overclockers.co.uk/showthread.php?t=18236540
> http://www.joystiq.com/2011/02/07/visceral-games-hears-disabled-gamer-adding-customizable-control/
> 
> Yes the mouse can be made invisible: and that could be part of the game.
> In full screen the mouse could be invisible, but you still know that if you
> move up to the top of the screen, you're going to get back to the UA.
> 
> Context menu is still a viable option for getting out of a capture state.

You can block the context menu in order to use the right click mouse button yourself. Relying on the context menu is a big no no. In a world where web apps are indistinguishable from a real application the right click operation belongs to the application and not the user-agent when it's in the web page. (Opera I believe is the only browser that has shown disagreement over this thinking).
Comment 10 Charles Pritchard 2011-02-09 03:13:24 UTC
(In reply to comment #9)
> You can block the context menu in order to use the right click mouse button

Good call. How about not allowing that block to happen, when capture is active? Only real use case I've had for capture has to do with screen magnification; I don't need contextmenu blocking for that.

Only time I've really needed contextmenu blocking was for kiosk operations, where the user's don't have a keyboard and I don't want them reaching the UA (and I use special UA flags on boot).
Comment 11 Sirisian 2011-02-09 04:54:33 UTC
(In reply to comment #10)
> (In reply to comment #9)
> > You can block the context menu in order to use the right click mouse button
> 
> Good call. How about not allowing that block to happen, when capture is active?
> Only real use case I've had for capture has to do with screen magnification; I
> don't need contextmenu blocking for that.
> 
> Only time I've really needed contextmenu blocking was for kiosk operations,
> where the user's don't have a keyboard and I don't want them reaching the UA
> (and I use special UA flags on boot).

You're forgetting the use in games which I believe is key to allowing WebGL and Canvas games to succeed over say their Flash competitor. Also you can never be sure how something will be used. I've already gotten changes into the spec to allow better button support:
http://www.w3.org/TR/DOM-Level-3-Events/#events-MouseEvent-buttons
Which supports 5+ buttons on a mouse so a game could bind to those if it wanted to (IE9 supports left,right and middle, but doesn't take the suggestion to support 4 and 5). Ideally the right button is no different than any other mouse button.

Not allowing that context menu block to happen would be confusing for developers that want to use the mouse in say a first person shooter (FPS) and utilize the right mouse button. Real-time strategy games are the same way. Trying to move the mouse out of the region of the screen scrolls the world around. However a developer would also like to use right click to issue move commands. So the idea is to give users the same access to the mouse as if they were making an application. Restricting this would just be a hindrance.
Comment 12 Sirisian 2011-02-11 05:13:30 UTC
Okay on the public-webapps@w3.org mailing list a lot of changes in ideas have happened for the better. So in the following sections I'll explain the current ideas and where things are at in the discussion.

This new feature will be known from now on as Mouse Lock and not capture.
The two current required events are:
mouselock(event)
mouseunlock(event)

Comments regarding these events:
Would event.successful be good to see if the mouselock was successful? Obviously if the browser doesn't allow mouseLock then it will need to report a failure.
The event in the function could be called MouseLockEvent.
Fits with MouseEvent, KeyboardEvent, WheelEvent

The WheelEvent's deltaX and deltaY should be moved into MouseEvent 
since both are the same data type. The wheel and mousemove event would just use 
them differently. Might be a poor design decision that will need to be checked 
out. On mousemove the deltaX and deltaY as mentioned before will show the change in the mouse position from the previous position. (This will probably mean the browser might need to reset the mouse to the center of the screen to implement things, but that's an implementer problem as mentioned in the mailing list).

Does the mouseunlock require an event? Would there be an option to cancel the event? (In other words should programs be able to reject a release key operation? This isn't a joke question. Think hard about it. There might be reasons to allow that). (Returning false would be an option to look into, much like how you return false to cancel the click event).

These events can be used on any element, not just canvas. Can you think of elements that wouldn't work?

Two methods would be added to elements:
element.lockMouse()
element.unlockMouse()

You might be wondering what these mean? Can someone just lock the mouse to an element whenever they want? That's implementation defined, but a warning would usually be suggested in the spec.

However, there is a reason for this change. The problem of defining how the mouse should be locked was a problem. Was it going to be on mousedown, click, mouseup, scroll wheel?

Allowing Javascript to make the call on an event is very powerful. On the first call per domain a warning could be displayed like "This website has requested to lock your mouse [Allow] [Deny] What is this?". I know a lot of people are thinking of users and how this could be exploited which is still up for discussion.

Here's a good point about fullscreen.
> Methods for relaxing that can be explored later.  For example (as mentioned
> just previously) a browser might allow locking the mouse without any
> interaction if the page is fullscreened, or if the page is acting as a
> standalone application (eg. an "application shortcut" in Chrome or a
> Chromium packaged app).
> -- Glenn Maynard

Expanding that, changing out of fullscreen would thus unlock the mouse probably and invoke the mouseunlock event callback. However if the user hasn't allowed the mouse to take over the mouse yet in non-fullscreen then a warning would pop up if the mouseunlock was cancelled or mouseLock was called.

A use case that's somewhat useful. Again from the mailing list:

> It applies to non-game uses, too.  For example, a common annoyance with
> Google Maps is when you're dragging the map and your mouse cursor hits the
> side of the screen, the map stops moving; you have to release the button and
> reposition the cursor.  Mouse grabbing would trivially fix this.
> -- Glenn Maynard

To illustrate this you have a mousedown event on the map region that calls element.lockMouse() then on mouseup calls element.unlockMouse() give or take.
Anyone have a use-case where you want to be able to set the mouse position when it's locked? Sounds safe and useful if you want to release the mouse in a certain spot. Is this exploitable?

So right now this idea is extremely flexible and places no real restrictions on web applications. Criticism or use-cases would be appreciated so that things aren't limited or short-sighted.

Mailing list entries can be found here which seems more active:
http://lists.w3.org/Archives/Public/public-webapps/2011JanMar/

I'll probably keep a log here to make sure everyone's ideas and concerns are reflected.
Comment 13 Sirisian 2011-02-11 19:25:00 UTC
> Would event.successful be good to see if the mouselock was successful?

After reading this over again I guess the mouselock event wouldn't need to be invoked if it failed. I think originally I wanted a way to detect that, but it's really not important.
Comment 14 Sirisian 2011-02-11 21:10:50 UTC
Okay I asked for implementors, since even though I have C++ experience, it seems like a lot of work.

Firefox:
https://bugzilla.mozilla.org/show_bug.cgi?id=633602
Chrome:
http://code.google.com/p/chromium/issues/detail?id=72754

Continue to discuss as normal. I'm still trying to find problems or situations that break what I said or ways to handle things better.
Comment 15 the.krio 2011-03-02 17:35:35 UTC
In desktop FPS games (MS Windows at least) 'mouse look' is usually implemented by:
1) disabling display of mouse cursor 
2) calculating the delta of cursor position relatively to the previous frame
3) changing the position of the mouse to the center of the window every frame
If the third point is omitted, the mouse pointer will run into the edge of the screen and there will be no more correct delta generated by moving the mouse in the same direction.
Originally the inability to implement 'mouse look' in browser games comes from the inability to change the position of the mouse cursor with javascript or flash.
So, it should be noted that this mouselocking should always allow for correct delta calculation.
Comment 16 scheib 2011-06-16 23:58:06 UTC
I'm working on an implementation of mouse lock in Chrome, and would appreciate collaboration on refinement of the spec proposal. Hopefully webapps is willing to pick up this spec work? I'm up for helping write the draft.

Some updates from Sirisian's Comment 12:
- We shouldn't need events for success/failure to obtain lock, those can be callbacks similar to geolocation API: (http://dev.w3.org/geo/api/spec-source.html#geolocation_interface).
 
My short summary is then:
- 2 new methods on an element to enter and exit mouse lock. Two callbacks on the entering call provide notification of success or failure.
- Mousemove event gains .deltaX .deltaY members, always valid, not just during mouse lock. 
- Elements have an event to detect when mouseLock is lost. 

Example
x = document.getElementById("x");
x.addEventListener("mousemove", mouseMove);
x.addEventListener("mouseLockLost", mouseLockLost);
x.lockMouse(
  function() { console.log("Locked."); }, 
  function() { console.log("Lock rejected."); } );
function mouseMove(e) { console.log(e.deltaX + ", " + e.deltaY); }
function mouseLockLost(e) { console.log("Lost lock."); }
Comment 17 John Villar 2011-06-17 14:23:22 UTC
Ok, i'm currently working on a FPS game based on WebGL. Who i need to contact in order to begin testing real use cases for this issue? Is there anyone with an already functional prototype? i was thinking of doing a plugin on PPAPI for chrome to test out the functionality (at least on windows) but have seen that theres' already some work in progress...
Comment 18 Sirisian 2011-06-17 18:25:28 UTC
(In reply to comment #16)
> I'm working on an implementation of mouse lock in Chrome, and would appreciate
> collaboration on refinement of the spec proposal. Hopefully webapps is willing
> to pick up this spec work? I'm up for helping write the draft.
> 
> Some updates from Sirisian's Comment 12:
> - We shouldn't need events for success/failure to obtain lock, those can be
> callbacks similar to geolocation API:
> (http://dev.w3.org/geo/api/spec-source.html#geolocation_interface).
> 
> My short summary is then:
> - 2 new methods on an element to enter and exit mouse lock. Two callbacks on
> the entering call provide notification of success or failure.
> - Mousemove event gains .deltaX .deltaY members, always valid, not just during
> mouse lock. 
> - Elements have an event to detect when mouseLock is lost. 
> 
> Example
> x = document.getElementById("x");
> x.addEventListener("mousemove", mouseMove);
> x.addEventListener("mouseLockLost", mouseLockLost);
> x.lockMouse(
>   function() { console.log("Locked."); }, 
>   function() { console.log("Lock rejected."); } );
> function mouseMove(e) { console.log(e.deltaX + ", " + e.deltaY); }
> function mouseLockLost(e) { console.log("Lost lock."); }

That works and sensibly puts the success and reject operations where the user is calling lockMouse.

I almost feel like looking at things now with your suggestion that things are either success or failure on mouseLock. Assuming lockMouse can be a blocking method we could simplify everything to just:

x = document.getElementById("x");
x.addEventListener("mousemove", mouseMove);
x.addEventListener("mouseunlock", mouseUnlock);
if (x.lockMouse())
{
	console.log("Locked.");
}
else
{
	console.log("Lock Failed.");
}

Also I still like the simple event name of mouseunlock since it follows the neat two word DOM3 event names. I like your idea of calculating deltaX and deltaY even if the mouse isn't locked. (I didn't intend for it to only be valid when the mouse was locked). It might require special handling though when the mouse is locked depending on the operating system. (Scrolling to the edge in windows I believe causes 0 delta to be returned thus the resetting of the mouse to the center of the screen for FPS games. Not sure though since I've never messed with that functionality much).
Comment 19 John Villar 2011-06-17 19:05:19 UTC
Sirisian i don't know about the blocking call... because if the UA is going to let the user know the page is requesting locking with a blocking dialog (say a popup like the alert() one) the user can default sometimes to click on a "Cancel" button as a default reaction... however, users tend to like more the idea of the non-blocking banner alert on the top of the window and give that kind of alert a lot more of tought (because it is an asycnhronous prompt).

What if the method is simplified like the XMLHttpRequest response? e.g.:

x = document.getElementById("x");
x.addEventListener("mousemove", mouseMove);
x.addEventListener("mouseLockLost", mouseLockLost);
x.lockMouse(
  function(event) {
    if (event.locked) {
      console.log("Locked.");
    } else {
      console.log("Not Locked.");
    }
  } );
function mouseMove(e) { console.log(e.deltaX + ", " + e.deltaY); }
function mouseLockLost(e) { console.log("Lost lock."); }
Comment 20 Sirisian 2011-06-18 04:07:37 UTC
Yeah you're right. It needs to be asynchronous. Your idea of one function leads me back to my proposal of two events mouselock and mouseunlock. It seems like an aesthetically appealing API (naming wise). When I was writing this I was thinking that the user would need to check the MouseLockEvent in mouselock to see if there was an error anyway. error would have standard defined reasons like "user denied" so a web page could actually react to those kinds of actions with a dialog box instead of trying to guess why the lock failed. The same is true for mouseunlock. It would return a standard defined reason why the mouse released the lock. "user action" or "blur" or something. Unless it's deemed not important to know why.

x = document.getElementById("x");
x.addEventListener("mousemove", mouseMove);
x.addEventListener("mouselock", mouseLock);
x.addEventListener("mouseunlock", mouseUnlock);

x.lockMouse();
function mouseMove(e) { console.log(e.deltaX + ", " + e.deltaY); }
// e in both of these is MouseLockEvent
function mouseLock(e) { if (!e.error) { console.log("lock."); } }
function mouseUnlock(e) { console.log("unlock: " + e.reason); }

Then again scheib's way is tempting. (I don't have a preference. It's merely aesthetics for me :P )
Comment 21 John Villar 2011-06-18 14:53:17 UTC
(In reply to comment #20)
> thinking that the user would need to check the MouseLockEvent in mouselock to
> see if there was an error anyway. error would have standard defined reasons
> like "user denied" so a web page could actually react to those kinds of actions
> with a dialog box instead of trying to guess why the lock failed. The same is
> true for mouseunlock. It would return a standard defined reason why the mouse
> released the lock. "user action" or "blur" or something. Unless it's deemed not
> important to know why.

This seems plausible, however, reasons would need to be numeric codes, because receiving text responses would impose certain i18n issues.

As far as i've seen, the proposal should include at least the following change to MouseEvent interface (http://www.w3.org/TR/2000/REC-DOM-Level-2-Events-20001113/events.html#Events-eventgroupings-mouseevents):

// Introduced in DOM Level 2:
interface MouseEvent : UIEvent {
  readonly attribute long             screenX;
  readonly attribute long             screenY;
  readonly attribute long             clientX;
  readonly attribute long             clientY;
  readonly attribute long             deltaX; // Added from W3C Bug 9557 MouseLock
  readonly attribute long             deltaY; // Added from W3C Bug 9557 MouseLock
  readonly attribute boolean          ctrlKey;
  readonly attribute boolean          shiftKey;
  readonly attribute boolean          altKey;
  readonly attribute boolean          metaKey;
  readonly attribute unsigned short   button;
  readonly attribute EventTarget      relatedTarget;
  void               initMouseEvent(in DOMString typeArg, 
                                    in boolean canBubbleArg, 
                                    in boolean cancelableArg, 
                                    in views::AbstractView viewArg, 
                                    in long detailArg, 
                                    in long screenXArg, 
                                    in long screenYArg, 
                                    in long clientXArg, 
                                    in long clientYArg, 
                                    in boolean ctrlKeyArg, 
                                    in boolean altKeyArg, 
                                    in boolean shiftKeyArg, 
                                    in boolean metaKeyArg, 
                                    in unsigned short buttonArg, 
                                    in EventTarget relatedTargetArg);
};


And modification to the HTMLElement interface too (http://www.w3.org/TR/2003/REC-DOM-Level-2-HTML-20030109/html.html#ID-011100101):

interface HTMLElement : Element {
           attribute DOMString       id;
           attribute DOMString       title;
           attribute DOMString       lang;
           attribute DOMString       dir;
           attribute DOMString       className;
  void               lockMouse(); // Added from W3C Bug 9557 MouseLock
};

I doubt the W3C review board will like the idea of mouse locking over HTMLElement, we should pick a few elements and stick to them (could be DIV or CANVAS, or both, this should be worked out, no much sense to have mouse locking over an AUDIO element).

Another point is that lockMouse should raise an exception if there isn't a relative movement pointing device (could be a mouse, trackball or trackpad) or do the UA just send a failed lock event?

Should we begin working on a proposal document? ideas ideas ideas...
Comment 22 John Villar 2011-06-18 15:19:53 UTC
Other ideas i've come up with:

* If mouse is locked, should the cursor dissapear? i guess its the best solution because having the mouse lock means the webapp is handling the mouse representation itnernally.

* The UA notification shouldn't be a recommendation, it should be a MUST for usability purposes. Every user should expect that their UA notifies them of the upcoming mouse locking and let them take action.

* Mouse locking shouldn't be a domain permission, but a subdomain permission. E.g. Google App Engine has domain appspot.com and granting access to it means i will give permission to each and every app hosted as a subdomain. Malintentioned apps could take advantage of this.

* Should we include other devices different than mouse? or those devices are going to have their own spec? (gyros, gamepads, etc).

Sorry for spamming you all.
Comment 23 Sirisian 2011-06-18 18:23:10 UTC
Is there a reason you left out unlockMouse in HTMLElement? Did you see a different way to unlock the mouse from an element within code? If not I'd imagine it would need to be in there. Not to mention it's intuitive that if you can call lockMouse on an element that a user would be able to then call unlockMouse.

Regarding making the cursor disappear I think the standards people are 100% against that. (There's a reasons cursor:none isn't standard). You can always change the cursor to an image which is what a game might do. (You can even make it a 1px by 1px image I believe.
Comment 24 John Villar 2011-06-18 19:06:40 UTC
(In reply to comment #23)
> Is there a reason you left out unlockMouse in HTMLElement? Did you see a
> different way to unlock the mouse from an element within code? If not I'd
> imagine it would need to be in there. Not to mention it's intuitive that if you
> can call lockMouse on an element that a user would be able to then call
> unlockMouse.

My bad, should have written this:

interface HTMLElement : Element {
           attribute DOMString       id;
           attribute DOMString       title;
           attribute DOMString       lang;
           attribute DOMString       dir;
           attribute DOMString       className;
  void               lockMouse(); // Added from W3C Bug 9557
  void               unlockMouse(); // Added from W3C Bug 9557
};

Although i think unlocking the mouse doesn't relates to the HTMLElement, maybe it is a Window method?. Btw, we should specify what happens when the lockMouse is called and the mouse was already locked (for the same Element or for another Element), i don't really like the idea of nesting locks, i would prefer failing, but maybe i haven't realized use cases where multiple locking would work.

> Regarding making the cursor disappear I think the standards people are 100%
> against that. (There's a reasons cursor:none isn't standard). You can always
> change the cursor to an image which is what a game might do. (You can even make
> it a 1px by 1px image I believe.

I haven't come up with a way to change the cursor (at least on Chrome and Firefox) to a custom cursor, let alone dissapear it. However, the use cases of MouseLock doesn't need representation of the cursor (because they are meant for relative mouse movement). I know the standards body is completely against the custom mouse cursors, but that's just because our use cases haven't been considered and there wasn't a way to specify complex permissions for different web pages.

I was considering that MouseEvent should have an attribute called "cancelLock" so you can unlock the mouse based on the current MouseEvent (like bubbles works). Don't know if this is really necessary.

Btw, MouseLock should call the lockLost event when reloading or leaving the page? don't know how this should be handled.
Comment 25 Olli Pettay 2011-06-19 12:27:30 UTC
(In reply to comment #24)
> interface HTMLElement : Element {
>            attribute DOMString       id;
>            attribute DOMString       title;
>            attribute DOMString       lang;
>            attribute DOMString       dir;
>            attribute DOMString       className;
>   void               lockMouse(); // Added from W3C Bug 9557
>   void               unlockMouse(); // Added from W3C Bug 9557
> };

So how are lockMouse/unlockMouse different from setCapture/releaseCapture?
Should we just standardize setCapture/releaseCapture, and maybe
extend them a bit if they lack some feature.


> I haven't come up with a way to change the cursor (at least on Chrome and
> Firefox) to a custom cursor
Use CSS?
https://developer.mozilla.org/en/Using_URL_values_for_the_cursor_property
Comment 26 John Villar 2011-06-19 17:15:13 UTC
(In reply to comment #25)
> So how are lockMouse/unlockMouse different from setCapture/releaseCapture?
> Should we just standardize setCapture/releaseCapture, and maybe
> extend them a bit if they lack some feature.

setCapture and releaseCapture jsut indicate where are going the mouse events to be sent. They don't restrict cursor bound nor enable relative mouse movement. We are proposing a standard where the mouse movement is represented as deltas and not on absolute coordinates. e.g. if the user moves the mouse rightward, the event will have a positive deltaX, if it is moved leftward it will have negative deltaX and it would be the same with the vertical axis (probably negative upward and positive downward).

> https://developer.mozilla.org/en/Using_URL_values_for_the_cursor_property
Worked neatly on chrome, ff and ie, seems to fail on Opera (couldn't test on safari). This could solve the visible cursor problem and the CSS user-select can disable changing the cursor when click dragging.
Comment 27 Sirisian 2011-06-19 20:19:02 UTC
Branch1Info

Branch2Info [10*10][10*10]

LeafData


(In reply to comment #24)
> Although i think unlocking the mouse doesn't relates to the HTMLElement, maybe
> it is a Window method?. Btw, we should specify what happens when the lockMouse
> is called and the mouse was already locked (for the same Element or for another
> Element), i don't really like the idea of nesting locks, i would prefer
> failing, but maybe i haven't realized use cases where multiple locking would
> work.

Refer to this:
http://code.google.com/p/chromium/issues/detail?id=72754
> lockMouse would lock the mouse to the DOM region. If called on another element it would unlock on the current element and then lock on the new one.

Pretty simple really. You can only have 1 lock at a time. Regarding calling lock on a control twice I think for simplicity it would unlock then lock. (That's more of a programming error).

(In reply to comment #24)
> Btw, MouseLock should call the lockLost event when reloading or leaving the
> page? don't know how this should be handled.

Yeah that's a good idea. If a user unloads the page or reloads the page it should unlock before onbeforeunload is called. Good catch. This needs to be defined in the standard.
Comment 28 John Villar 2011-06-20 16:04:07 UTC
Ok, i will begin writing a spec for this bug based on the input here, so we can begin stiting everything on an organized doc, is there anyone interested in giving a hand?

Olli, in response to comment #25, setCapture works great for some use cases, but i think it is lacking a lot, and our focus is on fixed mouse capturing for relative mouse movement, independent of a mouse button being held down or not. So, i don't know if setCapture should be extended to fit this issue or a complementary, separate spec that covers these specific use cases.
Comment 29 Olli Pettay 2011-06-20 16:09:52 UTC
If .deltaX/Y values are added, what is the origin for them?
I mean when are they 0?
Also, what happens to screenX/Y and clientX/Y, what kinds of values
do they get when deltaX/Y are used?

This all really needs a written specification draft.
Comment 30 John Villar 2011-06-20 16:19:35 UTC
(In reply to comment #29)
> If .deltaX/Y values are added, what is the origin for them?
> I mean when are they 0?
> Also, what happens to screenX/Y and clientX/Y, what kinds of values
> do they get when deltaX/Y are used?

.deltaX/Y should be 0 after lockMouse is successful, after that, with each MouseEvent .deltaX/Y should be reset to 0 till next MouseEvent. The .screenX/Y and .clientX/Y attributes are undefined.

Another question is... are we going to specify a scale for the deltas? i mean, are they going to be represented as pixels? or raw mouse values?... this can vary wildly.

> This all really needs a written specification draft.

Exactly... that's why i'm going to direct my efforts to writing everything down.
Comment 31 Olli Pettay 2011-06-20 17:31:22 UTC
(In reply to comment #30)
> 
> .deltaX/Y should be 0 after lockMouse is successful, after that, with each
> MouseEvent .deltaX/Y should be reset to 0 till next MouseEvent.
Well that would be strange. You don't know the place of the cursor when
lockMouse's success callback is called. Mouse cursor could be even in a
different screen.

> The .screenX/Y
> and .clientX/Y attributes are undefined.
We can't leave clientX/Y and screenX/Y undefined. MouseEvent do have those attributes and they need to have some value.
Comment 32 scheib 2011-06-20 18:14:04 UTC
API location: I believe Element is appropriate. E.g. Similar to the Fullscreen API that appears on Element. The api should also exist on Window. The element that is locked indicates the event target of the locked mousemove events. Many specifications offer this additive API. E.g. see http://trac.webkit.org/browser/trunk/Source/WebCore/dom/Element.idl#L27

Multiple Locks: I would initially suggest failing any subsequent requests from new elements if a lock is already held. Consider a user agent that only allows a lock on an element if there is a user gesture on that element. If a lock is already held, all gestures are being targeted at that locked element so no other would be satisfied. For the 'programming error' case of relocking the same element - I believe it could just call the success callback, no nesting, a single unlock would succeed.

Other Devices: I believe we should keep this focused at the moment on just mouse input. The next area I'd consider is tablets, which I believe could still benefit from a mouselock to fix the target of delta values. Other devices (e.g. dual analog stick game controllers) are specialized enough I believe they warrant a non-mouse input spec.

Scale: I don't see any use in a different scale than what screenX is based in.
Comment 33 scheib 2011-06-20 19:00:21 UTC
(In reply to comment #31)
> (In reply to comment #30)
> > 
> > .deltaX/Y should be 0 after lockMouse is successful, after that, with each
> > MouseEvent .deltaX/Y should be reset to 0 till next MouseEvent.
> Well that would be strange. You don't know the place of the cursor when
> lockMouse's success callback is called. Mouse cursor could be even in a
> different screen.
> 
> > The .screenX/Y
> > and .clientX/Y attributes are undefined.
> We can't leave clientX/Y and screenX/Y undefined. MouseEvent do have those
> attributes and they need to have some value.

deltas should be reported the same as if mouse events were delivered to a target and it subtracted the current clientX/Y from the previous. They should be 0 only if the mouse is not moving. A user agent should provide large values that would have been clamped if the mouse hits the side of the screen.

Screen and client positions should continue to be reported as if the mouse were stationary at the center of the target element. (note, different operating systems may require the mouse to be located elsewhere and hidden for the implementation to work, as discussed previously on email thread)
Comment 34 John Villar 2011-06-20 19:36:12 UTC
(In reply to comment #31)
> Well that would be strange. You don't know the place of the cursor when
> lockMouse's success callback is called. Mouse cursor could be even in a
> different screen.

I in fact use 2 screens, but for the mentioned use cases until now, getting the deltaX/Y from the mouse when the lock is granted (say, when the user clicks on a "Allow" button on the UA) doesn't make any sense since it isn't a locked movement.

> We can't leave clientX/Y and screenX/Y undefined. MouseEvent do have those
> attributes and they need to have some value.

I'm with Vincent here, we could make the spec report screenX/Y and clientX/Y equal to the center of the locked element.

(In reply to comment #32)
> Scale: I don't see any use in a different scale than what screenX is based in.
Certain Mouses (e.g. Razer mouses) have a fairly high resolution and for the game use case, precision is an important factor. Certain APIs enable you to obtain subpixel precision of mouse movements. Maybe the spec can tell that the scale must be the same that is used on screenX/Y but also recommend subpixel deltas to be reported whenever available?
Comment 35 Sirisian 2011-06-20 20:05:04 UTC
(In reply to comment #32)
> Multiple Locks: I would initially suggest failing any subsequent requests from
> new elements if a lock is already held. Consider a user agent that only allows
> a lock on an element if there is a user gesture on that element. If a lock is
> already held, all gestures are being targeted at that locked element so no
> other would be satisfied. For the 'programming error' case of relocking the
> same element - I believe it could just call the success callback, no nesting, a
> single unlock would succeed.

Yeah but that's not how a user agent will use this specification. A user agent can give a warning, but under no circumstances should it require mouse or keyboard focus to initiate a mouseLock. Once a program is given permission it can call mouseLock on any dom element. Thus calling mouseLock on a canvas element then calling mouseLock on another one will simply (in my original design) invoke the mouseunlock event on the first canvas before invoking mouselock on the second canvas.

I'm not a fan of a browser implementing in a special system for detecting a previous mouse click on a control before allowing mouseLock to be called on the dom element. (Or worse only allowing a mouseLock in a mousedown/mouseup event).
Comment 36 Sirisian 2011-06-21 04:32:19 UTC
Okay there's more discussion about security.

villarzavatti in the chrome issue tracker said:
> Just a suggestion: This spec should have another recommendation, that in case 
> a user persistently is tapping the "unlock mouse" key the UA should notify the 
> user with a "Do you want to prevent this page from locking the mouse" like when
> a page spits too much alert() messages in chrome or firefox.

This I believe was brought up before and is an excellent idea. It's important to remember that even if a user hits the recommend key (refer to my recommendation in the chrome and firefox issue trackers linked above) the program can still immediately lock the mouse again. It can be recommended that if this happens twice where a user hits escape then a lock and the user hits escape again that mouse lock is rejected and the recommendation to enable mouse lock (yellow bar in Firefox and Chrome) is reset.


Also I thought this was obvious but I guess not from reading the mailing list. If the user alt-tabs or the window changes focus then mouseUnlock is called. mouseLock only works if the current window has focus. I'll make this explicit since the specification will need to state this. If the UA's window (meaning the tab and window) is not currently focused then the mouseLock event will fail. (If my recommendation to have an e.error is allowed then it would have a number that relates to "blur or not in focus").

Olli on the mailing list:
> Yeah. Mouse could be locked on mousedown, but it should be automatically 
> released on mouseup.
> That is the way set/releaseCapture works in Firefox.

> Other cases should need explicit permission from user.

Yeah this was something I was thinking of. I'm not saying I like the idea, but if it's seamlessly implementable then it might as well be included. How did you think this should work for mouseLock?

I'll say one and see if anyone agrees. Without permission a user is allowed to call mouseLock on mouseDown however it will automatically have a binded unlockMouse for mouseUp (or any blur causing event).

As mentioned mouse locking is a special operation that on most OSs requires the mouse to be snapped to the center of the screen every frame to correctly calculate the mouse delta.

Tab Atkins said:
> Yes, which is why I suggest a persistent message be shown with
> instructions on how to release the lock.  Thus the user is aware that
> the website is being hostile and knows how to stop it long enough to
> get away (clicking Back or closing the tab).

The yellow message warning would explain the escape key. Another recommendation could fade the first time in the center of the screen the same way flash tells the user how to exit full screen. A persistant message in fullscreen would probably be misimplemented by browsers (making it so that the resolution is not the full screen) or it would simply be distracting.

I imagine the chrome implementation will quickly find if there are any implementation restrictions though. I think I've summed up a lot of mailing list talk for now.
Comment 37 scheib 2011-06-22 17:57:51 UTC
John Villar and I have produced a draft specification:
https://docs.google.com/document/d/1uV4uDVIe9-8XdVndW8nNGWBfqn9ieeop-5TRfScOG_o/edit?hl=en_US&authkey=CM-dw7QG

Some additional notes included in that draft not yet discussed here:
- .deltaX conflicts with WheelEvent, I've used .movementX/Y instead unless we feel DOM L3 should be modified.
- I've omitted the call of mouselocklost at a page close or onbeforeunload, as I can not determine a use case that benefits from it. 
- I've omitted the suggestion of cancelLock on a bubbling MouseEvent - as it is not needed, .target.unlockMouse() can be called.

Also, note that the mailing list discussion subject "Mouse Lock" can be found here: http://lists.w3.org/Archives/Public/public-webapps/2011AprJun/thread.html#msg1095
Comment 38 Sirisian 2011-06-23 17:52:04 UTC
(In reply to comment #37)
> - I've omitted the call of mouselocklost at a page close or onbeforeunload,
> as I can not determine a use case that benefits from it.

Well I was thinking it just made sense to notify the page the mouse was unlocked before onbeforeunload since when the dialog box pops up the mouse is no longer locked. The user can't call mouseLock when that box is displayed either. It's not really about a use case. It's just expected behavior.
Comment 39 Leonard Ritter 2011-08-26 12:44:38 UTC
Any progress on this? I need this feature.
Comment 40 John Villar 2011-08-26 13:17:24 UTC
#39 Check comment #37 Vincent Scheib is hard at work on that issue on the Chromium project, and Olli Pettay from Mozilla has said he's waiting for the specification to mature to begin implementing this feature.
Comment 41 Olli Pettay 2011-08-26 13:55:10 UTC
I don't know where the specification is? ;)
I've only seen some drafts from a vendor.

But anyone could start implementing this in
Mozilla. It supports already capturing content, so this would extend that
in some sense. Would just need to prefix all the APIs with moz.
Comment 42 John Villar 2011-08-26 14:15:47 UTC
There i got your attention ;)

We need a Mozilla Hero! Are those new Mozilla Pets available for hire!? lol

Now, speaking seriously, the draft is looking very good, and remember it has received community input too. I haven't talked with Vincent lately, but i know he's hard at work with this spec and there will be eventually a running sample of this, it would be beautiful to see mozilla in the same process so the community as a whole gets benefited.

I think this is one key feture to make the browser the next killer app that addresses all our computing needs.
Comment 43 Olli Pettay 2011-08-26 14:28:01 UTC
Some questions about the draft.

Why "When mouse lock is enabled clientX, clientY, screenX, and screenY must hold constant ", yet movementX/Y is counted based on changes to clientX/Y.

Interface MouseLockLostEvent is useless, just use Event.

The draft doesn't say if lock should be removed when the target of the
lock is removed from document or moved to another document.

"Escape can not be performed by mouse actions" seems very restricted.
User must have the control, not web app.
Comment 44 John Villar 2011-08-26 14:58:11 UTC
(In reply to comment #43)
> Why "When mouse lock is enabled clientX, clientY, screenX, and screenY must
> hold constant ", yet movementX/Y is counted based on changes to clientX/Y.

Some implementations will need to modify the mouse position to calculate the movementX/Y and thus will make the clientX/Y - screenX/Y irrelevant (because those values won't express the user desired mouse position). This is because some OSes don't support direct mouse acces and don't have a way to "read" the relative mouse movements.

> Interface MouseLockLostEvent is useless, just use Event.

'Nuff said ;-)

> The draft doesn't say if lock should be removed when the target of the
> lock is removed from document or moved to another document.

Good point, this should be considered in the draft.

> "Escape can not be performed by mouse actions" seems very restricted.
> User must have the control, not web app.

This draft specifies a new way of mouse interaction where the mouse doesn't represent a pointer anymore, and thus the mouse actions don't represent a direct action done by the user, but instead relative, incremental actions based on the state of the app (For more on this please refer to the use cases at the end of the draft). That's why you can't specify a gesture or a click area where the user can unlock the mouselock, there's no predictability on what combination of actions/gestures a webapp is going to support that can work along a predifined festure that acts as an unlock mechanism. Keyboard is the only way to unlock here because of that.

If you want first hand experience with this kind of interaction there are plenty of open source apps that use these scenarios:

* Cube / Sauerbraten http://sauerbraten.org/ (First person shooter game)
 - Pitch/Yaw crontol of the viewport, firing, switching weapons, using items, etc.
* Battle for wesnoth http://www.wesnoth.org/ (Strategy game)
 - Viewport scrolling
* Freeciv http://freeciv.wikia.com/wiki/Main_Page (Strategy game)
 - Viewport scrolling
* Blender http://blender.org (3D graphics environment)
 - Viewport/Object moving and rotation; Object scaling; spinner increment and decrement.

I recommend that you use them on "windowed" mode so you can get a feel of what really the mouse locking looks like, you'll see that the locking limits the interaction to the "window" area and no other app gets affected by the mouse movement regardless of what gestures/actions you do with it.

This is what's needed on our scenario.
Comment 45 Olli Pettay 2011-08-26 15:13:26 UTC
Other random comments...

It is not clear how movementX/Y should be counted for example
for mouseover/out events or click events.
Or is the assumption that movementX/Y are 0 with those events?
Same with certain touch.

"Mouse lock must succeed only if the window is in focus"
The window probably doesn't have focus atm when user clicks the
permission dialog. Or does the sentence mean that
lockMouse's failure callback should be called immediately/real-soon
after lockMouse call if the window in which element is isn't focused?
Or what?

What should happen if one does
document.createElement("div").lockMouse(function(){}, function(){})?
Comment 46 John Villar 2011-08-26 15:28:21 UTC
(In reply to comment #45)
> It is not clear how movementX/Y should be counted for example
> for mouseover/out events or click events.
> Or is the assumption that movementX/Y are 0 with those events?
> Same with certain touch.

I guess the spec should specify that movementX/Y isn't valid if the mouse isn't locked. This should be discussed on the draft so the changes needed get done there (same for the comments below).

> "Mouse lock must succeed only if the window is in focus"
> The window probably doesn't have focus atm when user clicks the
> permission dialog. Or does the sentence mean that
> lockMouse's failure callback should be called immediately/real-soon
> after lockMouse call if the window in which element is isn't focused?
> Or what?

Here the "window" should be something like "the current tab" and the browser being the main active window.
 
> What should happen if one does
> document.createElement("div").lockMouse(function(){}, function(){})?

This should fail immediatly. Good point.

Olli, do you have edit-access to the document?
Comment 47 scheib 2011-08-26 18:07:09 UTC
Draft has some open issues, I just posted a summary of them on the public-webapps email list: http://lists.w3.org/Archives/Public/public-webapps/2011JulSep/0960.html

Some of your particular points are addressed there. If possible, could you make replies there to keep the conversation in one place?

John Villar responded well to #45 and #46 and I'll update spec draft:
- Drop MouseLockLostEvent to just Event
- mouseover/out will never be called when in mouse lock - will clarify in spec. Other discussion thread notes that they will be zero values when not in mouse lock (pending feedback will update draft). I'll scrutinize for touch events also.
- Clarify wording of window being in focus, mention that permision prompt modal windows would be OK.

Bring up for discussion on public-webapps Mouse Lock thread
- What to do if target removed from DOM (or moved).
- What to do if target is not in DOM.
Comment 48 Leonard Ritter 2011-08-26 18:28:08 UTC
A few examples that make use of mouse-look have been posted, so I also wanted to post an example for how people work around missing mouse capturing today:

http://mrdoob.github.com/three.js/examples/webgl_geometry_minecraft_ao.html

This is a WebGL example with _hideous_ mouse controls ("drunken sailor" mode), because mouse capturing a la Quake is impossible.

I'm also making a first person game in WebGL, and I'm struggling with the same issue. The game I'm making uses both modes, captured and regular, and I intend to switch between them with a keyboard shortcut.
Comment 49 Klaas Heidstra 2011-08-28 19:51:16 UTC
(In reply to comment #47)
> Draft has some open issues, I just posted a summary of them on the
> public-webapps email list:
> http://lists.w3.org/Archives/Public/public-webapps/2011JulSep/0960.html
> 

See also: http://lists.w3.org/Archives/Public/public-webapps/2011JulSep/0804.html , http://lists.w3.org/Archives/Public/public-webapps/2011JulSep/0847.html and http://lists.w3.org/Archives/Public/public-webapps/2011JulSep/0977.html on that same mailing list, for a different approach (separate events for mouse deltas and and instead of locking the cursor constrain it to the bounds of a element).
Comment 50 Leonard Ritter 2011-12-19 18:54:58 UTC
Mouse Lock API and settings UI available in Chrome 17.0.963.12 dev after enabling pointer lock in about:flags, but the interface ignores my calls, it seems.

Are there some webkit-related examples available somewhere that I could test this on? Perhaps my code is buggy.
Comment 51 scheib 2011-12-20 02:02:55 UTC
Chrome JavaScript build will be coming soon. I've a prototype running with both NaCl and JavaScript bindings on my workstation, and will start landing in early January both Chrome and WebKit side patches. I'll need to build up API tests as I go on the WebKit side, Chrome will stay manual functional tests till I get it landed behind a flag.

I'll make it clear when it is available in a Canary build for people to easily start experimenting with. Thank you for your interest!

I should also point out that a patch for FireFox has been developed and may have test builds available: author's blog: http://vocamus.net/dave/?cat=28 and the bug: https://bugzilla.mozilla.org/show_bug.cgi?id=633602
Comment 52 Travis Leithead [MSFT] 2012-03-08 21:16:38 UTC
Assigning to myself to take action on these open bugs.
Comment 53 Travis Leithead [MSFT] 2012-03-12 20:07:53 UTC
Wow. Lots of fantastic discussion in this bug. I love the concept of mouselock. In the months since the last update to this bug, did the Chrome implementation get done? 

Given my goal for DOM3 Events (as noted here: http://lists.w3.org/Archives/Public/public-webapps/2012JanMar/1071.html), I think we need to find a more appropriate home for MouseLock. A new spec for WebApps, or perhaps grafted into the currently-named "DOM4 Events" spec? (see http://html5labs.interoperabilitybridges.com/dom4events/)

Looking forward to hearing from you on this. Meanwhile, I've changed the severity of this bug to "enhancement" as that matches my interpretation of this "bug".
Comment 54 Olli Pettay 2012-03-12 20:16:44 UTC
Chrome has an implementation, and Firefox's implementation is close to done.
Comment 55 scheib 2012-03-12 21:15:35 UTC
(In reply to comment #53)
> Wow. Lots of fantastic discussion in this bug. I love the concept of mouselock.
> In the months since the last update to this bug, did the Chrome implementation
> get done? 

Draft Spec http://dvcs.w3.org/hg/pointerlock/raw-file/default/index.html

It has been proposed for the WebApps recharter already.

Chrome isn't quite 'done', though it is close and can be played with in developer builds behind a flag: http://beautifulpixels.blogspot.com/2012/01/javascript-pointer-lock-mouse-lock-in.html for instructions.
Comment 56 Travis Leithead [MSFT] 2012-03-12 23:55:36 UTC
Cool. Looks like this is well on its way. 

Since this doesn't need to be a DOM3 Events issue anymore, I'm resolving this bug. Please let me know if that is satisfactory.