Draggable objects, 2022

 from Red Blob Games
DRAFT
23 Dec 2022

Many of my interactive pages have a draggable object. I want the reader to move the object around, and I want the diagram to respond in some way. Here I’ll document the code I use to make this work with both mouse and touch input, using browser features that are widely supported since 2020. Here’s what I want to support:

Drag the circle with mouse or touch
History →

From 2011 to 2014 I used d3-drag[1], but for my non-d3 projects, I ended up developing my own mouse+touch code, which I wrote about in 2018.

By 2012 MS IE had added support for pointer events[2] which unify and simplify mouse+touch handling. Chrome added support in 2017; Firefox in 2018; Safari in 2020[3].

Over the years browsers have changed the rules, including in 2017 when Chrome changed some events to default to passive mode[4] which causes the page to scroll while you’re trying to drag the object. This broke some pages[5]. Safari made this change in 2018[6]. Firefox maintained backwards compatibility.

I’ve run tests on Gecko/Firefox (Mac, Windows, Linux, Android), Blink/Chrome (Mac, Windows, Linux, Android), and WebKit/Safari (Mac, iPhone, iPad). I have not tested on hoverable stylus, hybrid touch+mouse devices, or voice input.

This is the simple model in my head:

state.svg

However it’s not so simple! Mouses have multiple buttons. Touch events can include multiple fingers. Events can go to multiple destinations.

{{ TODO :: just show the answer!! the rest of the page is explanation and demos of the things that go wrong }}

 1  Mouse events only#

The simplest thing to do is to capture mousedown, mouseup, and mousemove on the circle element. If the move occurs while dragging, then move the circle to the mouse position.

mouse-local.svg

Try the demo with a mouse:

Drag using mouse events on circle

This might seem like it works but it works badly.

To fix these problems we can use mousedown on the circle to add mousemove and mouseup on the document. Then on mouseup we remove the mousemove and mouseup from the document.

mouse-document.svg

Try it out. It works better.

Drag using mouse events on document

There are still a few issues, which we’ll solve one by one.

 2  Touch events#

Touch events automatically capture on touchstart and direct all touchmove events to the original element. This means we don’t have to temporarily put an event handler on document. We can go back to the logic in the first mouse example. If for any reason the browser needs to cancel the touch sequence, it sends touchcancel.

touch.svg

Try the demo with a touch device:

Drag using touch events

There are a few problems to deal with:

 3  Pointer events#

To handle both mouse and touch events we end up having lots of different event handlers:

mouse-and-touch.svg

Pointer events attempt to unify mouse and touch events. The pointer capture[7] feature lets us use the simpler logic that doesn’t require us to add/remove global event handlers to the document like we had to with mouse events.

pointer.svg

Much simpler! Try the demo with either a mouse or touch device:

Drag using pointer events

There are several things we still need to fix.

 3.1. Fix: text selection

When dragging the circle, the text inside gets selected sometimes.

Fix: CSS user-select: none on the circle.

If I want to select the text when not dragging the circle (left drag, right click, long press, or keyboard), I can apply that CSS only while dragging.

This problem won’t happen in Canvas and WebGL because that text is unselectable.

TODO: demo with three objects, one that always has the css, one that never has it, and one that only has it while dragging

Try thisWatch forCircle 1Circle 2Circle 3
drag circletext is selectedyes ⛌no ✓no ✓
drag texttext is selectedyes ✓no ⛌yes ✓

 3.2. Fix: scrolling with touch

On touch devices, single-finger drag will scroll the page. But single-finger drag also drags the circle. By default, it will do both.

Fix: CSS touch-action: none on the diagram.

But this prevents scrolling anywhere in the diagram. So a better fix is to apply it only to the circle. We can do this for SVG diagrams because each element has its own CSS separate from the containing diagram. But we can’t do the same for Canvas or WebGL diagams, because elements don’t have their own CSS. So another possible fix is to preventDefault() on touchstart. In Canvas and WebGL, check that the pointer is on the circle and conditionally prevent default.

TODO: demo with four separate diagrams side by side

  1. default
  2. touch-action none on diagram
  3. touch-action none on circle
  4. touchstart.preventdefault
Try thisWatch forCircle 1Circle 2Circle 3Circle 4
drag circlepage scrollsyes ⛌no ✓no ✓no ✓
drag diagrampage scrollsyes ✓no ⛌yes ✓yes ✓

 3.3. Fix: text drag

Try selecting all text on the page, then drag the circle. This will trigger text dragging on many browser+OS combination, where you can drag some text as an alternative to copy/paste. I want to allow this, except when dragging the circle.

Fix is: preventDefault() on dragstart on the circle. This works in SVG but not in Canvas or WebGL. For those, check that the pointer is on the circle and conditionally prevent default.

TODO: demo

Try thisWatch forCircle 1Circle 2
drag circlepage text dragsyes ⛌no ✓

 3.4. Fix: context menu

Context menus are different across platforms, and that makes handling it tricky.

Windows
right click (down+up), Shift + F10 key
Linux
right click, right button down, Shift + F10 key
Mac
right button down, Ctrl + left click
iPhone, iPad
long press on text only
Android
long press on anything

There are more ways to bring up context menus (examples: two finger tap on Mac or Windows; right side click on Windows; press-and-hold on Wacom pens) but I haven’t tested all of them.

Windows, right click, no capture:

Firefox, Chrome, Edge
pointerdown, pointerup, auxclick, contextmenu

Windows, right click, capture:

Firefox
pointerdown, get capture, pointerup, lose capture, auxclick, contextmenu
Chrome, Edge
pointerdown, get capture, pointerup, auxclick, lose capture, contextmenu

Linux right click, no capture:

Firefox
pointerdown, contextmenu, pointermove² while menu is up
Chrome
pointerdown, contextmenu, no pointermove² while menu is up

Linux hold right down, no capture:

Firefox
pointerdown, contextmenu, pointermove² while menu is up
Chrome
pointerdown, contextmenu, no pointermove² while menu is up

Linux right click, capture:

Firefox
pointerdown, contextmenu, get capture, pointermove² while menu is up tells us button released
Chrome
pointerdown, contextmenu, get capture; not until another click do we get pointerup, lose capture

Linux hold right down, capture:

Firefox
pointerdown, contextmenu, got capture, pointermove² while menu is up tells us button released; when releasing button, menu stays up but we get pointerup, lose capture
Chrome
pointerdown, contextmenu, got capture, no pointermove² while menu is up; when releasing button, menu stays up but we don’t get pointerup; not until another click do we get pointerup, click, lose capture

Mac, ctrl + left click:

Firefox
pointermove with buttons≠0, contextmenu (no pointerdown or pointerup)
Chrome
pointerdown with button=left, contextmenu (no pointerup)
Safari
pointerdown with button=left, contextmenu (no pointerup); but subsequent clicks only fire contextmenu

Mac, right button down:

Firefox
pointerdown with button=right, contextmenu (no pointerup)
Chrome
pointerdown with button=right, contextmenu (no pointerup)
Safari
pointerdown with button=right, contextmenu (no pointerup); but subsequent right clicks only fire contextmenu

If we capture events on pointerdown, Firefox and Safari will keep the capture even after the button is released. Chrome will keep capture until you move the mouse, and then it will release capture. [This seems like a Firefox/Safari bug to me, as pointer capture is supposed to be automatically released on mouse up]

It’s frustrating that on Mac, there’s no pointerup or pointercapture when releasing the mouse button. On Linux, the pointerup only shows up if you click to exit the context menu. It doesn’t show up if you press Esc to exit. The workaround is to watch pointermove events to see when no buttons are set. Windows doesn’t seem to have these issues, as both pointerdown and pointerup are delivered before the context menu.

Android, long press:

Firefox
pointerdown, get capture, contextmenu, pointerup, lose capture
Chrome
pointerdown, get capture, contextmenu, pointerup or pointercancel¹, lose capture

¹if the finger moves at all, this starts a scroll event which cancels the captured pointer ²Firefox lets the page see events outside the menu overlay, whereas Chrome doesn’t let the page see any events while the menu is up

--–— tests ---–—

The problem is that if I’m using pointerdown and then pointerup to track the state of the mouse, and I never get pointerup, my code thinks I’m still holding the button down (and the browser’s pointer capture is still active).

Try right clicking the circle. It will bring up a context menu. That itself is fine. Unfortunately the events aren’t consistent across platforms:

WindowsMacLinux
pointerdownpointerdownpointerdown
pointerup  
auxclick  
contextmenucontextmenucontextmenu
 pointerup 
 auxclick 

What should we do?

TODO: demo of both solutions?

Try thisWatch forCircle 1Circle 2Circle 3
right click on circlecircle movesyes ⛌no ?yes ?
right drag on circlecircle moves   

 3.5. Feature: handle drag offset

This is not implementation specific, but a design issue. If you pick up the edge of the circle then you want to keep holding it at that point, not from the center. The solution is to remember where the center is relative to where you started the drag. Then when you move the object, you add that offset back in.

TODO: demo both ways (although it duplicates little-details page) ; could draw something showing the pick up point

Try thisWatch forCircle 1Circle 2
drag from edge of circlecircle jumpsyes ⛌no ✓

 3.6. Feature: handle multitouch

isPrimary vs pointer id; need to test what happens if there are two independent drags going on

 3.7. Feature: handle simultaneous dragging

 3.8. TODO: what about multiple buttons?

So here’s a tricky one. If you are using multiple buttons at the same time, what happens? The Pointer Events spec says that the first button that was pressed leads to a pointerdown event, and the last one that was released leads to a pointerup event. But that means we might get a up event on a different button than the down event.

multiple-buttons.svg

This doesn’t seem to be what I want, but it’s what Pointer Events do[11]. The workaround is to check the button state in pointermove. But pointer capture continues until you release all the buttons, unless you explicitly release capture.

Mouse Events behave the way I want but don’t handle touch events.

 3.9. TODO: what about multiple mice?

Does PointerEvent.pointerId help here?

What happens when ipad is used to control mouse, or mouse is used to control ipad?

TODO: test middle clicking to drag, like some mice support on Windows and maybe Linux

TODO: test left click drag, then right button down, then left/right up. Since I only have one dragging state it might get confused.

https://www.w3.org/TR/pointerevents/#the-primary-pointer[12] says

Current operating systems and user agents don’t usually have a concept of multiple mouse inputs. When more than one mouse device is present (for instance, on a laptop with both a trackpad and an external mouse), all mouse devices are generally treated as a single device - movements on any of the devices are translated to movement of a single mouse pointer, and there is no distinction between button presses on different mouse devices. For this reason, there will usually only be a single mouse pointer, and that pointer will be primary.

 3.10. TODO: test nested dragging

 4  Vue version#

I think it’ll look something like this

<template>
  <g
    :transform="`translate(${pos.x},${pos.y})`"
    @pointerdown.left="start" @pointerup="end" 
    @pointermove="dragging ? move($event) : null"
    @pointercancel="end" @lostpointercapture="end"
    @touchstart.prevent="" @dragstart.prevent="">
    v-bind:class="{dragging}"
<slot />
  </g>
</template>

<style>
  g { cursor: grab; }
  g.dragging { user-select: none; cursor: grabbing; }
</style>

<script setup>
// pos is a prop {x: y:}

const dragging = ref(false);

function start(event) {
  if (event.ctrlKey) return;
  let {x, y} = convertPixelToSvgCoord(event);
  dragging.value = {dx: pos.x - x, dy: pos.y - y,
                    pointerId: event.pointerId};
  el.setPointerCapture(event.pointerId);
}

function end(event) {
  dragging.value = null;
}

function move(event) {
  if (!(event.buttons & 1)) return end(event);
  if (event.pointerId !== dragging.value.pointerId) return;
  let {x, y} = convertPixelToSvgCoord(event);
  $emit('move', {
    x: x + dragging.value.dx,
    y: y + dragging.value.dy,
  });
}
</script>

 5  Notes - event log#

eventlog.html

Testing a click:

 6  Notes on dragging#

tests.html

TODO: test tablet

 7  Variations#

nil

 8  Notes#

pointerdown + pointerup will trigger click (left mouse button) or auxclick (middle mouse button) or contextmenu (right mouse button)