Remove bevy_ui::Interaction in favor of a bevy_picking based solution

[alice-i-cecile]

bevy_ui's Interaction component is clunky and crude. Now that we have proper picking support, we should replace it completely in some form. See #8157, #10141, #9240, #7257, #420, #5769, #1239 and #2322 for prior related discussions.

I'm aiming to tackle this over the next week as my primary task in the lead-up to the Bevy 0.15 release candidate :)

[alice-i-cecile]

Bevy's Interaction component is very simple.

enum Interaction {
    Pressed,
    Hovered,
    None
}

Ever since its inception, it has had a number of serious problems:

1. It lives in bevy_ui, despite covering concepts "pressed" and "hovered" that are useful when referring to gameplay entities, including in games that don't use bevy_ui.
2. It doesn't cover related but distinct states that interact with the same properties: "focused" and "grayed out".
3. It's nowhere near sophisticated enough to handle complex but common behaviors like drag-and-drop.
4. Detecting if a button has been "just pressed" requires an O(1) check using change detection for each button.
5. Information about which button was pressed is lost, and so this cannot be reused to e.g. handle right clicks to open a context menu.

With the successful creation of bevy_picking (upstreamed from bevy_mod_picking), the time is right to finally improve the situation.

Use cases

There are several things you might want to do with an Interaction-shaped solution.
Let's lay them out, so we can evaluate each proposed solution against them.

Use case: clicking a button with the mouse

The user moves their mouse over a button and left-clicks it. The button is fired as soon as the button is pressed.

Use case: clicking a focused button using a gamepad or keyboard

Suppose we have some way of determining which element is focused.

The user presses Enter or A on their gamepad, and the focused button is fired as soon as the button is pressed.

Use case: context menus

The user moves their mouse over an element and right-clicks it. A context menu is opened with information and actions to perform relating to that object.

Use case: drag and drop

The user moves their mouse over an object, presses left click, moves their mouse somewhere else and releases. The object is moved (in a literal or abstract sense) to the new destination.

Use case: draggable tabs

The application has a collection of tabs, representing different workspaces.

If a user presses and releases the left mouse button on the tab without dragging (a click), the tab is selected.

But if the user instead clicks, drags and then releases, the tab is moved without focus swapping.

Use case: tooltips

The user moves their mouse over a button, waits for a second, but does not press any buttons. Some explanatory text appears about that object.

Use case: Dynamic button styling

All buttons in an app dynamically change their style (font, color, texture, drop shadow etc) based on their state.
We want the following states to be distinguishable:

1. Grayed out: the button is disabled for some reason.
2. Pressed: the left-mouse button or equivalent is currently depressed over top of the button.
3. Hovered: the pointer is over top of the button.
4. Neutral: no interactions with the button are happening.

Grayed-out state should take priority over pressed should take priority over hover should take priority over neutral.

Additionally, we need to be able to distinguish where or not an element is focused.

Proposed solutions

We could solve this in several ways.

Solution: boolean flags

Keep track of the following properties for each button-like UI element: hovered, pressed, focused and disabled.

For an initial implementation, only hovered and pressed will be set by the engine: the other fields are just dummies to allow users to hook into them in a future-compatible way.

This can be done in a single component, or a set of related components, which can be gathered via a QueryData impl. A single ElementState component is easier to work with but couples all of these notions together, which may or may not be desirable.

These values are updated by listening for picking events, except for hovering, which uses bevy_picking's hover map directly.

This solution has the advantage of making dynamic button-styling super easy: the data needed is directly on the object, and we can match on the boolean flags to determine how to style our objects.

Solution: yeet it completely

bevy_picking emits a wide range of events.

Rather than trying to reconstruct an event stream from stateful information (as is currently done in Bevy 0.14),
users should simple read these events to perform actions.

The only use case where this stateful information is helpful is for dynamic button styling. In that case, a dedicated GrayedOut (bikeshed pending) component can be combined with the HoverMap resource to extract the required information on a dynamic, as-needed basis.

This design requires a bit more complexity for the users and has a more painful migration path, but is much more CPU and memory efficient.

Path forward

I'm going to start implementing this tomorrow, aiming for a "just yeet it solution". If I can't get the ergonomics and clarity to where I want it to, I'll add a dedicated ElementPointerState component to replace Interaction, but encourage users to use events for every use case other than dynamic button styling.

[alice-i-cecile]

Prior art from @viridia: https://github.com/viridia/bevy_reactor/blob/9d3aaf730eb8cbc5d24bb204998997449aaed3de/crates/bevy_reactor_obsidian/src/controls/button.rs#L253

[BenjaminBrienen]

Another example of Hover/Pressed outside of UI is minecraft's feature where looking at a block gives it a black outline/border and clicking starts to break the block. This is a great place to use picking. Minecraft clones are a huge userbase! 😄

[alice-i-cecile]

I'm not really up to date at the moment, but PickingInteraction started as a drop in replacement for Interaction, which wasn't quite compatible.

So, it isn't very expressive. It should probably be a stateful mirror of all the existing picking interactions (hover, pressed_by, dragged_by), and future ones like enter (hovered_descendent).

If we want to retain all this state, we should probably also switch to bitflags to keep memory size low.

• @aevyrie

[viridia]

I have been doing these flags as separate components rather than as one component with multiple flags. There are a number of reasons for this:

1. Not every widget cares about every flag. Some widgets don't care about Pressed for example.
2. Some widgets represent these states differently. A slider will have different set of drag state properties than a button.
3. Complex widgets may want to track these states at a finer level of granularity. For example, a slider widget may want to distinguish "pressed" on the thumb/knob vs. "pressed" on the track/container.
4. Different flags might be consumed by different layers. On the web, the visual styling is implemented CSS, whereas the behavior and event handling is handled in JS or natively in the browser. My own widget library follows a similar organization in code. For example, for most widgets "hovering" is a purely stylistic effect which has no impact on widget behavior, so there's no need for the behavioral layer to know about it.
5. Some flags don't even need to be flags. For example, a widget is "focused" when the entity id == InputFocus.0. There's no need to store this as a bit on individual widgets, you can just poll the resource.
6. From an API standpoint, we want to be able to do things like disable a widget without touching any of the other states.

I would divide the set of flags into the following groups:

Universal States - that is, states that are used in the same way for almost all widgets:

• Hovered - used to detect whether the entity or widget part (widget, arrow, thumb) is hovered.
• InteractionDisabled - marker used to indicate that the widget should be grayed out.
• Focus: this isn't a component, it's just comparing against the focus resource.

Note that for hovering specifically, you could use the global picking hover map rather than a component or enter/leave events. However, the global hover map change bit changes every frame (the map is recomputed), which means that if you are doing change detection on the hover map you are updating every widget every frame.

I don't like using enter/leave events for this because the event could be intercepted or re-routed because of a hierarchy change, leaving you with a "stuck key" syndrome. Using the hover map here is more robust.

To make this efficient, we need a component which lets the widget author tell the system that they want to be notified when a given entity, or it's descendants, is hovered. The way I have implement this is to define Hovered(bool) - the widget author inserts the Hovered component, and then the framework updates the boolean variable when the hover map changes.

Specialized States - states which are used by a particular type of widget

• Pressed - used for buttons and button-like widgets
• Drag states:
  • SliderDragState - contains an is_dragging boolean, and an x/y drag offset.
  • ScrollbarDragState - etc.
  • SpinnerDragState - etc.
  • etc.

[viridia]

As a further example, let's dive deep into checkboxes for a second.

There are two ways you can make a checkbox. The "simple" way is that checkboxes toggle their state on mouse down (if they aren't disabled). That is, unlike Button, they don't bother with pressed / dragging / roll-off / etc. So they don't need a Pressed state flag.

The other way is that checkboxes work like Button, that is, the toggle happens on mouse up.

I tend to like the first approach:

• It's simpler
• It's slightly snappier (depending on how long you hold the mouse down)
• Because the action of checking a checkbox is trivially undoable (just click again), there's less of a need to offer the user the option to cancel the action in mid-click.

However, there's nothing wrong with doing the second approach.

Toggle switches are exactly the same as checkboxes, except for two differences:

• They have a different appearance.
• They use the "switch" ARIA role instead of the "checkbox" ARIA role.

Other than that, the behavior is the same.

Radio buttons are simply checkboxes that only toggle ON, not OFF. The mutual exclusion behavior is handled by the radio button group, which is a parent widget. In reactive apps, the mutual exclusion is done simply by setting a signal which stores the id of the current selected radio, and letting the radio buttons react to that signal.