Spine bounding box for UI Events

foriero

Hello Harald,

Is there a way to precisely detect UI events using Spine bounding box?

Thank you, Marek.

Just want to add that to use spine for fully featured UI systems we need bounding boxes to work with UI Events detection.

Nate

Harald (please note the spelling!) is on vacation for a few days. He'll catch up on forum and other things ASAP.

One approach is making UI controls in Spine and doing layout in Unity. By UI controls I mean buttons, checkboxs, etc. You'd position these as separate skeletons as needed in Unity. Some consideration may need to be made so they can be resized based on their content (eg, the text on a button) or layout rules (I'm a huge proponent of using tables for UI

fight me). Typically bounding boxes for UI are rectangular, so the bounds for user interaction can be that of the UI control's Unity GameObject, rather than needing to place bounding box attachments in Spine.

Another approach could be to build the whole thing in Spine. Layout is much more limited because things are placed absolutely in Spine, so if the screen is wider/etc there's no easy way to get a different layout. Also if the whole UI is one skeleton, you'd have to manage interaction somehow (probably bounding boxes). It just seems more complex overall than having a skeleton per UI control.

foriero

Yes we are pretty much ok to make UI layout ( buttons, dropdowns .. ) in spine but we need a direct approach to design clickable area in the Spine. Yes you are right that a typical approach is to have rectangles as active ui events sensible area BUT that is exactly what limits us to make a unique and stunning UIs.

So we have two pains here:

1) to reconstruct the clickable area ( for now even just rects ) is simply a tedious process. which means making bone hooks, exposing them in unity. This does not cover when animator resizing the whole or the parts. When that happens then we need come back and forth asking them to animate also the "hooked" bone.

2) we are not able to make changes in shapes and make them still clickable. ( it is always rect that is ok when hooked bones is scaled, but does not cover shear, polygon, or sprite swapping cases )

My idea is to provide in spine similar approach to bounding boxes. We can call it UI polygon? Definitely we need similar to what you see in alpha raycaster. Precisely detect UI polygons in Unity but defined in Spine.

https://assetstore.unity.com/packages/tools/gui/alpha-raycaster-28601

I don't think to make the whole UI in spine is a good idea. There you are very limited or can be used in very limited cases.

Nate

foriero wrote
Yes you are right that a typical approach is to have rectangles as active ui events sensible area BUT that is exactly what limits us to make a unique and stunning UIs.

I don't think this is true. It's extremely rare to need the precision of a polygon for UI interaction, especially for touchscreens where the input device itself doesn't have such precision. You might need the precision for gameplay, but not for UI (buttons/etc).

foriero wrote
My idea is to provide in spine similar approach to bounding boxes. We can call it UI polygon? Definitely we need similar to what you see in alpha raycaster. Precisely detect UI polygons in Unity but defined in Spine.

Seems to me we already have this, that is what bounding boxes in Spine can do. How would your polygons be different?

foriero

Very rare is what makes professionals stand out. And if we have this precise detection we would be happy to use it on many places.

Yes bounding boxes are fine. I can not imagine that I do some kind of hit sword detection in unity UI. But if you can imagine the usecase then we need exactly the same as bounding boxes but name it ui polygons ( boxes suggesting it is just a rect ) so that Harald can distinguish them in UI from bounding boxes.

Nate

foriero wrote
Very rare is what makes professionals stand out.

I'm not talking about cutting corners, I'm taking about what is actually needed to build good UI with good UX. I have built a lot of professional UIs in my life and have never found the need to use a polygonal clickable area. You've seen Spine, if you want to see my mobile UI design check out GHH. In general the clickable area should be larger than the visuals, especially for small visuals. If a rectangle is not a good fit then I would use a circle ((x1-x2)^2+(y1-y2)^2 < r^2).

foriero

Agree, most common shapes in UI are rectangles, rounded corners, circles, following ovals, and uneven shapes. Spine ui polygon should either follow graphics ( alpa raycaster ), or rectangles, rounded corners, and circles that are easy to be produced and then in special cases allow us to define polygon shape. In all cases the generated ui polygon will be later translated in runtimes into ui events area.

Anyway a streamlined workflow is very much needed so that we can just spawn the skeleton in the UI and have ui events out of the box in those areas that animator designed to be clickable.

Nate

foriero wrote
a streamlined workflow is very much needed so that we can just spawn the skeleton in the UI and have ui events out of the box in those areas that animator designed to be clickable.

That would be nice, but I doubt we will build it any time soon. There is no reason to wait for us though, everything needed is available.

foriero

Let's wait for Harald. I want to know his opinion on the roadmap for this feature. :-)

Harald is back? :-)

Harald

Back from vacation now! 🙂 (actually yesterday, but fixing an urgent issue and digging through the forum took it's time)

I will have a look at this topic from the implementation view tomorrow as my brain batteries are too exhausted for today.

Nevertheless I have to state my point of view on UI UX in terms of interaction precision and button size:
Please always be sure to test various enlarged button collider sizes on any device before going with the precise 1:1 shape. Depending on the device and the surrounding UI elements, it might feel much better to have a larger collision area than to be spot on. It will also depend on cursor shape (circular crosshair vs arrow, etc). Let alone when it comes to joypad input, as mentioned on this GDC talk. Without seeing your UI, I would bet that it feels better when you click on an enlarged rectangular collider than to have a precise collision shape.

foriero

Yes agreed that the active area should be bigger but the shape should be in the hands of spine illustrators/animators. Since we move things and scale or make other changes to the skeleton we need to be able to animate that shape and then in unity let the UI system automatically generate that UI active area. Does it make sense. This will give us tremendous creative UI freedom and the UI skeletons will work out of the box in Unity without needing to expose and follow bones with primitive rectangle. I really see it as THE next step for Spine Unity UI. Thanks to this we will be able to create super complex UIs without exposing super complex bone structure. Each UI area ( one skeleton can have more UI areas needs to have sort of like ID so that we can react properly on different active areas on one skeleton ) Does it make sense? It should be pretty straightforward. In Spine have similar to bounding box, I call it ui polygon, give it ID and match and animate the polygon as animators need it. Then in Unity "JUST" automatically create some Spine UI Event detector. Or we can use bounding box if it works as an animatable polygon.

Harald

While I can see your point, I see some problems even when assuming that the shape is considered important:
1) What components would you want to use for the UI ray cast intersection detection? I know of no UI equivalent to PolygonCollider2D. Having a parallel ray intersection system running for testing PolygonCollider2D UI objects does not seems like a clean option.
2) If you would want to use PolygonCollider2D (perhaps via manual raycasts), any bounding box shape changes (real deformations, not rotation and scale) through animation will require to discard and regenerate the collider vertices, which is somewhat suboptimal.

foriero

using UnityEngine;
using UnityEngine.UI;
 
[RequireComponent (typeof (RectTransform), typeof (Collider2D))]
public class Collider2DRaycastFilter : MonoBehaviour, ICanvasRaycastFilter
{
    Collider2D myCollider;
    RectTransform rectTransform;
   

void Awake ()
{
    myCollider = GetComponent<Collider2D>();
    rectTransform = GetComponent<RectTransform>();
}
 
#region ICanvasRaycastFilter implementation
    public bool IsRaycastLocationValid (Vector2 screenPos, Camera eventCamera)
    {
        var worldPoint = Vector3.zero;
        var isInside = RectTransformUtility.ScreenPointToWorldPointInRectangle(
            rectTransform,
            screenPos,
            eventCamera,
            out worldPoint
        );
        if (isInside)
            isInside = myCollider.OverlapPoint(worldPoint);
 
    return isInside;
}
#endregion
}

Also check Alpha Raycaster : https://assetstore.unity.com/packages/tools/gui/alpha-raycaster-28601

If we can get at least rectangles working out of the box in the UI I would be very happy. The only issue is if you define more bounding boxes in one skeleton. How would you discern it in Unity?

Harald

Thanks for posting the Collider2DRaycastFilter code. While including a component like this in spine-unity would take it too far away from the main focus, we can provide a version of BoundingBoxFollower capable of generating a PolgonCollider2D for a SkeletonGraphic object.

I have created an issue ticket for it:
[unity] BoundingBoxFollower does not support SkeletonGraphic · #1770
Nevertheless, it will take some time until we get to this ticket. Also e.g. one of the other more important tickets requested by you are in the queue before this one. I guess that you can come up with a quick implementation yourself until this feature is implemented.

PS: The alpha raycaster demo video is funny: imagine e.g. the winamp UI only reacting to clicks that precisely hit the symbols of the play/pause/next buttons - especially the pause button will be hard to hit! 😃

foriero

Well, we use spine for some notation objects on a stave. Not for SMuFL font as you can see in the video below but for animated characters. We would really love to see alpha raycaster working also in spine. I have special raycaster that is capable to detect mouse over for transparent pixels in a closed graphic shape.

https://www.dropbox.com/s/e4ztwct6wgbuwq5/snapping.mp4?dl=0

And here just what is possible in Unity.

https://www.dropbox.com/s/r8tfaobhlili73q/pathetique_4k_60fpsl.mp4?dl=0

Harald

No offense intended, but I do not like the requirement to hit exact pixels as a user. IMO the precise shape only becomes usable when dilating the pixel area by a certain amount. Just scaling the shape would not work here, as it would not expand it by a constant distance. When dilating, you basically end up with two to three rectangles again, one for the vertical bar, one for the notehead, and one box for all flags combined. Otherwise you will get frustrated when clicking between two flags of the same note, or miss the vertical bar since it is just so thin.

Having a pixel-precise solution would also be bad in terms of a half note which is not filled - do you really want to require the user to hit the black area only and not the transparent center?
https://upload.wikimedia.org/wikipedia/commons/thumb/0/09/Half_note_with_upwards_stem.svg/461px-Half_note_with_upwards_stem.svg.png

foriero

Well, as I wrote we have alpha raycaster that can detect that you are in the middle of half note and still keep it as it is filled with color. In other words those closed transparent pixels are perceived as they are not. I can share the code if you want. :-) Regarding clicking on stem lines or flags. Yes, you are right boxes are better there, but usually you don't click them. What is active is a notehead and may be a small square area at the end of the stem line.

 public enum OuterEdgeDetection
    {
        NativeRect, // recttranform
        ClosestRect,//closests rect to that graphics 
        Fill,//detects transparent pixels in closed graphics shapes, inside half note it is detected as fill//
        Alpha//full alpha detection, also inside half note//
    }

Nate

Pixels or polygons are the wrong type of hit detection to use for those musical notes and for all other user interaction. Just because you have the technology to do it doesn't mean you should. Rectangles or circles are the right solution, but it could also make sense to place a bounding box with 4 vertices in Spine that best defines what a user would expect the clickable area for a glyph to be. Then at runtime you could see which polygons you hit and choose the one that was hit closest to its center, so you have good behavior when glyphs overlap. However, you could and probably should just do that same approach using the AABB of the entire note. There's no reason to use a more precise box (it will just make hitting the notes harder) and you will still have good behavior when they overlap. You are going to have to write application code though, not everything can just be exported from Spine. :p

Rectangles or circles are still the right solution even for many cases where the hit detection is done for gameplay reasons rather than user input, eg as I just described in this post:
Spine - libGDX collision detection

foriero

I'm not saying I want spine do pixel detection. It would be nice to have but it is not relevant for games. Rectangles are all we need. And I like the bounding box follower given that we can spawn it with SkeletonUtility. For example we have skeleton that defines a few bounding boxes then SkeletonUtility will most likely have a button to spawn those bounding boxes followers so that we can hook up UI scripts on them. I will be happy with that solution.

Regarding the note detection, I follow how Sibelius works. ;-) And there is definitely pixel detection in place.

Harald, may I ask you to have this ui bounding box follower done first? Thank you.

Ok. I see that bounding boxes are bound to bones. So we need two spawning strategies.
1) mirror gameobjects hierarchy to bones where bounding boxes are located.
2) allow to spawn flat gameobject hierarchy structure so that we don't have to dig deep in the hierarchy. Some of our skeletons are very deep.

다음 페이지 »