This is sort of a fundamental issue with mesh based skinning in general, in 3D or 2D and avoiding it or minimizing its effects is kind of an art unto itself. A pretty archetypal example in 3D is rigging a character's shoulders. There's a reason a lot of games
particularly early 3D games
have characters with big shoulder pads that can hide this sort of thing!
One of the core issues here is that anything that causes the mesh to move or rotate in on itself is inherently going to change volume, pinch, or crumple, which is just inherent to the skinning systems that are fast enough to run in realtime applications.
A second issue is that as the mesh deforms, pixels don't deform smoothly, which you can see in that last image as the linework is thinned in some places and thickened in others. The Spine docs have a good example for this and how to think about it (Mesh attachments - Spine User Guide, scroll down to "Edge Examples" section). Again, inherent to the technology.
So, how to address it? Ultimately an engineering and design problem, as well as an exercise in the disciplines of "good enough" and "doing without" when it isn't worth the cost. Ultimately you need to make clever decisions about how much effort goes into the rig, what range of motion you want to have well-supported, as well as what kind of smoke and mirrors style stage magic you might employ to cover up any rigging crimes. The options are typically "cheap", "good looking", and "easy to use"; choose two.
Okay, enough of my blabbing, specific tips:
-
For highly articulated structures like a tail, this stuff is hard mode. It's easier to get a wide range of motion and good interactions if its initial drawing straight and in the middle of its expected range of motion, instead of stylishly curved to begin with (hence the infamous T-pose in 3D graphics). If it's already curved like it is in this image, the range of motion you can realistically pull off is going to be smaller.
-
Possibly not an option when working with someone else's art, but simplifying or obscuring the parts of the rig you know are at risk can help mitigate noticeable errors. High frequency detail in a spot that needs to bend really tightly is the worst case for this.
-
It looks like you're already doing this to a degree, but breaking bits and pieces out to deform independently can help and can cover up the messy bits. This is what the aforementioned shoulder pads do in 3D; they don't need to deform and sit on top, so they hide the ugly shoulder pinching.
-
I've found for tail-like rigs, it can help to weight with not just the nearest two bones, but the nearest three instead, which gives it a more bezier curve quality. The downside is this is more finicky to weight correctly, and when the tail bends tightly or in an S-curve you can often expect it to thin out dramatically. Sometimes that's okay, sometimes that's a big problem.
-
If it's already curved, more vertices around the tight corners will give you more control for this, but expect weight tuning to get much more finicky as a result, and you might start to notice a sawtooth pattern in your texture deformations as well. Don't go overboard.
-
If an overlap or deformation is too difficult or too expensive to address, consider ways of hiding it. In fast animations, for example, it doesn't really matter too much if a frame in the middle looks a little off as long as the start and end poses look solid, the rest tends to get lost in motion. That's one of my preferred ways to hide sprite swaps right now.
-
When setting up the mesh, any pixel that is very close to an edge will generally deform cleanly, and anything in a triangle whose vertices tend to move as a group will have the same. Knowing that, you can take steps to preserve your most important lines by making sure they there are edges running parallel to them to support them. When working with line art in Spine, it's generally more important to preserve and control the linework, and in deformation you do that by matching linework to edges.
-
A specific trick here: on a project that involved animating illustrations with very clean, distinct linework and crosshatching, whenever I traced the edge of a surface that was at risk of a bad deformation I would run a parallel line of vertices so that there was a line of quads that fully contained the linework I wanted to keep control of. As long as vertices on those parallel lines had similar weights it held up well against very aggressive animation. This is labour intensive, so use it sparingly.
Also, when setting up extra vertices like this sometimes the edges don't generate where I want them to. In this case, reference the documentation page above, there's a way to force an edge between two vertices by dragging from one to the next.
-
One of the solutions in 3D for this problem is to use morph targets to correct the mesh when the rig moves to an extreme position. Spine has animator-derived vertex deformation support in the animation timeline, so you can manually move the vertices to a place that's less offensive if needed.
-
Another solution pulled from 3D is the idea of adding additional bones to help control and manage the shape of the structure. A shoulder rig, for example, might have a set of bones that roughly map to individual shoulder muscles, and be set up by the rigger to automatically follow arm motion. The parts of the shoulder that correspond to those muscles can be much more accurately weighted this way.
In Spine you can do something similar by adding bones and then either animating them manually or driving them with constraints. Adding additional controls to places where you don't feel you have enough control of the deformation can help a lot.
-
Finally: sometimes the right call is just to accept that this will happen. Maybe that means ignoring the deformation because nobody will notice it in practice, or maybe it means being careful when animating to avoid showing any of the seams. It's worth picking your battles on stuff like this; avoid wasting a lot of time fixing errors with low impact, and instead spend it on places where the error is most obvious, or where you can get your biggest wins and flourishes by going the extra mile.
Hopefully some of that was helpful!
