Making a skeleton animation state match another

pixilestudios · 2023년 10월 19일

Hello, at the moment when I need to make a spine character match another one (over the network), I send their current track animation names and track times, as well as any IK bone positions. Then the other client takes that info to pose in exactly the same way. It seems to be 100% correct except when mixing is involved, ie if one track was within our 0.1s mix time, then it can actually get quite inaccurate for that portion.
The code that poses the skeleton basically does this:

state.Update(0.0f);
state.Apply(skeleton);
state.ClearTracks();
foreach (var animInfo in receivedAnimationsData)
{
    var trackEntry = state.SetAnimation(animInfo.trackIndex, animationsByName[animInfo.animName], animInfo.looping);
    trackEntry.TrackTime = animInfo.animTime;
}
state.Update(0.0f);
state.Apply(skeleton);
skeleton.UpdateWorldTransform();

So I tried to send mixingFrom / mixTime with each track so that can be reproduced/set on the other client... but haven't had any luck making it work. I have tried setting the mixingFrom animation with SetAnimation first and then the actual one after, and I also tried manually overriding the variables themselves. Any help or tips is appreciated!

Nate · 2023년 10월 19일

In general network synchronization is quite a hard problem to solve. When an animation changes on one client, it won't change on the other client until it receives that event. You could take into account the event's delay and skip part of the animation so the second client's playback is in the correct place, but this jump will be noticeable.

The general approach in networked games is to have client simulations dynamically catch up to the "real" positions. For example, when a client determines it is behind, it plays its animations faster. Another example, if a client's prediction for where an object is located is wrong, when the server corrects it, don't just teleport it to the right place. Instead, animate it over some amount of time so the movement is smooth and small adjustments can go unnoticed. If you look carefully you can see this happening in a game like Rocket League, especially when playing with poor internet.

For that to work you probably don't want your game state to be reliant on your animation system. Separate the two so that the game state can be correct even if the animations are wrong sometimes, then you can work on ways to get from a wrong animation or animation time to the correct one smoothly so hopefully the user doesn't notice.

Harald · 2023년 10월 19일

pixilestudios It seems to be 100% correct except when mixing is involved, ie if one track was within our 0.1s mix time, then it can actually get quite inaccurate for that portion.
The code that poses the skeleton basically does this:

In order to rule out the problems induced by the network or net-code, you could test what your result is when trying to match the animation state of one GameObject to another in the same scene. If your animation looks equally wrong, then it's likely due to the code applying animation state. If so, could you please share more details (show more code) about how you fill out receivedAnimationsData from your to-be-copied AnimationState?

pixilestudios · 2023년 10월 20일

Harald Hmm well basically the issue is that one client has ongoing character animations in real-time and then when I want to snapshot that to another client, I send the relevant track animations/times. And so far it has been spot on, except if any track was mixing... so really I am just trying to figure out how I can copy/force mixing state I guess.
The to-be-copied code is quite simple:

This list of animation info is sent over the network and the other client then uses the code I posted originally to pose. This is for a case where we need it to match for hitbox reasons, not just general animations where we could ignore discrepancies. And so far the only discrepancy I can find is if any track was mixing during this to-be-copied code. It seems I can get mixingFrom and mixTime, but I wasn't able to apply it properly on the receiving end (or make it match I mean).
Thanks for response!

Nate · 2023년 10월 20일

There is a linked list of TrackEntry objects, starting with TrackEntry mixingFrom. Note there is not just one, the mixingFrom entry can have a mixingFrom entry, etc. You'd need to serialize the entry and the mixingFrom linked list of entries.

pixilestudios · 2023년 10월 23일

Nate Ah yeah so I dug a bit deeper to try and capture the full chain and as many variables as possible but it seems I still can't get it to match properly during mix. I also realized I was not applying <empty> animations as well, but fixing that didn't change the outcome. And I did confirm that there was a linked list of mixingFrom going on.
On the sending client I changed to this:

foreach (var track in cachedSkeletonAnimationRef.AnimationState.Tracks) {
    if (track != null && track.Animation != null) {
        var animStruct = new AnimationIndexTimeTrack();
        animStruct.trackIndex = (byte) track.TrackIndex;
        animStruct.animName = track.Animation.Name;
        animStruct.animTime = track.TrackTime;
        animStruct.looping = track.Loop;
        animStruct.mixDuration = track.MixDuration;
        animStruct.mixTime = track.MixTime;
        reuseableArrayAnimIndexesHitReg.Add(animStruct);
        var mixingFrom = track.MixingFrom;
        while (mixingFrom != null) {
            var animStruct2 = new AnimationIndexTimeTrack();
            animStruct2.trackIndex = (byte) track.TrackIndex;
            animStruct2.animName = mixingFrom.Animation.Name;
            animStruct2.animTime = mixingFrom.MixTime;
            animStruct2.looping = mixingFrom.Loop;
            animStruct2.isMixingFrom = true;
            animStruct2.mixDuration = mixingFrom.MixDuration;
            reuseableArrayAnimIndexesHitReg.Add(animStruct2);
            mixingFrom = mixingFrom.MixingFrom;
        }
    }
}

and on receiving end for each received entry:

Animation foundAnim;
if (animationsByName.TryGetValue(animInfo.animName, out foundAnim)) {
    if (animInfo.isMixingFrom) {
        var trackMain = state.GetCurrent(animInfo.trackIndex);
        if (trackMain != null) {
            TrackEntry addMixFromEntry = state.NewTrackEntry(animInfo.trackIndex, foundAnim, animInfo.looping, trackMain);
            addMixFromEntry.mixDuration = animInfo.mixDuration;
            var mixingParent = trackMain;
            while (mixingParent.mixingFrom != null)
                mixingParent = mixingParent.mixingFrom;
            mixingParent.mixingFrom = addMixFromEntry;
            mixingParent.MixTime = animInfo.animTime;
            addMixFromEntry.mixingTo = mixingParent;
        }
    } else {
        var trackEntry = state.SetAnimation(animInfo.trackIndex, foundAnim, animInfo.looping);
        trackEntry.mixDuration = animInfo.mixDuration;
        trackEntry.TrackTime = animInfo.animTime;
        trackEntry.mixTime = animInfo.mixTime;
    }
} else if (animInfo.animName == "<empty>") {
    if (animInfo.isMixingFrom) {
        var trackMain = state.GetCurrent(animInfo.trackIndex);
        if (trackMain != null) {
            TrackEntry addMixFromEntry = state.NewTrackEntry(animInfo.trackIndex, AnimationState.EmptyAnimation, false, trackMain);
            addMixFromEntry.mixDuration = animInfo.mixDuration;
            var mixingParent = trackMain;
            while (mixingParent.mixingFrom != null)
                mixingParent = mixingParent.mixingFrom;
            mixingParent.mixingFrom = addMixFromEntry;
            mixingParent.MixTime = animInfo.animTime;
            addMixFromEntry.mixingTo = mixingParent;
        }
    } else {
        var trackEntry = state.SetEmptyAnimation(animInfo.trackIndex, animInfo.mixDuration);
        trackEntry.TrackTime = animInfo.animTime;
        trackEntry.mixTime = animInfo.mixTime;
    }
}

I had to make NewTrackEntry a public method to do this code so probably not the best... I wonder if I missed any variables or if I should try another approach like using SetAnimation several times to simulate the chain of mixing, as opposed to trying to force variables.

Nate · 2023년 10월 23일

The proper way would be to serialize a TrackEntry (and other referenced TrackEntrys) to bytes and deserialize back into the same object graph. I'd probably hack on serialize and deserialize methods to TrackEntry and get that working in the same program before trying it over the network.

pixilestudios · 2023년 10월 23일

Ah yeah that'd be an interesting approach, although probably a lot of bytes to send.