Issues using Custom Toon Outlines Render Feature in Unity

Vargon

We're creating a 2.5D game in Unity 2022.3.23f1 URP and replacing all our old Unity based sprite characters with Spine generated models. For the most part, this has worked wonderfully, and we've been able to figure out how to get the models to sit properly in the scene depth and interact with the 3D lighting using the Spine-Lit shader.

The issue we're encountering now is that it is not interacting well with the toon outline shader that we're using as a fullscreen effect in a Render Feature. Essentially the shader uses a Sobel technique to detect screen depth and normals and then draws an outline where there are significant changes. It does this by looking at the URP Sample Buffer and Scene Depth in the shader and comparing nearby values.

What I'm finding now is that unless I put the spine material into the Transparent Render Queue, the Render Feature outlines from objects behind the model draw on top of the character texture. For our previous sprite based characters we were using the AlphaTest render queue, and this worked great, and in all other ways, the Spine characters work the best there too, but this rendering issue is a definite no-go.

Does anyone have any experience with this, or at least understand what's going on? I'm very new to working with Spine models, and I've tried just about everything I can think of.

Spinebot

The issue you're encountering with the toon outline shader in Unity's URP when using Spine models is related to the render queue and how the Sobel technique interacts with the scene depth and normals. Here are some steps and considerations to help you resolve this issue:

Render Queue Adjustment: As you mentioned, placing the Spine material in the Transparent Render Queue causes the outlines from objects behind the model to draw on top of the character texture. This is because the Transparent Render Queue renders after the opaque objects, which can cause depth sorting issues. Instead, try using the AlphaTest render queue or a custom render queue that fits your needs. You can set the render queue in the material settings or via script.
Depth and Normals Handling: Ensure that your Spine models are correctly writing to the depth and normal buffers. This is crucial for the Sobel technique to work properly. You might need to modify the Spine-Lit shader to ensure it writes the necessary information to these buffers.
Custom Shader Modifications: If the default Spine-Lit shader does not support writing to the depth and normal buffers, you may need to create a custom shader. This shader should ensure that the depth and normals are correctly written, similar to how your previous sprite-based characters were handled.
Render Feature Configuration: Check the configuration of your Render Feature. Ensure that it is set up to correctly handle the depth and normal information from all objects, including those using the Spine-Lit shader. You might need to adjust the settings or add additional passes to handle the Spine models correctly.
Example Scene and Documentation: Refer to the example scenes provided with the Spine URP shaders package. These examples demonstrate how to set up and use the RenderExistingMesh component and outline shaders. You can find these examples in the com.esotericsoftware.spine.URP-shaders/Examples/Outline Shaders URP directory.
Post-Processing Considerations: If the above steps do not resolve the issue, consider using a post-processing approach to apply the outlines. This can help avoid issues with render queues and depth sorting, as the outlines will be applied in screen space after all objects have been rendered.

Here is a summary of the key points:

Adjust the render queue to avoid depth sorting issues.
Ensure the Spine-Lit shader writes to the depth and normal buffers.
Modify the shader if necessary to support depth and normal writing.
Check and configure the Render Feature to handle Spine models correctly.
Refer to example scenes and documentation for setup guidance.
Consider using post-processing for applying outlines.

By following these steps, you should be able to resolve the rendering issues with your toon outline shader and Spine models in Unity's URP.