Outline & Selection highlight experiments [preview]

I've been playing around with outlines recently.

There is a very simple way to make outline shaders without a single line of code that is very easy to use and setup by using the the 'grow' property of the standard material.
So I thought I'd do a simple video about it

But before I noticed it, I was already doing a highlight effect and though it might also be interesting to show in a video
And then entered a rabbit hole of trying to make the outline look perfect.

First I though that usually those show up behind walls too.

So I've also made the outlines also work behind walls.

There was a bit of fiddling around with render priorities to make both shader with no depth test work together, but I think I've eventually made it work.

----

So here's a bit about outlines

 Making a basic outline using the 'grow' strategy is very easy, but it doesn't produce the best effects.

But due to Godot still not having a stencil shader, there not much we can do without getting really advanced with math. So that's what we'll use.

I've also found while searching an interesting project that claims to have made a proper outline shader with godot 4.2 but no code yet.

What godot can do it use a very complex post-effect shader to calculate outline based on edge detection techniques. But I think that's a bit overkill. And it can have some problems with TAA.

On the other hand, the grow outline is very simple. You scale you mesh vertex outwards and flip the culling faces, that way, only the faces that go over the original mesh shows up. Godot even has a way to create that new mesh for you to have the outline made without shaders.

There are a few problems this strategy. But we can overcome most it them with a few things.

First problem, depending on the mesh complexity, you can see outlines in places that you'd not expect when they overlap. But as long as you don't use big outlines on very detailed meshes that's not a big issue.

Second problem, we also have an inconsistent outline thickness, as the mesh keeps a constant grow no matter the camera distance. But that's easy to fix.

There is a very simple shader for that. It is a very simple shader with a couple of lines of scary math to figure out how much it needs to grow to keep the line thickness constant based on the camera zoom.

And another problem that caught me by surprise.

When using the grow shader, somethings the mesh faces scales up disconnected.

I originally though this was a bug, but that's actually just how the GPU behaves. Turns out, when a face uses flat shadding normals and you scale them with a vertex fragment shader, they'll scale to different positions.

This character has some faces with smooth shading, and some with flat shading.

The sections of the helmet that uses flat shading breaks apart. Making the outline look very bad.

The fix for this is actually by modifying your models. You can make the models keep a somewhat flat look with smooth shading by applying weighted normal it them.

It changes, a little. But still keeps a somewhat flat shadows.

Here's the original with flat normals below.

But the difference in-game is very obviously better.

Its a bit of an extra step, but one very much worth it if you indent to use outlines in your game using this outline technique.

----

So that's what the next video will probably be about.

I'm still not sure if I should make a full video talking about all of this, or separate things in smaller 2-3 videos.

• One for grow shader without code

• One for making outline shader look good

• One for drawing behind walls and selecting highlight

I'm still trying to think on the 3D scene I'll use and how to show it for the tutorial, so the video will probably take a while.

But that's it, a quick preview of what's coming next.