my shader works! I made a tricky shader from scratch! yay learning new things! 😁
This thing takes tile data in the form of a tiny image where each red pixel represents a tile ++ what kind of edge it is in the green channel (making it look yellow) and the shader turns that into a smooth bordered shape.
So the battle engine can now generate one of these tiny pixel-data images for weapon/movement range and get a nice way to show that range.
I was asked what the colors meant so figured would put answer here with pictures.
So: the shader needs to know what kind of edge a tile is; does it have neighbors to the north, northeast, east, etc? It turns out with 8 neighbors, there are exactly 256 possible combinations of neighbors (2^8).
Know what else gets passed around as a number between 0 - 255? That's a RGB color channel if I've ever seen one! (Background: colors are represented on computer screens as three numbers; Red, Green, and Blue, each with a value between 0 and 255).
So while it's possible for the shader to query the pixel image and figure out the edge itself, it's more efficient to just do that calculation once and bake the results into one of the color channels. I was already using Red for the binary question "is this tile filled or not" so Green was the next available one.
I arbitrarily assigned each of the eight directions a power of two, which you can see in the first part of the image. Then, using the power of BITWISE MATH, we can very quickly pack and unpack those values to and from a single number for that pixel.
For example, if a tile had neighbors to the north and east (the top-right example), we would say:
edge = edge | 1 (edge is now 1)
edge = edge | 4 (edge is now 5)
Now that may seem like a fancy way to say "plus"... and you're completely correct. But the really cool part is that we can now ask that final "5" value whether 1 was added to it, no matter how many other powers of 2 we added as well!
has_northern_neighbor = edge & 1 (this will return a truthy value!)
This is really wild. If I gave you a single number and asked you "what numbers did I add together to get this?" you would have no way of knowing. But if we know that we're ONLY adding together powers of 2 and I give you a 5 and ask you "did I add a 1 to this?", the answer has got to be a yes because that's the only way an odd number would have been added. The same is true for the rest of the powers of two.
So to answer the question "what do the yellower pixels mean in that little map?", those just have higher a higher "edge" number because the higher powers of two ended up on the western side of the pixel, which means when they're present they're adding a lot more to the yellow channel.
A much nicer line-of-sight experience than the hard-cornered tiles I was using before.