It took me way too long to make this work reliably, but I can now move my eyebrows. It's kind of subtle but also not since people pay surprisingly much attention to them.
The face is also now modeled entirely in Blender using shape keys, so the motions are much smoother than with the old image-based approach. Got to love shape keys. So smooth and simple both in terms of making the art and using it from the code. Learning how to export them took a little bit of time, but well worth it.