The buttons on trackballs (and ergonomic mice in general) are commonly remappable, but you don't usually get to decide what the ball itself does.
And that's a shame - a ball is a good way to do plenty kinds of input.
So I wrote an AutoHotKey script that tracks mouse input from connected devices and briefly locks the cursor to translate motions into custom actions when needed. This is done using a low-level Raw Input API on Windows, but similar might be doable on Mac/Linux.
This approach offers a good deal of flexibility and works with any kind of trackball (or anything that the system considers a "mouse", really).
The source code and examples can be found on GitHub.
I also have a little documentation page on GitHub pages.