That's a fine gut instinct, as long as you recognize that moderators can still be a huge problem at times...
to add onto this: i had a very difficult time on a forum I was on as a kid, not because I caused problems on purpose, but because I did work on a newsletter alongside other folks on the site, and someone who was close friends with the moderators got taken out of a higher position in this unofficial newsletter and that position was given to me
and so, they found every conceivable reason they could to get me off of that website. i got banned for the most ridiculous things. at one point, i got banned for something i had on my user title for months and months - a reference to the internet channel, an (at the time) freely available wii shop channel from nintendo, a company that they literally had a subforum dedicated to talking about.
i eventually got out of there as well as a bunch of other friends. i went on to try to run forums of my own, bringing along the knowledge that i should try to not have a trigger-finger for banning folks, although i then discovered that being able to run forum software does not qualify a person to be able to moderate a forum, and that while one shouldn't necessarily be trigger happy, there's a time you have to decide someone is a problem and you don't need to provide them with a platform, no matter how well they sneak around the rules as written.
but anyways, there's always a level of nuance to things. people might have a bad time with mods because they were problem posters, but they might also have a bad time with mods because some whiny entitled brat is friends with them and they made that person upset. that's kinda what i meant to get across with this send post