Since I might be looking to start/run/play a new game, may as well put in writing the custom rules I have laid out in my brain that have accrued over the ages. These are in general system neutral, and more for making a harmonious and fun table. I'll add reasoning behind each as well, since some may seem counter-intuitive.
Edit: Edited to add number 6
- Every player character must know one other player character in some way.
This one is stolen from one of my early GMs, but it always made sense. Beyond avoiding just 'I don't trust anyone and thus we go nowhere' game starts, it also helps character building, especially for new RPers. Knowing others gives your character motivations and help to guide how you play them.
- The group must decide as a whole for any course of action, and follow through on it to their best ability.
Similarly, this one helps to cut down on potential infighting. A character can grumble or complain, but the players agree that any action the group decides on they will follow to the best of their ability, and their character won't sabotage it. If this isn't a rule that can be followed, a character likely needs to change.
- Any situation that is uncomfortable for players in any sense can be stopped with no questions asked. Any further play of the scene for the story can be handwaved or retconned, depending on the need.
Obviously GMs and players should agree on situations that are ok and not ok that might come up for a game, but sometimes games go places, and you want to be sure all players know they can voice a desire to stop and not be judged for it.
- You don't need to know every rule in the rulebook, but you should at least learn the ones your character will use regularly.
This can vary heavily by system and circumstances. My biggest example is Shadowrun. If you plan to play a mage in Shadowrun, you should know the magic rules. I am always glad to help people figure it out, but if after that you are unsure, it might be best to play a different kind of character than have to ask for help every turn. However, a mage doesn't need to memorize the hacking rules.
- You get one free mulligan on character creation. At any point in the game, for any reason, you can change your character to another one, with the same resources you would have had at that point as if you had always played them, and other players treat that character as if it was always there.
If you play games with new players, or players who are unfamiliar with the system, sometimes they won't know what they will enjoy until they try it. You don't want people playing a bland/boring character because its the safe option, out of fear they will be stuck with a riskier design they might love. I've had players use this to change pure fluff stuff about their character, or do a complete rewrite because what they thought would be fun to play simply wasn't.
- If the game is looking for new people, and you find someone who might be interested, cool! Please talk to the DM first, so they can clear up any potential issues before an invite is formally given to the person to join.
Beyond just 'don't invite people randomly to events without clearing it with the people who are running said event' deal, TTRPG tables should be safe places for people to play a game and enjoy themselves with comfort. A new person can upset the whole balance of the table, and more than a few of my tables have been ruined because a new person got added, made others feel uncomfortable, and the game was over.