so there's a few concepts that all get called brightness that it's worth understanding each of
first off, you want to carve out the concept of saturation. saturation is NOT brightness and it's not fair to call it brightness, but it gets confused with that so we have to start there. if you look at an HSL color picker and play with the saturation slider, or if you browse around the paint chips section of the hardware store, you can get a sense of what this means: pastels are low-saturation; specifically they are close to white (but not as close as off-whites are), which is to say they mostly stimulate the eye in all three ways to about the same amount, with just slight deviation from there. you can also find a family of, like... low-saturation dark colors that are like dusty grays, and this kind of points to the things that ARE brightness
so basically there's like "luminosity" and "value" and honestly we don't know quite how they relate but we're pretty sure they are measuring the same thing in different ways. and that thing, as we understand it, is basically how many photons are there
each photon has the same amount of energy as any other photon the same wavelength. photons are always moving at exactly light speed (or the speed of light in air or whatever, but it's close enough), and if you run the equation for how much energy each one has you'll find that that means they all have the same amount. but there can be LOTS of them
this does work slightly differently when you're talking about phosphors that glow, like with a monitor, compared to when you're looking at a sheet of paper and light is being reflected off it, but it comes down to the same thing in terms of how does your eye know. the set of colors you can create with the two techniques is different though
it's also worth noting that brightness is a contextual assessment. if it's the middle of the night and all the lights are off, it takes way less light to look bright compared to when you're in a brightly lit room, which is also way less compared to standing in direct sunlight. plus in addition to large scale factors there's what the color is next to, and what you already know about the thing you're looking at. your brain takes a lot of stuff into account when deciding how to perceive things.