Part of learning to say 'no' is becoming comfortable not doing things you don't want to. And with that comes the realization that other people won't do things they don't want to.

So you have to stop believing that people are around you unwillingly.