Note that the contact privacy setting was only introduced in 2021, so if you made your transactions private a long time ago, you still have the contacts to do. (Both settings are set to public by default.)
Also, maybe don't feed your contacts list to apps you use. Any time an app invites you to upload your contact list, mentally replace “your contacts” in the prompt with “other people's contact information”.
“Upload other people's contact information so you can have a more convenient experience” would get at least slightly fewer takers, I think.