I can relate. I have anxiety symptoms that become aggravated when I am put in severely overstimulating environments, from which I am not socially or physically permitted to take control of. Being denied the ability to negotiate with my environment fills me with dread. Unfortunately, sometimes the people that we care about, as much as we love them, are terrible at managing their own personal lives, and often times their inadequacies spill over on those around them. Always complaining about money, refusing to take care of themselves, poorly managing their children or pets, or making brash life choices---these are all things that through empathy, can end up hurting you as well. Its easy to ignore the problems of strangers, whose troubles have no effect on your life, but watching the people you care about struggle or be cruel to you, and then fail to improve their own situations becomes worry on your part, which soon enough turns to frustration and distress over repeated occurrences. When you are already struggling to manage your own worries and fears, their problems become more than you can bear, and that isn't your fault. There is NOTHING wrong with removing these people from active participation in your life. You can care from a distance. Because these are the type of people who don't care that they make you feel worried or upset, they are so busy with the problems they cause themselves they can only care about what YOU can do to alleviate their stress. You may struggle with some feelings of guilt for putting some of these people on the back burner, but ask yourself "if I am not going to take care of myself, will they?" Typically the answer is no. You can always allow them into your life conditionally, or periodically (aka "We can hang out as long as you don't talk about/do xyz, because it upsets me." OR cut social interactions way back. Perhaps instead of socializing with said person twice a month for 4 hrs maybe get together for lunch once every three months. You don't need to apologize for taking care of yourself, because they certainly won't.