People mix up "feel responsible for their words and actions and their effects" and "feel bad" an awful lot. Also apparently sometimes holding oneself responsible feels "bad". To people for whom this feels bad I say, "get over it". Since life isn't fair or kind, trying to make it as fair and kind as possible for as many people as possible takes work.
I would add to that, not feeling bad about injustice or harm (particularly on a scale like, say, slavery) is kind of callous. You don't want to feel bad because your ancestors may or may not have been slaveowners? Oh? Just what do you want to feel about it? Happy? Nothing? I feel bad that anyone's ancestors were slaveowners. Do you mean that you don't want to be personally blamed? Why don't you examine why you feel personally blamed by whatever discourse just occurred? In some way that does not cast blame back?
"I don't want to feel guilty for being white/male." is not an excuse to not feel responsible for something you just said or did, and not a reason to blow off someone who brings prejudice or an -ism into the conversation.
And this has been said a million times, but just bears repeating.
It is for the offended to decide whether they can be offended, and not the offender. The offender gets to decide whether they care enough about that person to consider the effects of what they just said or did and decide if it warrants further action, an apology, an examination of things.