Because at its core your argument is "group of people X feels threatened by group of people Y, but group Y should listen and not feel attacked if someone from group X tells them group Y is dangerous". Replacing group X and Y by any group of people should give you an idea of why this is a bad argument. In other words I'm just applying analogical reasoning to your argument to showcase that in an analogous situation the same argument would be considered aggressive.
Granted, it's not always possible to substitute groups, but if your counterargument is that the substitutions are not analogous you need to present evidence of why that is the case. In other words, why do you think this argument applies to women who are afraid of men but not to whites that are afraid of blacks.