It's important to not stereotype based on a tiny number of samples. If i saw 15 white social workers who were mostly nice, and 2 black social workers who were rude, do you think it would be reasonable for me to say white social workers are nice and black social workers are rude? There are literally hundreds of thousands of social workers. You've seen 17 of them. And only 2 were men. Plus i bet they were mostly all from the same agency, so the person doing the hiring there will very much be filtering who you work with, and the individual doing the hiring at that particular place may choose to hire nice women and rude men.
Bottom line - I totally believe what you say about your personal experiences, but the number of men you worked with isn't anywhere near big enough to say anything about men social workers as a group.