That isn't the type of behavior that I think most find annoying but I'm sorry that you get that reaction at all.
I think many people are so annoyed with feeling they are attacked for eating meat (and I do eat meat) that when that button gets pressed the anger just rises up.
For me I get a little true guilt. I know I'm not helping in the best possible ways that I can, all the time. I'm not perfect and won't ever pretend that I am, and I also haven't given up on getting better. When I go a day without eating meat, I congratulate myself. With a burger. (No, not really.)