, you should take a helicopter view and see that a punch, will lead to counter punches, which will lead to potentially full blown civil war.
You got that wrong. Nazis are already being violent; punching a nazi is not starting violence, it's a defensive measure, it's a response to violence. But sure if your response to violence is "let's sit with the nazis and make a nazi bar" then sure, you do you.