I would say that the audience can be "wrong", where I mostly mean "inappropriate for the specific comedian" at least.
One example that comes to mind is this https://m.youtube.com/watch?v=M4ckxcHx1q4
(Which unfortunately is in Italian). This monologue is an incredible piece on feminism, and the audience was extremely silent and unresponsive, probably because they were a "TV crowd" with a stand-up comedian (the best Italy has ever had IMHO) who was totally out of their league.
In this case, the comedian ended up "rebuking" the audience and I think he was right at that.