This is why we need to bring back yae and nay. We used to have two different yes and no words, one set was used in exactly this context. French still has it IIRC. I can't remember which were which in English, I think yae and nay were for positive questions, and yes and no were for negative questions. Aha, quick Google shows that is right, neat.