If someone is seriously a supremacist about using darkmode I think their input is invalid enough to be discarded.
On the other hand I don't think downvoting someone you disagree with on anything is in itself toxic, but going out of your way to downvote someone over something this - let's be honest - unimportant is sort of stupid.
EDIT: To be clear I'm saying darkmode is unimportant not the issue of people being toxic.