As long as at least 1 other person in the country is still voting, "not voting" will never invalidate the system. There's no quorum that has to be met. If only 7 ballots get cast in your state, then all the votes go to the guy that got 4. All you've done by refusing to participate is give the make the other voices louder.