I can absolutely relate to what you're describing. Sometimes you need a break from a specific person, and it is not because you dislike them, you just really need a break. This is, from what I can tell, not a common neurotypical experience and so not something that neurotypical folks can relate to, and it's hard to put it into words as a way that sounds anything other than "I don't like being around you".
The closest I can come to putting it into words is that sometimes my mind behaves like a cat. One moment you're both enjoying each other's company, the next I'm totally full and need to go. It's not about any specific person, it can happen with anybody I'm around.
It's good that you're working to understand how to express your experience to others, it's difficult but worth the time to strengthen relationships. Hopefully your partner understands that it's not the easiest thing.