You left their party after a couple hours to relax. I'm guessing you were feeling a little overstimulated and needing a break, and your partner probably realized this as well and was willing to let you decompress.
Then you accepted an invitation to socialize with someone else, which to your partner probably seemed like you needed a break from them but not someone else. Of course, this is not a one-to-one situation, I'm sure swimming and a sunset is much more restful and relaxing than a birthday party, but from their perspective they probably wished you could have stayed but respected you needed a break. Now they're finding out that you didn't necessarily take a break from socializing, so it looks somewhat more like you needed a break from them.
Again, I know that isn't exactly what happened, and it isn't the message you meant to give with your actions, but that's my attempt to see things from their perspective based on what we've been told.