That, and I think they trace a direct lineage back to the original Harvey Ross Ball smiley face, which was also yellow.
Me, I don't particularly care about matching emoji skintones to myself. Rather, I'm much more annoyed that I can't tune the 🏍️ emoji to match the color of my motorcycle. What a rip off.