I feel the same and I think I've narrowed it down to a handful of things, in no particular order.
The environment design. Fallout is mostly wastelands, with just a few settlements scattered around. Everyone is fighting each other, plus the monsters that are encroaching on civilization. Everything is a shabby remnant of the past shoddily cobbled together. Even the entire settlement system in Fallout 4 is based on gathering scrap and taping it together. In Skyrim, you can mine and process the minerals to make the nails to put your house together. Skyrim has ruins and remnants of past civilizations, but a lot of the buildings and infrastructure are still in good condition, and there's fresh growth. The wilds of Skyrim are much more diverse than the wastes of Fallout. Fallout 3 in particular has the annoying green filter on everything unless you mod it out. It doesn't feel like there's really a world left to save- it seems like everything is doomed to chaos and anarchy.
"Survival". I would not put Fallout in a list of survival games, but it does borrow a lot of elements from the genre. I understand what they're going for, but I don't like the resulting gameplay. Constantly scrounging for weapons, ammo, and resources getd really boring really fast for me. Managing health and Rads too. Every combat effectively takes twice as long when you factor in the time you spend to recover the resources you used.
Guns. I know there's a schism in the Skyrim community between those who mod in guns and those who don't. I see a few problems with guns in Skyrim, and most apply to the vanilla Fallout games too. BGS just isn't great at gun play. The feel of the weapons, the environmental design, ammo distribution, enemy AI, physics engine, the sound design... BGS isn't particularly great at any of it. When the ranged combat is a supplemental element of the gameplay that's fine- Bioshock has 2 great games despite mediocre combat mechanics, and the Elder Scrolls games are similar with their bows and ranged magic. Fallout puts the ranged combat front and center, and it falls apart.
Progression. I think this is why I love Skyrim, and the source of it's commercial success. I was no stranger to RPG's before Skyrim (both videogames and tabletop), but the ones I enjoyed were imusually in spite of the leveling systems. Usually a lot of grinding and overly complicated systems with points, skills, abilities, etc.
Fallout uses one of my least favorite systems- general experience gained (mostly through combat) that leads to an overall character level increase, which then grants points that can be used to improve specific skills. You want to get better at lockpicking? Go kill something. Barter, speech, science, repair, medicine... The answer is to kill something. Improve the Energy Weapons skill? You can kill something with Small Guns or Melle and it's just as effective. It completely disconnects the actions you take as a player from the development of the character.
Skyrim is the opposite. To get better at lockpicking, you pick locks. To get better with a shield you use a shield. It's both intuitive and satisfying. Other RPG's boast more complexity, flexibility, or realism, but I think Skyrim really hits the sweet spot between accessibility, realism, and customization.
This also ties back to the survival aspects I mentioned earlier, because I also felt like equipment was much more important in Fallout. Your damage there is often more about what gun you're able and willing to use than anything to do with your character. In Skyrim, a character with a high one-handed skill and perks can have pretty good damage with just about any one-handed weapon. There's variance of course- you can tell the difference between an iron sword and dragonbone. But the smithing and enchantments mitigate a lot of those differences. If you haven't focused on enchanting yet you might choose a lower-pedigree weapon with a better enchantment.
Lore. This is subjective of course, but I think Skyrim and the rest of the Elder Scrolls just has better lore. The alt-history of Fallout isn't terrible, but it's hard to compete with thousands of years of over a dozen races, various factions, and pantheons of gods interacting with each other. I love reading the books, listening to the dialogue, finding carvings and paintings in the textures or on the item models. Fallout's lore is mostly either "where were you when the bombs fell?", "that asshole leading a group of roughians is being a real jerk", or "Wow Vault-Tex was really unethical ". My wife and I have spent dozens of hours watching YouTube videos breaking down ES lore- everything from speculation about the godhead and very nature of the universe to the one NPC who is vaguely connected to a faction thought extinct.