I wouldn't call it an immersion breaker, but it does rob you of a sense of achievement.
If your playing Skyrim on PC, you absolutely MUST get the Deadly Combat mod. It changes the entire game. Armor is less effective, but blocking is more effective, the player and enemies are almost always staggered by small hits, and get really knocked around by big ones, power moves and bashes take up more stamina, but stamina recovers faster, arrows hurt more, and magic like Ice Spike or Fireball sends you and enemies flying.
So, one on one fights are now way more dynamic, with you both having to move around more, group fights are tense as hell since being bracketed is now an actual tactic, and getting flanked is going to stagger you into another guys blade, Mages are now fucking deadly from a distance, but near helpless if you close in, an Archer is practically dead already if you close in, and a group of Mages? Forget about it. You'll never win.
I dunno if the shield perk Elemental Protection helps. I'll come back when I've gotten it.