Something to think about is the "adventure" part of the whole thing. The more realistic you get with the setting, the more limited you are. Dungeons full of treasure and monsters might be pretty much off the menu. A mundane murder mystery might be something to occupy the players, but that would be more investigative than action-oriented. Dealing with some bandits or wild animals might be more realistic, but then, that#s what professional hunters and soldiers are for. Only if the players are that, they will be part of a chain of command, having to take orders and only limited freedom to make their own choices.
Traveling around could be another problem. Most people in medieval times probably lived a pretty stationary life. Then again, perhaps restricting player mobility a little isn't bad in a freeform game, if the game plays out more as a sandbox than a GM-driven thing. If everyone can go off in a different direction, things might get difficult to handle.
How about non-humans? Do they exist at all? If they exist, will they be playable races, or NPC "background elements"?
Lets brainstorm this a bit and see what comes up.