I was engaged in a poly-amorous relationship for quite a while, and the thing is, we always had rules and boundaries. See, I can see sex and romance as separate to a degree, but the thing is, oddly enough, society's own insistence that it's a romantic activity has ingrained that notion into our minds. On top of that, we all like to feel special, and sex, being a very pleasurable activity, makes us feel special for being able to make our partner that happy. We don't like thinking of the possibility that someone else is capable of bringing them as much, or more pleasure, so we tend to like being the only ones to have sex with them.
That said, I admit, I can be possessive and jealous, so maybe it's good I'm not in a relationship like that anymore, because I do enjoy having... exclusive rights to that activity, shall we say. The big thing though is trust. Even if you're okay with sharing your partner, most people would still be at least a bit upset if their partner went and slept with someone else before even asking if they were okay with it. In that case, it's more about trust. Even more than that exclusivity, we like to feel like we can trust our partners. That's one of the whole points. We live in a world that can be scary and depressing, and we like to think that our partner will be there for us, and that we can trust them with all our hearts, and it hurts a lot to have that trust betrayed by the other partner acting in disregard to our feelings.