How about the elf-society is fragmented enough that there are still many elves that have never seen a human. The humans have been coming in waves, similar to the colonization of America. The warring desert tribes are a combination of colonizing humans from the mainland to the north (much like the Middle East) and a Europe-like country to the far west. These two groups are in direct conflict for land and resources within the extensive northern desert, but still manage to trade peacefully within cities (each built around an oasis or rock formation). The European-style humans continue to attempt to colonize the strange new island land by sending ships (the pirates). Most of the European-like desert colonies are similar to the Wild West-type villages and attempt to farm in the desert as well as mine.
The human king tries to organize the European-settlers and lives in the European-settler capital trading city and makes all the European settlers pay taxes to him. The capital city is a very diverse city where beings from all over the island come to trade.
A line of hostile elves and other dangers have successfully held back the colonization of the south (wild forest and jungle country), despite the fact that it would be easier to farm there and the humans are sure there are precious resources down there as well.
How about the elves aren't centrally organized. They live in small family groups. Like Native Americans almost, they have different nations and tribes that they hold allegiance to, but there is no central nations that controls the other nations.
How about there are tons of other magical beings and creatures living in the southern forests and jungles.
And how about there are mountains between the desert and the jungles where dragons and other dangerous things live that have also made it hard for the humans to move south. It would be easier for them to come down from the west (where the European-like humans originally settle) but that's where the elf resistance is strongest. How about the land near the west where the humans come in is a lot like Lousiana, a billabong/marshy area and the elves are a kind of southern-acting marsh-living tough society.