Hm, those lists can be very subjective. It depends on your native tongue and its proximity to the one you're aiming to learn, as well as your own aptitude for languages and diligence in study. I took about a year and a half of Mandarin and found that I really enjoyed it, so I picked it up with some ease. The written part can be a bit daunting and many complain about the nuance of tone, but I found that context often helped with the tone problem, at least. If you're coming from, say, Japanese or Korean, the structure/written part might not be nearly as hard as it would for Westerners.
I think what makes English difficult for foreigners is that we have many, many inconsistencies, idioms, homophones (wound/wound, produce/produce, read/read/red, marry/merry/Mary, just to name a few), and we speak very heavily in slang. Our spelling system is pretty wonky at times, as well.
A few I typically see at the top of the "hardest languages" list is French, Arabic, Finnish, Hungarian, Chinese, and Russian. The French one always surprises me, but I think it's mostly due to its inherently sadistic grammatical structure :|