Homophobia does not mean a phobia of homosexuals. If you want to get into semantics, it means "Fear of the Same" or (if one is combining Latin and Greek roots, which is done in some poorly created neologisms) "Fear of Humans" (I'll assume the former is correct). Thus a homophobe would be a man who is afraid of men, a human who is afraid of humans, a Latvian who is afraid of Latvians, a blonde who is afraid of blondes, a 1.7m tall person who is afraid of other people who are 1.7m tall, or whatever quality of sameness you choose. However, to believe that is the primary meaning of the word is absurdity. In almost all useages of the word, it means one who is prejudiced against homosexuals. To play these games about a homophobe needing to be one who has a phobia of homosexuals would be the equivalent of someone saying that since Arabs are technically a semitic peoples (they speak a semitic language), that one who hates Arabs is an Anti-Semite. Or that an Anti-Semite is one who hates the Phoenicians (who were also a Semitic people). Linguistically, you are right. But every use of the term "Anti-Semite" and "Anti-Semitism" refers to a hatred of the Jewish people, not against Semitic people at large.
Also, I feel that harboring anti-homosexual feelings is homophobia, even if it does not emerge as outward aggression. If I were to say I think African-American culture is bad or wrong, I would be a racist. If I were to say that Judaism is wrong, I would be an anti-semite. If I were to say all poor people are scum who just need to stop being lazy, I would be classist. If I were to say that women are inferior to men, I would be a sexist. I do not need to join a lynch mob, the KKK, or protest integrated schools to be a racist, I don't need to join a Nazi party, organize a pogrom, or publish a book about how the Jews use the blood of Christian infants in their rituals to be an anti-semite, I don't need to want to harm the poor economically or create a legal class system to be a classist, and I don't need to be a "men's rights" advocate or physical oppressor of women to be a sexist. The simple expression of these views, even if I could tolerate the presence of such people, would make me those things. I do not see how homosexuality and homophobia is any different. And the reason why is that the expressions of these views do two things: 1) they encourage those more radical into believing that their extremist views are secretly harbored by the majority; and 2) they contribute to institutionalized racism, classism, homophobia, anti-semitism, and sexism.