Why its considered "fatphobic" not wanting to date a fat person?

Isn't it like saying a homosexual man is misogynistic for not dating women just for being women? I mean, we don't choose to what we are attracted to?