It has very little to do with logic.
One of the places I've seen for this assumption reinforced comes from a typical interaction over this subject. When it's face to face, it can go something like this:
Man (marching up to woman who advocates for victims of domestic violence or rape): Men are victims just as often as women. Why won't you admit that half of all women who make rape allegations are bold-faced liars?
Woman: While some men are victims, the statistics and research show that women are the victims in nearly 3 out of 4 murders committed by intimate partners. As for the number of false--
Man (cutting her off mid-sentence): You're sexist. Admit it.
The conversation usually gets worse from there or it ends abruptly.
On anti-violence blogs, men who make similar assumptions often come across as trolls. These men rarely get a friendly response to their claims that speaking out against sexual violence equals sexism. Quite a few of these men assume they are getting an unwelcoming response simply because they are men rather than because they said or did something offensive.
In most cases the unwelcoming response occurs because the man's words make him sound like just another in a long line of bullies attempting to silence and blame women while minimizing the harm violent men inflict on women and children.
The same pattern can develop in non-gender specific ways. Religious people who blame rape victims for luring their rapists may believe the backlash is caused by the blamers religion (Christianity, Islam, etc.) rather than bias so deeply entrenched that it might as well be a religious belief.
Whenever entrenched beliefs collide with attempts to permanently change the status quo, there will be conflict. The only thing we can control individually is how we deal with that conflict.
Technorati tags: rape crime sexism feminism