Gender itself is ambiguous. However it's social expectations of gender roles, that enforces that notion that by biological sex that all men are masculine, all women are feminine. If you try to deviate from expectation, you'll become alienated by those who strongly believe in those expectations. It's also hard to alter expectations to add diversity, because people feel secure about themselves in certainty, and having absolute conceptions of what is right or wrong, what is good or evil, what is safe or dangerous helps fulfil that certainty. When that certainty is threatened, they'll automatically perceive it as a threat or be so scared of it, they will prefer to destroy or hide it, rather than try to understand it.
You'll find that problem applies to nearly every issue of prejudice that exist.
was it something I said?