I believe that that is where you are wrong.If genetically determined characteristics truly mattered, we couldn't have all those 'genetically-differing races' competing with each other. But we do, because the differences between 'races' are no more significant than the differences within them, and the same is true of gender.
The assumption is that all races are equal with some having certain small advantages over others. Who can possibly say that males and females develop anywhere close to equal physically? Anyone who feels that there is only a minor difference should have no issue with females competing against males across the board.
I am 100% for LGBTQ rights, but in the field of athletics/physicality transgender females have a verrry big advantage.