Since its inception, quidditch has counted players on pitch based on gender, not sex. They are not the same thing, as there are more than two genders. Quidditch recognises gender as whatever gender the player identifies as, be it male, female, genderqueer, agender, genderfluid, etc. Transgender individuals can play as well, and like everyone else they count towards whatever gender they identify as. If you are still not sure what this means, the internet has some excellent resources - here's one to get you started.
The other part of quidditch accepting all genders, not just the two binary genders, is that people have the right to say how they identify and it is vital that we accept people at their word. There is no way to "prove" a person's gender; if somebody says they're agender, even if they present traditionally feminine or masculine, they are agender and may play as such.
The rules of quidditch state that there may not be more than 4 players on pitch at any given time who identify as the same gender, ensuring a balance of genders on the team and on the field.