What defines things that are worn by both sexes as belonging to one sex or another? It can be something as subtle as the direction which a garment closes and fastens. Men's shirts close left side over right, as well as pants, jackets, and coats. Women's, though they may be in every other respect identical to menswear, close right over left. Its a subtle social distinction. We see it, understand it, and file it away in our heads when we perceive it. And its an example of how far reaching the patriarchy of our culture is; that a woman wearing a man's shirt is assumed to be wearing the clothing of her boyfriend, fiancee, or husband. A man wearing a woman's shirt would gather no such connections. In fact, the resulting assumptions would be deeply, and wrongly, negative.
Certain textiles belong to the world of men, others to women. Men as we perceive them, do not wear silk chiffon, and women as we think of them, don't wear rawhide. Sure, there are exceptions, mostly for women, interestingly. And the result is that we have a sexual/cultural divide, supported by the very things we wear on our backs, and how we choose to view them.
Even in cultures where men where skirts routinely, there are skirts, and skirts. A man in Malay would not wrap his lower half the same way his wife would. It simple isn't done. He would be thought effeminate. And we do the same things here. Women in jeans don't wear plain 501 Levi's. Theirs are subtly different in cut and fit, and we can see it when we see them.