Human sex differences are thought to arise from gonadal hormones and genes on the sex chromosomes. Here we studied how sex and the sex chromosomes can modulate the outcome of mutations across the genome. We used the results of genome-wide CRISPR-based screens on 306 female and 396 male cancer cell lines to detect differences in gene essentiality between the sexes. By exploiting the tendency of cancer cells to lose or gain sex chromosomes, we were able to dissect the contribution of the Y and X chromosomes to variable gene essentiality. Using this approach, we identified 178 differentially essential genes that depend on the biological sex or the sex chromosomes. Integration with sex bias in gene expression and the rate of somatic mutations in human tumors highlighted genes that escape from X-inactivation, cancer-testis antigens, and Y-linked paralogs as central to the functional genetic differences between males and females.