Volumetric variations of the human brain are heritable and are associated with many brain-related complex traits. Here we performed genome-wide association studies (GWAS) of 101 brain volumetric phenotypes using the UK Biobank sample including 19,629 participants. GWAS identified 365 independent genetic variants exceeding a significance threshold of 4.9 × 10-10, adjusted for testing multiple phenotypes. A gene-based association study found 157 associated genes (124 new), and functional gene mapping analysis linked 146 additional genes. Many of the discovered genetic variants and genes have previously been implicated in cognitive and mental health traits. Through genome-wide polygenic-risk-score prediction, more than 6% of the phenotypic variance (P = 3.13 × 10-24) in four other independent studies could be explained by the UK Biobank GWAS results. In conclusion, our study identifies many new genetic associations at the variant, locus and gene levels and advances our understanding of the pleiotropy and genetic co-architecture between brain volumes and other traits.