Short sleep is held to cause poorer brain health, but is short sleep associated with higher rates of brain structural decline? Analysing 8153 longitudinal MRIs from 3893 healthy adults, we find no evidence for an association between sleep duration and brain atrophy. In contrast, cross-sectional analyses (51,295 observations) showed inverse U-shaped relationships, where 6.5 (CI: 5.7, 7.3) hours were associated with the thickest cortex and largest volumes relative to intracranial volume. This fits converging evidence from research on mortality, health and cognition which points to ~7 hours being associated with good health. Genome-wide association analyses suggested that genes associated with longer sleep for below-average sleepers were linked to shorter sleep for above-average sleepers. Mendelian randomisation did not yield evidence for causal impacts of sleep on brain structure. The combined results challenge the notion that habitual short sleep causes brain atrophy, suggesting that normal brains promote adequate sleep duration – which is shorter than current recommendations.