The Apolipoprotein-E (APOE) ε4 gene allele, the highest known genetic risk factor for Alzheimer's disease, has paradoxically been well preserved in the human population. One possible explanation offered by evolutionary biology for survival of deleterious genes is antagonistic pleiotropy. This theory proposes that such genetic variants might confer an advantage, even earlier in life when humans are also reproductively fit. The results of some small-cohort studies have raised the possibility of such a pleiotropic effect for the ε4 allele in short-term memory (STM) but the findings have been inconsistent. Here, we tested STM performance in a large cohort of individuals (N = 1277); nine hundred and fifty-nine of which included carrier and non-carriers of the APOE ε4 gene, those at highest risk of developing Alzheimer's disease. We first confirm that this task is sensitive to subtle deterioration in memory performance across ageing. Importantly, individuals carrying the APOE ε4 gene actually exhibited a significant memory advantage across all ages, specifically for brief retention periods but crucially not for longer durations. Together, these findings present the strongest evidence to date for a gene having an antagonistic pleiotropy effect on human cognitive function across a wide age range, and hence provide an explanation for the survival of the APOE ε4 allele in the gene pool.