While animal data suggest a protective effect of caffeine on cognition, studies in humans remain inconsistent. We examined associations of coffee and caffeine intake in midlife with risk of dementia, its neuropathologic correlates, and cognitive impairment among 3494 men in the Honolulu-Asia Aging Study (mean age 52 at cohort entry, 1965–1968) examined for dementia in 1991–1993, including 418 decedents (1992–2004) who underwent brain autopsy. Caffeine intake was determined according to self-reported coffee, tea, and cola consumption at baseline. Logistic regression was used to calculate odds ratios (OR) and 95% confidence intervals (CI) for overall dementia, Alzheimer's disease (AD), vascular dementia (VaD), cognitive impairment (Cognitive Abilities Screening Instrument score <74), and neuropathologic lesions at death (Alzheimer lesions, microvascular ischemic lesions, cortical Lewy bodies, hippocampal sclerosis, generalized atrophy), according to coffee and caffeine intake. Dementia was diagnosed in 226 men (including 118 AD, 80 VaD), and cognitive impairment in 347. There were no significant associations between coffee or caffeine intake and risk of cognitive impairment, overall dementia, AD, VaD, or moderate/high levels of the individual neuropathologic lesion types. However, men in the highest quartile of caffeine intake (>277.5 mg/d) were less likely than men in the lowest quartile (≤115.5 mg) to have any of the lesion types (adjusted-OR, 0.45; 95% CI, 0.23–0.89; p, trend = 0.04). Coffee and caffeine intake in midlife were not associated with cognitive impairment, dementia, or individual neuropathologic lesions, although higher caffeine intake was associated with a lower odds of having any of the lesion types at autopsy.