Memory T cells induced by previous pathogens can shape susceptibility to, and the clinical severity of, subsequent infections1. Little is known about the presence in humans of pre-existing memory T cells that have the potential to recognize severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Here we studied T cell responses against the structural (nucleocapsid (N) protein) and non-structural (NSP7 and NSP13 of ORF1) regions of SARS-CoV-2 in individuals convalescing from coronavirus disease 2019 (COVID-19) (n = 36). In all of these individuals, we found CD4 and CD8 T cells that recognized multiple regions of the N protein. Next, we showed that patients (n = 23) who recovered from SARS (the disease associated with SARS-CoV infection) possess long-lasting memory T cells that are reactive to the N protein of SARS-CoV 17 years after the outbreak of SARS in 2003; these T cells displayed robust cross-reactivity to the N protein of SARS-CoV-2. We also detected SARS-CoV-2-specific T cells in individuals with no history of SARS, COVID-19 or contact with individuals who had SARS and/or COVID-19 (n = 37). SARS-CoV-2-specific T cells in uninfected donors exhibited a different pattern of immunodominance, and frequently targeted NSP7 and NSP13 as well as the N protein. Epitope characterization of NSP7-specific T cells showed the recognition of protein fragments that are conserved among animal betacoronaviruses but have low homology to 'common cold' human-associated coronaviruses. Thus, infection with betacoronaviruses induces multi-specific and long-lasting T cell immunity against the structural N protein. Understanding how pre-existing N- and ORF1-specific T cells that are present in the general population affect the susceptibility to and pathogenesis of SARS-CoV-2 infection is important for the management of the current COVID-19 pandemic.