Abstract
Background More than 6 million people in the United States have Alzheimer’s disease and related dementias, receiving help from more than 11 million family or other informal caregivers. A range of traditional interventions have been developed to support family caregivers; however, most of them have not been implemented in practice and remain largely inaccessible. While recent studies have shown that family caregivers of people with dementia use Twitter to discuss their experiences, methods have not been developed to enable the use of Twitter for interventions.
Objective The objective of this study was to develop an annotated data set and benchmark classification models for automatically identifying a cohort of Twitter users who have a family member with dementia.
Methods Between May 4, 2021 and May 20, 2021, we collected 10,733 tweets, posted by 8846 users, that mention a dementia-related keyword, a linguistic marker that potentially indicates a diagnosis, and a select familial relationship. Three annotators annotated one random tweet per user to distinguish those that indicate having a family member with dementia from those that do not. We used the annotated tweets to train and evaluate deep neural network classifiers based on pretrained transformer models. To assess the scalability of our approach, we, then, deployed automatic classification on tweets that were continuously collected between May 4, 2021 and March 9, 2022.
Results Inter-annotator agreement was 0.82 (Fleiss’ kappa). A classifier based on a BERT model pretrained on tweets achieved the highest F1-score of 0.962 (precision = 0.946, recall = 0.979) for the class of tweets indicating that the user has a family member with dementia. The classifier detected 128,838 tweets that indicate having a family member with dementia, posted by 74,290 users between May 4, 2021 and March 9, 2022—that is, approximately 7500 users per month.
Conclusions Our annotated data set can be used to automatically identify Twitter users who have a family member with dementia, enabling the use of Twitter on a large scale to not only explore family caregivers’ experiences, but also directly target interventions at these users.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This work was supported by the National Library of Medicine [R01LM011176].
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
This study used publicly available Twitter data. The Institutional Review Board (IRB) of the University of Pennsylvania reviewed this study and deemed it exempt human subjects research under Category (4) of Paragraph (b) of the US Code of Federal Regulations Title 45 Section 46.101 for publicly available data sources.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Data Availability
All data produced in the present study are available upon reasonable request to the authors.
