Abstract
Early detection of dementia is critical for intervention and care planning, but remains difficult. This study evaluated a computerized cognitive testing battery (BrainCheck) for its diagnostic accuracy and ability to distinguish the severity of cognitive impairment. 99 participants diagnosed with Dementia, Mild Cognitive Impairment (MCI), or Normal Cognition (NC) completed the BrainCheck battery. Statistical analyses compared participant’s performance on BrainCheck based on their diagnosis group BrainCheck battery performance showed significant differences between the NC, MCI, and Dementia groups, achieving ≥88% sensitivity/specificity for separating NC from Dementia, and ≥77% sensitivity/specificity in separating the MCI group from NC/Dementia groups. Three-group-classification found true positive rates ≥80% for the NC and Dementia groups and ≥60% for the MCI group. BrainCheck was able to distinguish between diagnoses of Dementia, MCI, and NC, providing a potentially reliable tool for early detection of cognitive impairment.
- 8.
- Computerized Cognitive Test
- BrainCheck
- Mild Cognitive Impairment
- Dementia
- Cognitive Decline
- Discriminability
- Health Technology
1. INTRODUCTION
In proportion with the increasingly aging population, the incidence of dementia is on the rise; projected to affect nearly 14 million people in the United States and upwards of 152 million people globally in the coming decades [1–3]. Current rates of undetected dementia are reported as high as 61.7% [4], and available treatments are limited to promoting quality of life rather than reversal or cure of the disease process. The ability to properly identify and treat dementia at this scope requires an active approach focused on early identification and intervention. Early detection of dementia provides access to timely interventions and knowledge to promote patient health and quality of life before symptoms become severe. Known treatment interventions help to address underlying conditions and minimize distressing symptoms, providing reason to implement these earlier on. Early and accurate diagnosis allows for proper preparation of patients, caregivers, and families which results in improved caregiver well-being and delayed institutionalization [5, 6]. Further, it helps to characterize early-stage dementia patients for clinical trials, exploring the latest therapeutics and validating biomarkers indicative of specific pathologies. Despite the benefits, early detection is a current challenge with current clinical protocols, leaving many patients undiagnosed until symptoms are more apparent in later stages of the illness.
Considered an early symptomatic clinical stage of dementia, mild cognitive impairment (MCI) signifies a level of cognitive impairment between normal cognition (NC) and dementia [7]. While not all MCI cases progress, the conversion rate of MCI to dementia has been observed to be approximately 5-10% [8]. This stresses the importance of MCI identification in the early detection and clinical intervention for dementia, included in recommendations from The National Institute on Aging and the Alzheimer’s Association [9]. Detection of MCI has been fairly successful using brief cognitive screening assessments. The widely used Montreal Cognitive Assessment (MoCA) has demonstrated 83% sensitivity and 88% specificity in distinguishing MCI from NC its ability to distinguish dementia from MCI has demonstrated 90% sensitivity and 63% specificity [10, 11]. The Saint Louis University Mental Status (SLUMS) exam and the Mini-Mental State Examination (MMSE) also show the ability to distinguish impaired individuals. For example, the diagnostic accuracy of the MMSE has been reported to be 62% sensitivity and 87% specificity for distinguishing MCI from NC and 80% sensitivity and 85% specificity for distinguishing dementia from MCI [12, 13]. Similar performance has been observed with the SLUMS exam, reporting 94% sensitivity and 87% specificity for diagnosing dementia and 82% sensitivity and 86% specificity for diagnosing cognitive impairment [14]. While traditional ‘paper and pen’ based versions of these assessments have typically been used in standard practice, digitized or computerized versions are now available; however, they have been slow in adoption in clinical practice. Although these screening tests do well in their ability to detect MCI, they often are utilized to prompt an indication for further comprehensive testing to better specify patterns of cognitive impairment seen in neurological disorders [15].
After a diagnosis of MCI, there is often a referral for neuropsychological testing (NPT); a comprehensive evaluation of cognitive domains (e.g., attention, working memory, language, visuospatial skills, executive functioning, memory), that are used to support clinical diagnoses and further delineate specific neurocognitive disorders. NPT can determine patterns of cognitive functioning that relate to normal aging, MCI, and dementia progression with specificities from 67 - 99% [16]. A major strength of NPT is the ability to characterize cognitive impairments, providing clues to underlying pathology, and thereby improving diagnostic accuracy to guide appropriate treatment. However, NPT comes with downsides including financial cost, long appointment times, and high levels of required training and expertise to conduct and interpret tests. To address these pitfalls and meet future demands, computerized and/or digitized NPT have also been developed to address issues of accessibility and efficiency [17, 18]. Prior studies have shown that some computerized cognitive tests demonstrate high accuracy in differentiating dementia patients from healthy participants, but do not have adequate psychometrics to distinguish MCI from dementia [19–22]. Ideally, computerized cognitive tests would aim to be more rapid and maximize accessibility to both patients and providers to address the expected uptick in dementia. This would also yield multiple benefits including maintaining testing standardization, alleviating time pressures of modern clinical practice, and providing a comprehensive assessment of cognitive function to strengthen a clinical diagnosis. Importantly, in the new era of practicing amidst the COVID-19 pandemic [23], increasing the accessibility of remote cognitive testing for vulnerable/high-risk patients is essential.
This study evaluated BrainCheck, a computerized cognitive test battery previously validated for its diagnostic accuracy for the detection of concussion [24] and dementia-related cognitive decline [25]. BrainCheck is portable, allows for self and remote administration, and can be reimbursed by insurance plans in the US. Furthering its validation for dementia-related cognitive decline, we sought to assess BrainCheck’s utility as a diagnostic aid to accurately distinguish the severity of cognitive impairment using existing clinical diagnoses of Normal Cognition (NC), Mild Cognitive Impairment (MCI), and Dementia. We compared BrainCheck scores to comprehensive clinical diagnosis (subjective cognitive impairment or NC, MCI, or Dementia). Our goal was to further demonstrate the utility of BrainCheck for cognitive assessment, specifically as a diagnostic aid in cases where NPT may be unavailable, or when a comprehensive evaluation is not indicated.
2. METHODS
2.1 Recruitment
This study was approved by the Institutional Review Board of The University of Washington for human subject participation. Participants were recruited using the University of Washington Alzheimer’s Disease Research Center’s (UW ADRC) research registry [26], a database of individuals willing to participate in dementia-related research. Those on the registry were contacted by phone or email provided within the registry. While there were no formal exclusionary criteria, those with listed addresses outside a 70-mile radius of Seattle, Washington were not contacted to participate. If the person was unable to physically use an iPad, too cognitively impaired to understand or follow instructions, or if the primary contact (e.g., spouse) indicated that the person was unable to participate, they were not recruited for the study. When study procedures were modified from in-person to remote administration due to the COVID-19 pandemic (approximately March 2020), participants outside of the initial geographical range were contacted. We required that these participants had access to either an iPad with iOS 10 or later, or a touchscreen computer with Wi-Fi connectivity. A small number of participants were recruited via snowball sampling by participants who had completed the study.
Most participants from the research registry were former or current patients of a memory disorders clinic in Seattle, Washington where a recent clinical diagnosis by a physician had been made. Using their primary cognitive diagnosis by the memory clinic, participants were divided into one of three groups: 1) Normal Cognition (NC), indicated by subjective cognitive complaint or no diagnosis of cognitive impairment, 2) Mild Cognitive Impairment (MCI), representing both amnestic (aMCI) and non-amnestic (naMCI) subtypes, or 3) Dementia, which included dementia due to Alzheimer’s Disease (AD), Frontotemporal dementia, Vascular dementia, Lewy body dementia, Mixed dementia, or Atypical AD, delineated by the clinical diagnosis. Participants who did not have a formal diagnosis and denied cognitive symptoms or a history of cognitive impairment were placed into the NC group.
2.2 Participant Characteristics and Demographics
241 individuals were contacted to participate and 99 participants completed the study. Participant ages ranged from 47 to 90 years old (mean age=70.7; SD=8.9) and the sex distribution was balanced. The diagnosis groups consisted of 35 in the NC group, 22 in the MCI group, and 42 in the Dementia group. 75 participants completed BrainCheck via on-site administration and 24 participants completed BrainCheck via remote administration.
2.3 Study Design and Procedures
2.3.1 On-site Administration
Data collection for on-site administration was collected from October 2019 to February 2020. A session was held either in the participant’s home or in a public setting that was well-lit, quiet, and distraction-free. Consent forms were reviewed and signed by the participant, or legally authorized representative, and examiner, with both parties obtaining a copy. The study was designed for participants to complete one session with a moderator using a provided iPad (Model MR7G3LL/A) connected to Wi-Fi to complete the BrainCheck battery. Prior to testing, participants were briefed on BrainCheck, and moderator guidance was limited to questions and assistance requested by the participant during the practice portions. Participants received a gift card for participation at the conclusion of the study session.
2.3.2 Remote Administration
Due to the COVID-19 pandemic, study procedures were modified to accommodate stay-at-home orders in Washington state. Data collection resumed from April 2020 to May 2020, with modified procedures using remote administration. These participants provided written and verbal consent and were administered the BrainCheck battery remotely over a video call with the moderator. Participants used their personal iPads or touchscreen computer browser to complete the BrainCheck battery. The same method for on-site administration (described above) was used for remote administration.
2.4 Measurements
A short description of each of the 5 assessments that make-up the BrainCheck battery (V4.0.0) is listed in Table 1. More detailed descriptions may be found in a previous validation study [24]. After completion of the BrainCheck battery, the score of each assessment is calculated using assessment specific measurements by the BrainCheck software (shown in Table 1). The BrainCheck Overall Score is a single, cumulative score for the BrainCheck battery that represents general cognitive functioning. This score is calculated by taking the weighted average of all assessment scores. The normalized assessment scores and overall scores are corrected for participant age and device used (iPad versus computer) using the mean and standard deviation of the corresponding score from a normative database previously collected by BrainCheck. The score generated follows a standard normal distribution, where a lower score indicates lower assessment performance and cognitive functioning.
2.5 Statistical Analysis
To evaluate BrainCheck performance among participants in different diagnosis groups, Kruskal–Wallis tests with post-hoc Dunn’s tests were conducted on individual BrainCheck assessment scores and BrainCheck Overall Scores. P-values were corrected using the Benjamini-Hochberg method for multiple comparisons. To assess the accuracy of the BrainCheck Overall Score in binary classification of participants in the different diagnosis groups (NC vs. Dementia, NC vs. MCI and MCI vs. Dementia), receiver operating characteristic (ROC) curves with area under the curve (AUC) calculations were generated to determine diagnostic sensitivity and specificity. In assessing BrainCheck for three group classification, we used an R package developed by Luo et al. [27], to define optimal cutoffs for the BrainCheck Overall Score and find the maximum diagnostic accuracy. The analysis was performed using Python version 3.8.5 and R version 3.6.2 programming languages.
3. RESULTS
3.1 Completion of Assessments
We found most participants in the NC group were able to complete the assessments whereas the Dementia group had a higher ‘timeout’ rate, with MCI falling in between the two (Figure 1). The timeout function occurs when a participant can not complete a trial of the assessment in 30 seconds; it is embedded in the Stroop and Trails A/B assessments. Timeouts were mainly due to response delays; where participants were attempting the test but could not answer quickly enough. Overall, the Dementia group took significantly more time to complete the BrainCheck battery (m=35.4 min, sd=21.4 min) compared to the NC group (m=18.2 mins, sd=3.7 mins), on average.
3.2 BrainCheck Performance
BrainCheck assessments were compared across the three groups. Individual scores, such as the BrainCheck Overall Score, were normalized for age and device. Overall, participants with greater cognitive impairment showed lower BrainCheck assessment scores. All individual assessments showed significant differences in performance between the NC and Dementia groups. Three of the seven assessments (Digit Symbol, Flanker, and Trails A) showed significant differences in performance between all three groups on post-hoc Dunn’s test (Table 2). The Immediate and Delayed Recognition tests showed significant differences (p<0.05) in performance between the MCI and NC group, but there was not a significant difference between the MCI and Dementia group (Table 2). The Stroop and Trails B tests showed significant differences between the NC and Dementia groups but could not significantly differentiate the MCI group from the other groups (Table 2).
The BrainCheck Overall Score is a composite of all individual assessments within the BrainCheck battery, representing overall performance (See details in Method 2.4). Using an existing normative population database, partly compiled from controls in previous studies [24,25], the Overall Score was adjusted for age and the device used to generate the normalized BrainCheck Overall Scores. The normalized BrainCheck Overall Scores differed significantly among these three groups (p<0.05). The NC group scored significantly higher than the MCI and Dementia groups. The Dementia group demonstrated significantly lower scores than the other two groups (Figure 2).
3.3 BrainCheck Diagnostic Accuracy
Using ROC analysis, Braincheck Overall Scores achieved a sensitivity of 94% and a specificity of 88% for classifying NC and Dementia participants (AUC=0.954); a sensitivity of 83% and a specificity of 86% for classifying between NC and MCI participants (AUC=0.842); and a sensitivity of 77% and a specificity of 83% for classifying between MCI and Dementia participants (AUC=0.785) (Figure 3).
Using methods described by Luo and colleagues for three group classification [27], the optimal lower and upper cutoffs of the normalized BrainCheck Overall Score in maximizing diagnostic accuracy were -3.64 and -0.06, respectively. This achieved true positive rates of 83% for the NC group, 64% for the MCI group, and 83% for the Dementia group (Figure 4).
4. DISCUSSION
Consistent with prior findings in concussion [24] and dementia/cognitive decline [25] samples, this study demonstrated that BrainCheck is consistent in its capability to detect cognitive impairment. Our study demonstrated that BrainCheck can detect MCI, and reliably differentiate between the severity of cognitive impairments in the diagnosis groups (NC, MCI, and Dementia). As expected, participants with more severe cognitive impairment performed worse across the individual assessments and on BrainCheck Overall Scores. The BrainCheck Overall Scores separated participants of different diagnostic groups successfully with high sensitivity and specificity.
BrainCheck Overall Scores were more robust in distinguishing these groups where participants in the Dementia group had significantly lower scores than ones in the NC group. The BrainCheck battery was able to distinguish between NC and Dementia participants, with 94% sensitivity and 88% specificity. These findings show that the BrainCheck Overall Score demonstrates better accuracy for differentiating NC from Dementia, compared to the MMSE, SLUMS and MoCA screening measures[14, 15, 29]. People with MCI usually experience fewer cognitive deficits and preserved functioning in activities of daily living compared to those with Dementia [30], and our findings of sensitivity and specificity with separating MCI from other groups were slightly lower than the NC versus Dementia differentiations (Figure 3). Nonetheless, the BrainCheck Overall Score showed sensitivities and specificities >80% in distinguishing MCI from NC and Dementia groups, which is comparable to the MoCA, SLUMS, and MMSE [11-15]. Furthermore, a review of validated computerized cognitive tests indicated AUC’s ranging from 0.803-0.970 for detecting MCI, and AUC’s of 0.98 and 0.99 in detecting dementia due to AD [28], which were comparable with the results found in this study.
Although not all individual assessments in the BrainCheck battery could differentiate NC, MCI, and Dementia, we observed a general trend for each assessment showing that Dementia participants had the lowest scores whereas the NC participants had the highest scores. The individual assessments that did show significant differences in the scores between NC and MCI groups and between Dementia and MCI groups included the Digit Symbol, Flanker, and Trails A. Notably, in this study the BrainCheck Digit Symbol test showed significant differences in performance between all three diagnosis groups, whereas a previous study found that the BrainCheck Digit Symbol test did not show significant differences between cognitively healthy and cognitively impaired groups (n=18, p=0.29) [25]. This is likely due to the present study having a larger sample size. The individual assessments with no significant differences between the MCI group and the NC/Dementia groups were the BrainCheck Stroop and Trails B tests (Table 2). Both of these tests include timeout mechanisms if participants are unable to complete the test, and time out rates were higher in the more cognitively impaired groups (Figure 1).
In comparison to comprehensive NPT, which can typically last a few hours and sometimes require multiple visits [30], BrainCheck demonstrated shorter test duration, with average completion times of 18.2 mins (sd=3.7 mins) for NC participants and 35.4 min (sd=21.4 min) for dementia participants (Figure 1). Shorter test durations observed in individuals with no/less cognitive impairment suggest that computerized cognitive tests could be useful for rapid early detection in this population, prompting further evaluation, whereas those with dementia have likely already undergone a comprehensive evaluation. The wide variance in completion time for the Dementia group may have uncovered the difficulty participants with more severe cognitive impairment may have faced in completing the BrainCheck battery, in comparison to the lower variance observed in the NC group.
A limitation of this study was that all participants were not professionally evaluated and diagnosed by a physician at the time of BrainCheck testing. Thus, participants were placed into the diagnosis groups based on their most recent clinical diagnosis available in their electronic health record, or for the few NC participants without medical evaluations, based on their report of no cognitive symptoms or diagnosis or cognitive impairment. The time period from the most recent clinical diagnosis to the date of BrainCheck testing varied among the diagnosis groups; where the Dementia group had the fewest days from last clinical evaluation (m=111 days; sd=104), followed by the MCI group (m=342 days, sd=345) and the NC group (m=753 days; sd=570). These large time intervals in a degenerative population leave room for cognition to worsen over time, potentially blurring the lines in severity of cognitive impairment, where participants may have progressed to MCI from NC, and to Dementia from MCI during that time period. This would make the distinguishing of NC from cognitive impairment more difficult, yet diagnostic accuracy among the groups still remains high. Furthermore, the mean number of days since the last clinical evaluation for NC participants was as high as 753 days. This could suggest that the NC participants did not feel an inclination to seek out further cognitive evaluation during the extended time period, and not have experienced noticeable cognitive decline. Future validity studies should ensure that a physician evaluation and diagnosis occur closer to the time of BrainCheck testing to address these limitations.
When administration type was considered, scores showed significant differences among all three groups for onsite administration but not for remote administration. While scores showed similar trends, the sample size of NC (n=6) and MCI (n=6) within the remote groups were fairly small. While remote administration was not designed within the original study, stay at home orders due to COVID-19 required modifications, and efforts were made to provide preliminary data for remote use and self-administration abilities. With preliminary outcomes indicating feasibility for remote administration, a more robust study and increased sample size will be needed to fully validate BrainCheck’s cognitive assessment via it’s remote and self-administered feature.
5. Future Directions
The use of computerized cognitive tests is an opportunity to increase test accessibility for an aging population and an increasing presence of cognitive impairment. Remote test options with quick and accurate results could save on time and costs. Technological advances have allowed for these tools to obtain greater sensitivity and specificity as well [28]. Our findings in this study demonstrate that BrainCheck can distinguish between three different levels of cognitive impairment, NC, MCI, and Dementia. This study paves the way for a comprehensive longitudinal study, exploring BrainCheck in early detection of dementia and monitoring of cognitive symptoms over time, including comparison to ‘gold standard’ neuropsychological assessments. BrainCheck is automated, self-administrable, and quick to administer. These qualities are critical to increasing accessibility, especially in early detection, for a population facing an expected increase of dementia in upcoming years. Increasing access may lead to the growth in the normative data needed to better characterize dementia subtypes and provide earlier detection and diagnostic utility.
Data Availability
Data may be made available by contacting the corresponding author with a data use agreement.
https://depts.washington.edu/mbwc/adrc/page/research-resources
7. Acknowledgements/Conflicts/Funding Sources
The following authors declare the following competing interests: BH, KS, RHG, SY, HP, BK report personal fees from BrainCheck, outside the submitted work; BH, RHG reports receiving stock options from BrainCheck.
Acknowledgments: Funding was provided by BrainCheck, Inc. Registry was provided by the University of Washington Alzheimer’s Disease and Research Center along the Memory and Brain Wellness Center
Data Availability - Data may be made available by contacting the corresponding author with a data use agreement.