October 25, 2023

Introduction & Summary

The review “Interventions to Prevent Age-Related Cognitive Decline, Mild Cognitive Impairment, and Clinical Alzheimer’s-Type Dementia” is a timely effort that clearly required an impressive amount of work. As noted in the review, the development and validation of interventions to reduce cognitive decline and lower the risk of the onset of dementia are key national – and in many cases personal – concerns. I submit this commentary through the limited public review process.

Before turning to my own area of expertise (brain training), I should note that this report finds insufficient or low evidence of efficacy for such widely recommended interventions as physical exercise, diet, mental stimulation, sleep, and socialization – each of which is currently recommended by the NIA and the AARP. This seems on its face, very odd.

I focus my commentary mostly on the sections of the review related to cognitive training (rated to have moderate evidence of efficacy), which is my own field of expertise. As a disclosure, I work at (and hold stock in) Posit Science, which is a developer of plasticity-based brain training programs. My knowledge of this field derives from acting as a principal investigator in multiple large-scale multi-site randomized controlled trials (RCTs) in brain training, acting as an advisor to independent academic studies of brain training, authoring a number of published papers regarding brain training, and active maintenance of a database of publications from RCTs in computerized cognitive training.

I note three areas of significant concern that limit the validity and usefulness of this draft review.

  • Process Problems: First, the review suffers from very significant process problems in that 1) the reviewers actively chose to decline input from experts in the field of cognitive training during the review planning and initial data analysis process, and 2) there was little or no notification that the draft review was complete and that a very time-limited comment period had begun, such that domain experts had very limited opportunity to contribute significant comments.
  • Methodological Problems: Second, the review of cognitive training suffers from very significant methodological problems that render its overall conclusions incorrect. These problems include 1) missing publications (regardless of the inclusion criteria) that suggest problems in the search technique, 2) missing publications (meeting the inclusion criteria) that provide crucial results, 3) incorrect reporting on ACTIVE study booster methodology, 4) incorrect reporting on ACTIVE study attrition rates and their implications, and 5) the decision to exclude all publications with less than a 6-month duration.
  • Policy Problems: Finally, the review creates very significant policy problems, including 1) the implicit suggestion that 20 years of NIA funded research has led to few useful conclusions for cognitive health, 2) negative repercussions for the dissemination of evidence-based brain training, as this review undermines any business rationale for investments in science, and 3) that consumer-facing guidelines arising from this report seem likely to conclude that seniors who want to proactively take action to maintain their cognitive wellness have no options to do so.
    Each of these problems – process, methodological, policy – are addressable, even at this late stage, by 1) adding missing papers to the review, 2) consulting with domain experts, particularly with regard to key studies, 3) revising the study inclusion criteria to include all RCTs of cognitive training regardless of duration to create an appropriately comprehensive review, and most importantly, 4) extending the scope of the analysis to include patient-centered guidance – particularly to individuals who want to take proactive actions, today, based on the current state of the evidence.

Process Concerns

The design and conduct of the review to date specifically excluded input from academic experts in the field of cognitive training, and all industry experts. For example, I in particular was made aware of this review only incidentally during a discussion with NIA staff on another topic in January 2016. At that time I offered to contribute a list of publications to ensure that the reviewed publication list was complete. I was told that no such opportunity was permitted.

My colleague, Dr. Michael Merzenich, recent Kavli Laureate for his work in brain plasticity (a foundational science for modern plasticity-based brain training), developer of multiple brain programs, professor emeritus at UCSF, and member of both the National Academy of Science and the National Academy of Medicine, was not made aware of the review nor was he asked to provide his input.

I have spoken with several key members of the academic cognitive training community, and non were aware that this review was ongoing or had been contacted for input.

I have reviewed the list of planning committee members, and while each is an expert in their specific field, it would not be appropriate to characterize any of them as domain experts in the field of cognitive training. The review mentions key informants, and technical and content experts; but does not provide a list, making it difficult to assess their expertise in the relevant domain.

Furthermore, the review’s discussion of the issues around “diffusion” and “spillover” (which an expert in the field would call “transfer” and “generalization”) suggests that the reviewers have a point of view highly influenced by an older literature from psychology that takes it as a given that learning cannot transfer from a trained task to an untrained task (as originally shown in the lack of transfer between the memorization of digits and the memorization of letters). Modern brain training programs take their foundational science instead from the literature of brain plasticity, where it is not surprising that improving the speed and accuracy of information processing and lowering signal/noise ratios in early cortical systems can improve the ability of frontal systems depending on such input to direct attention or maintain information in working memory. From this viewpoint, transfer between notionally distinct cognitive domains that nonetheless share key neural substrates is less surprising. The lack of this perspective may have limited the group’s discussions.

Cognitive training is a complex field, and achieving a good result from a review depends on substantial, sustained input from experts.

The commentary period suffers similar challenges. The review opened for commentary on October 4th. I am not aware of any academic experts in the field who knew that the comment period was open, with the exception of one researcher who was invited on short notice to provide brief commentary at the in-person meeting October 25th. I myself was only made aware of the fact through a chance in-person conversation with a senior NIA staff member on October 19th. This leaves very little time for the in-depth commentary from experts that is required to address the issues in the review, and limits my contributions in this commentary.

This process approach has consequences – significant methodological concerns (discussed below), and a lack of buy-in for the conclusions from experts. If experts have strong doubts about the process, this also will undermine the confidence of opinion leaders, policy makers, and the public in the review.

Methodological Concerns

I have five specific significant methodological concerns with the review.

Missing Publications That Meet the Review’s Search Criteria
There are a surprising number of peer-reviewed publications in the field of computerized cognitive training that are well known to experts and can be found in PubMed that are simply missing from the review (both the main body and the appendix). Their absence raises significant concerns about the literature search process.

Each of these missing publications derives from a randomized controlled trial with cognitive function measures, and virtually all document transfer of improvement to generalized measures of cognitive function. Here are the first twenty missing publications that I identified by a quick check through a publicly available database of RCTs in computerized cognitive training1–20. There are quite likely more, but time for this response is limited.

Missing Publications That Meet the Review’s Inclusion Criteria
Seven key publications from ACTIVE and IHAMS relating to key a priori outcome measures documenting generalization of cognitive gains and real-world improvements are missing, bucketed into the following four bullets:

  • In ACTIVE, health-related quality of life (HRQoL) was assessed using the well-established SF-36 measure, with the viewpoint that HRQoL is an important measure of what domain experts call the generalization or transfer (which the authors of this review seem to describe as “diffusion” or “spillover”) of training to real-world benefits. Published analyses documented that speed training selectively protected against significant decline in health-related quality of life at the two-year21 and five-year22 follow-up points. Further analysis of the HRQoL data documented a significant reduction in predicted total medical expenditures using a predictive model developed by AHRQ based on its correlation of SF-36 data to Medicare expenditures23.
  • In ACTIVE, depressive symptoms were assessed using the well-established CES-D 12 measure, with the viewpoint that depressive symptoms are a key measure of quality of life as well as themselves related to the risk of future functional decline; and furthermore an important measure of the generalization of cognitive training to real-world benefits. Published analyses documented that speed training selectively protected against significant declines in depressive symptom scores24, and in a related way reduced the risk of the onset of clinically significant levels of depressive symptoms25.
  • In ACTIVE, motor vehicle crashes were assessed with objective state-level department of motor vehicle records, with the viewpoint that driving is a key real-world measure of cognitive performance. Published analysis documented that speed training (overall) and reasoning training (adjusted for baseline depression scores) each significantly reduced the incidence of at-fault crashes26.
  • In IHAMS, IADLs and depressive symptoms were assessed to confirm and extend prior results from ACTIVE. Published analysis documented that speed training reduced IADL decline and lowered the risk of depressive symptom worsening27.

These missing publications raise concerns about the review’s publication search and/or review process. The review is missing 7 key publications from an overall total of 12 publications from ACTIVE and IHAMS covering a priori outcome measures, for an effective search rate of 42% or a missing publication rate of 58%.

As a result, the review’s summary of ACTIVE is factually incorrect (in addition, please also see commentary regarding booster training and attrition below). The argument that “diffusion to other domains was rare” in ACTIVE cannot be sustained with a full review of the results from ACTIVE.

For reference, the summary of the ACTIVE study investigators regarding their findings is “Results to date clearly support the effectiveness of three cognitive interventions (memory, reasoning, and speed of processing) in maintaining cognitive health. Training produces immediate positive effects that are modality specific. These cognitive improvements do dissipate over time but are still detectable at least 5 years after training. Additional booster training does enhance training gain. ACTIVE is the first study of its kind to demonstrate modest but detectable far transfer to health-related quality of life and instrumental activities of daily living.”28

This oversight in the literature review would have been easily addressed by better incorporation of expert advice early in the process, rather than at this late stage (see process concerns, above).

Treatment of Attrition in ACTIVE
The review has a puzzling viewpoint towards issues arising from longitudinal studies and attrition. On the one hand, the review states that “Focusing on longitudinal investigations with follow-up periods of 10 years or more would greatly benefit the field and provide more insight about prevention.” On the other hand, the review dismisses the ACTIVE study 5 and 10 year findings, stating that “The strength of evidence is low for longer follow up periods largely because of high attrition.” Longitudinal studies in aging will suffer from attrition – that is an inevitable consequence of the study population. Given the importance of longitudinal studies and the inevitability of attrition, the focus in the review should be on 1) the methodological and statistical approaches used to control for attrition, and 2) any observed issues in the data that suggest that attrition is causing analytical problems.

The review states that “much of the sample loss [in ACTIVE] was unexplained.” This is factually incorrect. As shown in the CONSORT table in the 10-year follow-up publication29 only 5.1% of participants are lost due to “family refuses access” or “lost to follow-up” – quite an accomplishment for a 10-year study. Twenty-three percent are lost due to death, which is outside the control of the study, another 21.3% are due to the subject’s decision to withdraw, which would be unethical for investigators to interfere with, and 9.8% are due to the site’s decision to drop the participant, which are typically due to participant behavioral issues and compliance with study requirements.

The rate of attrition in ACTIVE – given that it followed community-dwelling elderly for 10 years from average age 74 to 84 – was neither unexpected nor surprisingly high. In longitudinal studies of community-dwelling aging populations, it can be expected that significant numbers of participants will be lost due to mortality, morbidity, and other significant life changes. The review suggests no specific methods by which attrition in future studies could be improved, and it is likely that the total opportunity for such improvement is minor given the above attrition rate breakdown.

Missing data resulting from attrition is inevitable in any study, and ACTIVE handled that missing data with statistically appropriate methods. For example, ACTIVE documented that a number of well-known factors contributed to attrition, including multiple imputation and the use of variables contributing to overall attrition as covariates in analysis.

In addition, in ACTIVE there was no group-specific attrition, indicating that the attrition was not likely to affect group differential outcomes in this randomized controlled trial.

As a result, the low evidence rating given to ACTIVE 5- and 10-year results is inappropriate and not supported by the data.

Treatment of Booster Training in ACTIVE
The review misstates how booster training was assigned in ACTIVE, and as a result, ignores important conclusions from the booster training analysis. The review states that “the booster effect was also biased, because 80 percent of those receiving boosters had a compliance rate on the initial training of 80 percent or better.” This is not correct. ACTIVE randomized participants into booster or non-booster groups, and then offered booster training to participants randomized to booster training who had completed 8 or more (of 10) initial training sessions into booster. This means that 100% of those receiving boosters had a compliance rate of 80% or better in the initial period. It should be noted that most of assigned to no booster also had a compliance rate of 80% or better in the initial period, because this group was comprised of those randomized to non-booster who had a compliance rate of 80% or better and the 8.6% of participants (based on a calculation from the CONSORT table29) who had a compliance rate of lower than 80% in the initial period. This population was considered in the non-booster condition in the main analysis.

Important observations from the booster group analysis include that speed booster group showed significantly better IADL function at the 1-year point compared to the non-booster group, and better everyday speed (directly obsvered functional performance) at the 1-year and 5-year points.

Furthermore, extensive additional analysis was performed, including analyzing the randomized-to-booster against the randomized-to-non-booster groups on an intent-to-treat basis, and analyzing the data on a treatment-received basis30–32. Particularly for speed training, these analyses documented larger cognitive effects with booster training and larger transfers to real-world measures with boosters.

The lack of discussion of this data in the review contributes to the inappropriate conclusion that ACTIVE did not show transfer effects.

Six-Month Requirement
A key aspect of any systematic review is the inclusion/exclusion criteria for studies. In the current review, studies are required to have a “minimum follow-up of 6 months for intermediate outcomes.” I am not able to locate anywhere in the review where this requirement is justified. One possibility is that the criterion was chosen because the review felt that interventions must have benefit over 6 months to have the opportunity to significantly affect cognitive decline. A second possibility is that the criterion was chosen as a proxy for the evaluation of the endurance of training effects after the completion of training. However, both of these goals are ill-served by this requirement. For an intervention to have effects over time, it must first have effects at all, and a review would benefit from including all studies evaluating such effects, regardless of their duration.

As a result of this decision, the review identifies exactly 6 trials to review in normal aging (discussed in 9 publications), only four trials of which involve computer-based cognitive training. In the database that I maintain of computerized cognitive training publications, a quick count shows 99 published papers that analyze results from RCTs (including healthy adults and MCI, excluding non-randomized trials, case studies, reviews, and meta-analyses).

Thus this choice eliminates virtually every study of cognitive training conducted in the past twenty years from consideration. A few key examples of excluded studies are discussed in the bullets below. Please note that a longer list exists, however time is limited to prepare this response.

  • IMPACT33,34: IMPACT was a multi-site RCT that enrolled 487 cognitively healthy adults over the age of 65, randomized into an intervention arm (40 hours plasticity-based brain training) and a control arm (40 hours of content learning). Assessments were performed pre-training, post-training, and after a three month follow-up. Analysis showed improvement in the trained domain (auditory speed) that generalized to untrained standardized measures of memory and to a validated participant reported outcome measure. IMPACT is one of the most highly cited papers in cognitive training, with more than 450 citations since publication in 2009.
  • Auditory Training for Auditory Cognition5–7: IMPACT was replicated and extended in two follow-on studies which replicated to core results and extended the observed benefits to include standardized measures of auditory cognition including hearing in noise.
  • Auditory Training in MCI8: A further study extended the IMPACT trial into an MCI population, and showed benefits that transferred into broad measures including the 3MSE.
  • Long-Lasting Memories9–12: A European-Union funded trial examined the combination of plasticity-based brain training as used in IMPACT with physical fitness training in the Long Lasting Memories trial, and examined dosing issues and relative effects in health, MCI, and early AD populations. Broad transfer effects to untrained measures of cognitive function were documented.
  • Timed Instrumental Activities of Daily Living in health aging1,2: Speed training, as used in ACTIVE and IHAMS, has also been used in two independent studies, both of which documented transfer to timed instrumental activities of daily living (TIADLs), a well-validated measure of real-world cognitive function.
  • Speed Training in amnestic MCI14: Speed training has also been used in amnestic MCI, using video games as a control activity, in a study documenting transfer to cognitive measures and TIADLs, as well as relative improvements in fMRI measures of the default mode network and the central executive network.

All of the above trials have direct and significant implications for the ability of cognitive training programs to improve cognitive function, prevent decline in cognitive function, and lower the risk of dementia. They all should have been included in the review.

This is even more clear when recent meta-analyses are considered. There have been four major meta-analyses of cognitive training published since 201235–38, and each has agreed that specific forms of cognitive training can drive improvements that transfer to untrained assessments and real-world function. These reviews show what the current review misses because they include all RCTs of cognitive training, rather than excluding the vast majority of studies.

Thus, the interpretation in the review that with regard to cognitive training “diffusion to other domains was rare” is simply in conflict with the vast majority of the actual published literature.

Policy Concerns

These process and methodological concerns in turn lead to very significant policy concerns.

First, the strong conclusion from the review is that there has been little to no value to American taxpayers of NIA funded and organized research into how to maintain cognitive function in aging. Of course, I disagree with this conclusion – not only as it applies to brain training but also as it applies to other widely recommended interventions including exercise, diet, sleep, and socialization. I believe that NIA funded and organized research has driven important, actionable advances in this field, through ACTIVE, IHAMS, and many other studies. However, others may see this review as a verdict on the NIA’s effectiveness over the past twenty years.

Second, this review will accelerate an existing trend to its final conclusion: lack of investment in science by venture investors and brain training companies. If a NIA-organized review can arbitrarily create requirements for evidence (e.g., trials must be 6 months in duration, 5 and 10 year studies cannot be interpreted due to normal age-related attrition) without academic and industry consensus building processes, and those de novo requirements eliminate large numbers of studies from consideration (e.g., IMPACT, most speed training studies), then there is no case that can be made for commercial investment in science by such companies. Industry will invest in marketing (a sure payoff) rather than science (a risky investment in any case), and the bench-to-bedside innovation pipeline will be broken – not by industry but by the NIA itself.

Finally, the intent of the review is “to make recommendations that inform public health strategies and messaging on preventive interventions.” The only conclusion possible from the draft review is that there are no such recommendations (I note that all other interventions are rated as even less supported by evidence than cognitive training). Does the NIA truly believe that there are no actions that older Americans should take to maintain their cognitive health? This recommendation is at odds with essentially every other current set of best practices including those of the Alzheimer’s Association and the International Association of Gerontology and Geriatrics. Given the published peer reviewed evidence in this area (much of which was funded by the NIA), this is a shockingly hopeless message to senior citizens and other taxpayers.

Recommendations at This Juncture

The opportunity still exists to significantly improve the review, if the will to do so exists.

First, the missing papers should be added to the review process and fully reviewed. In particular, the full set of results from ACTIVE and IHAMS documenting transfer to IADLs, health measures, depressive symptoms, and driving measures should be included. As part of this, a set of meaningful discussions with ACTIVE and IHAMS investigators should be organized, to ensure that the full conclusions of the key NIA-funded trials are incorporated.

Second, the requirement of that studies have a duration of 6 months should be eliminated, the review should fully consider those papers, and consider evidence for benefits immediately post-training, as well as at the 6 month (and longer) periods.

Finally, and most importantly, the recommendations should incorporate a patient-centered  approach. Right now, there are a tremendous number of seniors who want to invest time and effort taking proactive steps to maintain their cognitive health. Those seniors deserve to know what is the best advice, given the evidence at this time, that the NIA and AHRQ can provide them – not at some point in the future when more research has been done – but right now.

References

  1. Edwards JD, Wadley VG, Myers RS, Roenker DL, Cissell GM, Ball KK. Transfer of a speed of processing intervention to near and far cognitive functions. Gerontology. 2002;48:329–340.
  2. Edwards JD, Wadley VG, Vance DE, Wood K, Roenker DL, Ball KK. The impact of speed of processing training on cognitive and everyday performance. Aging Ment Health. 2005;9(3):262–271.
  3. Edwards JD, Ruva CL, O’Brien JL, Haley CB, Lister JJ. An examination of mediators of the transfer of cognitive speed of processing training to everyday functional performance. Psychol Aging. 2013;28(2):314-321.
  4. Roenker DL, Cissell GM, Ball KK, Wadley VG, Edwards JD. Speed-of-processing and driving simulator training result in improved driving performance. Hum Factors. 2003;45(2):218-233.
  5. Anderson S, White-Schwoch T, Parbery-Clark A, Kraus N. Reversal of age-related neural timing delays with training. Proc Natl Acad Sci. 2013;110(11):4357–4362.
  6. Anderson S, White-Schwoch T, Choi HJ, Kraus N. Training changes processing of speech cues in older adults with hearing loss. Front Syst Neurosci. 2013;7:97.
  7. Anderson S, White-Schwoch T, Choi HJ, Kraus N. Partial maintenance of auditory-based cognitive training benefits in older adults. Neuropsychologia. 2014;62:286-296.
  8. Gooding AL, Choi J, Fiszdon JM, et al. Comparing three methods of computerised cognitive training for older adults with subclinical cognitive decline. Neuropsychol Rehabil. 2015:1–12.
  9. Bamidis PD, Fissler P, Papageorgiou SG, et al. Gains in cognition through combined cognitive and physical training: the role of training dosage and severity of neurocognitive disorder. Front Aging Neurosci. 2015;7:152.
  10. González-Palau F, Franco M, Bamidis P, et al. The effects of a computer-based cognitive and physical training program in a healthy and mildly cognitive impaired aging sample. Aging Ment Health. 2014;18(7):838-846.
  11. Klados MA, Styliadis C, Frantzidis CA, Paraskevopoulos E, Bamidis PD. Beta-band functional connectivity is reorganized in mild cognitive impairment after combined computerized physical and cognitive training. Front Neurosci. 2016;10.
  12. Styliadis C, Kartsidis P, Paraskevopoulos E, Ioannides AA, Bamidis PD. Neuroplastic effects of combined computerized physical and cognitive training in elderly individuals at risk for dementia: an eloreta controlled study on resting states. Neural Plast. 2015;2015:e172192.
  13. Mahncke HW, Connor BB, Appelman J, et al. Memory enhancement in healthy older adults using a brain plasticity-based training program: a randomized, controlled study. Proc Natl Acad Sci. 2006;103(33):12523-12528.
  14. Lin F, Heffner KL, Ren P, et al. Cognitive and neural effects of vision-based speed-of-processing training in older adults with amnestic mild cognitive impairment: a pilot study. J Am Geriatr Soc. 2016;64(6):1293-1298.
  15. Owen AM, Hampshire A, Grahn JA, et al. Putting brain training to the test. Nature. 2010;465(7299):775-778.
  16. Corbett A, Owen A, Hampshire A, et al. The effect of an online cognitive training package in healthy older adults: an online randomized controlled trial. J Am Med Dir Assoc. 2015;16(11):990-997.
  17. Anguera JA, Boccanfuso J, Rintoul JL, et al. Video game training enhances cognitive control in older adults. Nature. 2013;501(7465):97–101.
  18. Mishra J, Gazzaley A. Cross-species approaches to cognitive neuroplasticity research. NeuroImage. 2016.
  19. Mishra J, de Villers-Sidani E, Merzenich M, Gazzaley A. Adaptive training diminishes distractibility in aging across species. Neuron. 2014;84(5):1091-1103.
  20. Van Vleet T, DeGutis J, Merzenich M, Simpson G, Zomet A, Dabit S. Targeting alertness to improve cognition in older adults benefits executive functions and skill acquisition. Cortex. 2016;in press.
  21. Wolinsky FD, Unverzagt FW, Smith DM, Jones R, Wright E, Tennstedt SL. The effects of the ACTIVE cognitive training trial on clinically relevant declines in health-related quality of life. J Gerontol B Psychol Sci Soc Sci. 2006;61(5):S281-287.
  22. Wolinsky FD, Unverzagt FW, Smith DM, Jones R, Stoddard A, Tennstedt SL. The ACTIVE cognitive training trial and health-related quality of life: protection that lasts for 5 years. J Gerontol A Biol Sci Med Sci. 2006;61(12):1324-1329.
  23. Wolinsky FD, Mahncke HW, Kosinski M, et al. The ACTIVE cognitive training trial and predicted medical expenditures. BMC Health Serv Res. 2009;9:109.
  24. Wolinsky FD, Vander Weg MW, Martin R, et al. The effect of speed-of-processing training on depressive symptoms in ACTIVE. J Gerontol A Biol Sci Med Sci. 2009;64(4):468-472.
  25. Wolinsky FD, Mahncke HW, Vander Weg MW, et al. The ACTIVE cognitive training interventions and the onset of and recovery from suspected clinical depression. J Gerontol B Psychol Sci Soc Sci. 2009;64(5):577-585.
  26. Ball K, Edwards JD, Ross LA, McGwin Jr. G. Cognitive training decreases motor vehicle collision involvement of older drivers. J Am Geriatr Soc. 2010;58(11):2107-2113.
  27. Wolinsky FD, Weg MWV, Howren MB, Jones MP, Dotson MM. The effect of cognitive speed of processing training on the development of additional IADL difficulties and the reduction of depressive symptoms results from the IHAMS randomized controlled trial. J Aging Health. 2015;27(2):334-354.
  28. Unverzagt FW, Smith DM, Rebok GW, et al. The Indiana Alzheimer Disease Center’s symposium on mild cognitive impairment. cognitive training in older adults: Lessons from the ACTIVE study. Curr Alzheimer Res. 2009;6(4):375-383.
  29. Rebok GW, Ball K, Guey LT, et al. Ten-year effects of the Advanced Cognitive Training for Independent and Vital Elderly cognitive training trial on cognition and everyday functioning in older adults. J Am Geriatr Soc. 2014;62(1):16–24.
  30. Ball KK, Ross LA, Roth DL, Edwards JD. Speed of processing training in the ACTIVE study: Who benefits? J Aging Health. 2013;25(8 suppl):65S-84S.
  31. Rebok GW, Langbaum JBS, Jones RN, et al. Memory Training in the ACTIVE study: How Much is Needed and Who Benefits? J Aging Health. 2013;25(0).
  32. Willis SL, Caskie GIL. Reasoning Training in the ACTIVE Study: How Much is Needed and Who Benefits? J Aging Health. 2013;25(8 0).
  33. Smith GE, Housen P, Yaffe K, et al. A cognitive training program based on principles of brain plasticity: Results from the Improvement in Memory with Plasticity-based Adaptive Cognitive Training (IMPACT) study. J Am Geriatr Soc. 2009;57(4):594-603.
  34. Zelinski EM, Spina LM, Yaffe K, et al. Improvement in Memory with Plasticity-based Adaptive Cognitive Training (IMPACT): Results of the 3-month follow-up. J Am Geriatr Soc. 2011;59:258-265.
  35. Lampit A, Hallock H, Valenzuela M. Computerized cognitive training in cognitively healthy older adults: a systematic review and meta-analysis of effect modifiers. PLOS Med. 2014;11(11):e1001756.
  36. Kueider AM, Parisi JM, Gross AL, Rebok GW. Computerized cognitive training with older adults: a systematic review. PLoS ONE. 2012;7(7):e40588.
  37. Howren MB, Vander Weg MW, Wolinsky FD. Computerized cognitive training interventions to improve neuropsychological outcomes: evidence and future directions. J Comp Eff Res. 2014;3(2):145–154.
  38. Shao Y, Mang J, Li P, Wang J, Deng T, Xu Z. Computer-based cognitive programs for improvement of memory, processing speed and executive function during age-related cognitive decline: a meta-analysis. PloS One. 2015;10(6):e0130831.