A variety of AI tools can be used during the systematic review or evidence synthesis process. These may be used to assist with developing a search strategy; locating relevant articles or resources; or during the data screening, data extraction or synthesis stage. They can also be used to draft plain language summaries.
The overall consensus is that the AI tools can be very useful in different stages of the systematic or other evidence review but that it is important to fully understand any bias and weakness they may bring to the process. In many cases using new AI tools, which previous research has not assessed rigorously, should happen in conjunction with existing validated methods. It is also essential to consider ethical, copyright and intellectual property issues for example if the process involves you uploading data or full text of articles to an AI tool.
Examples of Key Terms:
Artificial Intelligence
Computer Vision
Deep Learning
Generative AI
Machine Learning
Natural Language Process (NLP)
Automated synthesis of information
Speech and text analysis
Facial recognition
Purpose
Using AI as a mediating step in between sections of the systematic review process
Creates efficient operations and reduces the amount of time spent on more time-heavy portions
Using AI as an aid to make faster decisions
Increasing transparency and clarity in review questions
Determine the strengths and weaknesses of different sections of the systematic review process
Identify the areas that take the most amount of time
Assess the risk in automation
Talk to research and library team about where automated processes would benefit in the process
Human Review Primary (in between first and second step):
Human Review Secondary (in between second and third step):
Human Review Tertiary (in between third and fourth step):
Of course all of this will change. The use of AI for evidence synthesis is a rapidly developing field, but for clinical use it will still be necessary that syntheses meet the underlying standards of transparency and rigour which are so far absent. Keep this in mind when reading the latest tech hype.
Khalil H, Ameen D, Zarnegar A. Tools to support the automation of systematic reviews: a scoping review. J Clin Epidemiol 2022; 144: 22-42 https://www.jclinepi.com/article/S0895-4356(21)00402-9/fulltext
"The current scoping review identified that LitSuggest, Rayyan, Abstractr, BIBOT, R software, RobotAnalyst, DistillerSR, ExaCT and NetMetaXL have potential to be used for the automation of systematic reviews. However, they are not without limitations. The review also identified other studies that employed algorithms that have not yet been developed into user friendly tools. Some of these algorithms showed high validity and reliability but their use is conditional on user knowledge of computer science and algorithms."
Khraisha Q, Put S, Kappenberg J, Warraitch A, Hadfield K. Can large language models replace humans in systematic reviews? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Res Syn Meth. 2024; 1-11. doi:10.1002/jrsm.1715
"Although our findings indicate that, currently, substantial caution should be exercised if LLMs are being used to conduct systematic reviews, they also offer preliminary evidence that, for certain review tasks delivered under specific conditions, LLMs can rival human performance."
Mahuli, S., Rai, A., Mahuli, A. et al. Application ChatGPT in conducting systematic reviews and meta-analyses. Br Dent J 235, 90–92 (2023). https://doi.org/10.1038/s41415-023-6132-y
Explores using ChatGPT for conducting Risk of Bias analysis and data extraction from a randomised controlled trial.
Ovelman, C., Kugley, S., Gartlehner, G., & Viswanathan, M. (2024). The use of a large language model to create plain language summaries of evidence reviews in healthcare: A feasibility study. Cochrane Evidence Synthesis and Methods, 2(2), e12041. https://onlinelibrary.wiley.com/doi/abs/10.1002/cesm.12041
Qureshi, R., Shaughnessy, D., Gill, K.A.R. et al. Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation?. Syst Rev 12, 72 (2023). https://doi.org/10.1186/s13643-023-02243-z
"Our experience from exploring the responses of ChatGPT suggest that while ChatGPT and LLMs show some promise for aiding in SR-related tasks, the technology is in its infancy and needs much development for such applications. Furthermore, we advise that great caution should be taken by non-content experts in using these tools due to much of the output appearing, at a high level, to be valid, while much is erroneous and in need of active vetting."
van Dijk SHB, Brusse-Keizer MGJ, Bucsán CC, et al.Artificial intelligence in systematic reviews: promising when appropriately used. BMJ Open 2023;13:e072254. doi: 10.1136/bmjopen-2023-072254
Suggests how to conduct a transparent and reliable systematic review using the AI tool ‘ASReview’ in the title and abstract screening.