Using artificial intelligence to semi-automate trustworthiness assessment of randomized controlled trials: a case study

Ling Shan A et al.

Highlights

• Problematic RCT data are common, but detection is time-consuming and labor-intensive.

• Case study of ChatGPT’s potential to semi-automate trustworthiness assessments.

• ChatGPT shows strong capability in extracting and synthesizing information from RCTs.

• ChatGPT automates table extraction from PDFs for analysis and verification.

• Refining AI models could save time in future RCT assessments.

Abstract

Background and Objective

Randomized controlled trials (RCTs) are the cornerstone of evidence-based medicine. Unfortunately, not all RCTs are based on real data. This serious breach of research integrity compromises the reliability of systematic reviews and meta-analyses, leading to misinformed clinical guidelines and posing a risk to both individual and public health. While methods to detect problematic RCTs have been proposed, they are time-consuming and labor-intensive. The use of artificial intelligence large language models (LLMs) has the potential to accelerate the data collection needed to assess the trustworthiness of published RCTs.

Methods

We present a case study using ChatGPT powered by OpenAI’s GPT-4o to assess an RCT paper. The case study focuses on applying the trustworthiness in randomised controlled trials (TRACT checklist) and automating data table extraction to accelerate statistical analysis targeting the trustworthiness of the data. We provide a detailed step-by-step outline of the process, along with considerations for potential improvements.

Results

ChatGPT completed all tasks by processing the PDF of the selected publication and responding to specific prompts. ChatGPT addressed items in the TRACT checklist effectively, demonstrating an ability to provide precise “yes” or “no” answers while quickly synthesizing information from both the paper and relevant online resources. A comparison of results generated by ChatGPT and the human assessor showed an 84% level of agreement of (16/19) TRACT items. This substantially accelerated the qualitative assessment process. Additionally, ChatGPT was able to extract efficiently the data tables as Microsoft Excel worksheets and reorganize the data, with three out of four extracted tables achieving an accuracy score of 100%, facilitating subsequent analysis and data verification.

Conclusion

ChatGPT demonstrates potential in semiautomating the trustworthiness assessment of RCTs, though in our experience this required repeated prompting from the user. Further testing and refinement will involve applying ChatGPT to collections of RCT papers to improve the accuracy of data capture and lessen the role of the user. The ultimate aim is a completely automated process for large volumes of papers that seems plausible given our initial experience.

What is new?

Key findings

• Artificial intelligence large language model (LLM) ChatGPT shows potential to accelerate RCT trustworthiness assessment by automating information extraction, synthesis, and data table extraction from research PDFs.

What this adds to what is known?

• Methods to detect untrustworthy RCTs exist but are labor-intensive and time-consuming. Automating this process with LLM could accelerate assessments and handle more papers efficiently.

What is the implication and what should change now?

• ChatGPT’s current performance suggests the entire workflow could be fully automated with further model training and API integration, enabling software to assess RCT papers automatically.

• Future efforts should prioritize enhancing LLM’s capability to evaluate RCT trustworthiness through improved model training and validation against human assessment results and established standards.

Figure 1 Example of ChatGPT responses to (A) TRACT items targeted questions and (B) data table extraction task.

LEGGI TUTTO https://www.jclinepi.com/article/S0895-4356(25)00005-8/fulltext#:~:text=10.1016/j.jclinepi.2025.111672

INTELLIGENZA ARTIFICIALE: ChatGPT VALUTA LA VALIDITA’ DI UN TRIAL AL PARI DELL’OSSERVATORE UMANO! IMPORTANTE O META-ANALISTI! DA”J OF CLINICAL EPIDEMIOLOGY” (FREE!)