e-ISSN 2148-0532 DOI 10.54233 Open Access · CC BY-NC-SA 4.0 Türkçe
Endoüroloji Bülteni — Endourology Bulletin Submit Manuscript
Original Article Open Access

Evaluating Kidney Cyst Information from Chatbots: A Comparative Study

Chatbotlardan Elde Edilen Böbrek Kisti Bilgilerinin Değerlendirilmesi: Karşılaştırmalı Bir Çalışma

Author information
  1. Polatlı Duatepe Devlet Hastanesi
  2. Hatay Mustafa Kemal Üniversitesi
  3. Sincan Training and Research Hospital, Department of Urology
Issue
Volume 18 · Issue 3
Pages
126-133
Published
September 30, 2026
DOI
https://doi.org/10.54233/endourolbull-1848567 (opens in a new tab)

Abstract

Objective: Kidney cysts are the most common acquired lesions of the kidney. This study aims to assess the reliability and quality of responses provided by three different artificial intelligence chatbots to questions related to kidney cysts.

Materials and Methods: Using the Semrush platform, we analyzed kidney cyst-related search trends over five years. We examined 25 questions, categorized into four subgroups: “general information”, “symptoms and diagnostic methods”, “treatment and follow-up procedures”, and “side effects and complications”. These questions were posed to three chatbots (ChatGPT-4, Gemini Pro, Llama 3.1 Large). Response readability was assessed using Flesch-Kincaid Grade Level (FKGL) test and Flesch Reading Ease Score (FRES) test. Reliability and quality were evaluated using the Ensuring Quality Information for Patients (EQIP) score and modified DISCERN score.

Results: Llama had a significantly higher FKGL score than ChatGPT (p = 0.022), with no significant difference between Llama and Gemini (p = 0.500), or Gemini and ChatGPT (p = 0.525). On the FRES scale, ChatGPT scored higher than Gemini, and Gemini outperformed Llama (p = 0.006 and p = 0.012, respectively). The median modified DISCERN score was lowest for ChatGPT (2), compared to Gemini (3) and Llama (3) (p < 0.001). The mean EQIP score was 71.22 ± 3.91 for ChatGPT, 77.72 ± 1.16 for Gemini and 73.26 ± 2.70 for Llama (p < 0.001). ChatGPT had significantly lower quality scores than the other two chatbots.

Conclusion: Although ChatGPT generated the most readable responses according to the FRES score, it demonstrated lower quality and reliability than Gemini and Llama. Overall, chatbot responses remained challenging in terms of readability and comprehension, highlighting the need for further improvements before they can be considered a reliable standalone source of patient education.

More From This Issue