Physicians preferred conversational database queries, but accuracy remained a concern

From questions to database queries

Electronic health records store information across interconnected tables. Retrieving a patient group or calculating a clinical statistic can require both knowledge of the database and proficiency in Structured Query Language, or SQL. Text-to-SQL systems use language models to translate everyday questions into executable database queries.

In this preprint, researchers from the Korea Advanced Institute of Science and Technology and Samsung Medical Center compared two ways of using such a system. One answered each question in a single interaction. The other supported clarification, continued questioning, and suggested follow-up analyses.

The authors argue that evaluations based on fully specified questions miss an important part of clinical information seeking: users often need help working out exactly what they mean before a database can answer reliably.

What the physicians tested

The main study involved 14 physicians who regularly worked with electronic health record data. Each used both interfaces, with exposure order counterbalanced through randomized assignment. Most had substantial experience with clinical records but limited SQL proficiency, and nearly all reported almost daily conversational AI use.

Both interfaces used GPT-5.4. Participants were not told the model’s identity. The database was the MIMIC-IV Clinical Database Demo, a de-identified dataset containing structured records for 100 intensive care patients, without free-text clinical notes.

Participants first selected three predefined questions, requiring information from one, two, or three tables, and tried each with both interfaces. They then explored the database using questions reflecting their own clinical or research interests.

For ambiguous requests, the single-turn system inferred an interpretation and explained it alongside its answer. The conversational system could ask for clarification before generating SQL.

Higher usability, stronger preference

The authors report that average System Usability Scale scores, measured from 0 to 100, were 68.2 for the single-turn interface and 85.9 for the conversational interface. The paired difference was statistically significant, with p below 0.001.

Participants also rated the conversational interface more favorably on perceived usefulness, ease of use, confidence in using it, and intention to use it. Anxiety did not differ significantly.

The paper contains a discrepancy in its satisfaction reporting. Its abstract states that 93% of participants reported greater overall satisfaction. However, Section 2.2.3 reports that 13 of 14 physicians, or 93%, found the conversational interface “much” more satisfying, while the remaining physician, or 7%, found it “somewhat” more satisfying. The detailed results therefore indicate greater satisfaction among all 14 participants, or 100%; 93% describes only the strongest satisfaction category.

Twelve preferred the conversational interface for real-world adoption, while two would use either interface depending on the task. Ten trusted its results more, and four reported comparable trust. None preferred or trusted the single-turn system more.

These findings describe perceived usability and trust, not proof that the answers were correct.

Why clarification mattered

The dialogue analysis pooled the main study with two pilots, covering 21 healthcare professionals, 183 dialogue sessions, and 425 user turns. The pilots included physicians, nurses, and clinical data-team members and used evolving study procedures.

Within the conversational interface, 40 of 161 freely composed questions triggered clarification, or 24.8%. Only one of 56 predefined questions did so, or 1.8%. Of 42 clarification exchanges, 36 were resolved in one round, four required multiple rounds, and two ended without a recorded response.

The researchers classified 64.3% of exchanges as involving electronic-record or clinical-domain knowledge. Common issues included defining the patient population, translating a clinical concept into a computational rule, and choosing the appropriate data source. For example, a medication prescription and a recorded administration are not interchangeable.

Participants valued being able to refine questions and revisit uncertain answers. However, only 32.8% of eligible follow-up suggestions were taken up in the next turn.

Conversation did not guarantee correctness

An additional evaluation examined 186 evaluable SQL-generating turns from the main study’s open-ended conversational phase. Turns corresponding to fixed questions were checked against reference results; other outputs were manually reviewed by the study designers using the preceding dialogue. They judged 134 turns correct, an accuracy of 72.0%.

Errors included confusing drug names through substring matching, counting admissions instead of unique patients, imposing unintended filters, and misinterpreting database fields. Approximately one-quarter of incorrect turns reflected errors carried forward through the conversation. Participants also reported lost context and excessive clarification.

The small physician sample and limited demonstration database constrain generalization to larger clinical systems. The authors characterize conversation as a practical prerequisite for natural-language record access. Their findings support its usability value, while the accuracy review shows why preference and trust alone cannot establish readiness for routine deployment.