Can language models learn to rank medical risk when follow-up is incomplete?

A patient who leaves a study without having the event under investigation has not necessarily avoided it. Perhaps the event never occurred; perhaps observation simply ended first. This incomplete follow-up, called censoring, is a routine feature of medical data—and a challenge for machine-learning methods that expect a clear outcome label for every person.

The study presents LLMSurvival, a way to train large language models (LLMs) for survival analysis despite censoring. Survival analysis asks not only whether an event occurs, but also when. Rather than asking a model to predict a precise time or assign a yes-or-no label at a chosen date, the researchers ask it to compare two patients: based on their clinical details, whose event is expected sooner?

That shift can make some incomplete records useful. If one patient experiences an event before another patient’s follow-up ends, the pair’s order is known—even if the second patient’s eventual outcome is not. But if follow-up ends before the first patient’s event, their relative order may be uncertain, so that pair is not used. In the paper’s example, a person censored at year eight can be compared with someone whose event occurred at year five, but not reliably with someone whose event occurred at year ten.

From patient details to a risk ranking

The team converted structured clinical variables into short text descriptions, such as a patient’s age or a bone-density category. They fine-tuned instruction-following LLMs on comparable patient pairs, teaching the models to identify who experienced the event earlier. To score a new patient, the system compared that person with a set of 50 reference patients, or “anchors,” selected from the training data. The share of comparisons in which the new patient was ranked as higher risk became the final risk score.

The researchers tested the framework in two settings: ICU mortality using MIMIC-IV data, and fragility fractures in a NewYork-Presbyterian/Weill Cornell Medicine cohort. The ICU analysis included 9,928 patients; the fracture cohort included 11,522 people aged 50 or older who had a bone-density scan and at least two years of follow-up. The fracture data were retrospective clinical records, and access is restricted. The authors used training, validation and test splits, and estimated confidence intervals by resampling patients.

They evaluated two publicly available LLMs, Llama and Qwen, and compared their results with conventional and deep-learning survival models, as well as established clinical scores. The main measure, the concordance index, assesses how often a model correctly ranks which of two comparable patients experiences the event sooner. It measures discrimination, not whether a predicted probability is accurate or whether using the model improves care.

What the tests found—and what they did not

For ICU mortality, the Llama version achieved a concordance index of 0.776, compared with 0.753 for the Cox proportional hazards model. For fracture risk, the corresponding scores were 0.742 and 0.737. These are differences in the metric’s score, not percentage-point changes in patients’ outcomes. The paper also reports better overall concordance than several deep-learning survival models in these tests, though performance varied by task and comparison. A second LLM backbone showed similar patterns.

The findings suggest that comparison-based training can make censored survival data usable in an end-to-end LLM workflow. The method does not recover the unknown event time for a censored patient; it uses only comparisons supported by the observed follow-up. That distinction matters: patients with incomplete records contribute information, but not necessarily to every comparison.

The evidence remains limited. The framework was assessed on two cohorts, including one from a single health system, and the authors say training performance varied with sampling choices. Comparing every test patient against anchors also adds computational work. Inputs were structured tabular data, not clinical notes or images. Although the paper describes the models as locally deployable and reports comparisons with clinical scores, local deployment alone does not establish privacy protections, clinical safety or usefulness in practice.

This is an arXiv preprint, so its results should be treated as preliminary rather than as settled clinical evidence. The study shows a promising way to frame a difficult modeling problem, but broader external validation would be needed to establish how well the approach transfers to other hospitals and patient populations. A better ranking in these datasets is a starting point—not proof that an AI system is ready to guide decisions about individual patients.