Telecom churn study finds selective gains for foundation models over tuned trees

A comparison built around retention decisions

In an arXiv preprint dated 6 November 2026, Ismail Hossain Polas of Jahangirnagar University examines whether tabular foundation models offer practical advantages for telecom customer retention. These models work with tables of customer information, using labeled examples as context for predictions rather than fitting a new model through conventional training for each dataset.

An earlier churn benchmark reported large advantages over tree-based models, but used untuned baselines. This study asks a narrower question: do those advantages survive tuning, and do they translate into higher modeled campaign profit and usable probability estimates?

The main benchmark covers five public datasets. Cell2Cell, Iranian and Orange come from real telecom operators; IBM Telco and Maven are fictional teaching datasets. Four foundation models—TabPFN-2.5, TabPFN-3.5, TabPFN-3.5-Fast and TabICL v2—face tuned LightGBM, XGBoost and CatBoost, alongside two additional conventional models.

Separating ranking, profit and probabilities

For each dataset, the researcher created 10 seeded splits, allocating 60% of customers to training, 20% to calibration and 20% to testing. Models shared the same splits. The foundation models received the full training portion as context, while the three gradient-boosted tree models underwent automated searches over their settings.

The study distinguishes ranking likely churners from estimating their probabilities accurately. A ranking can identify useful campaign targets even if its numerical probabilities are systematically too low or too high.

Its primary measure, expected maximum profit, or EMP, evaluates contacting customers from the top of a model’s ranking. It assumes a customer lifetime value of 200 units, an incentive cost of 10 and a contact cost of one, while averaging over uncertainty in offer acceptance. For each acceptance probability, EMP uses the most profitable cutoff. It therefore measures ranking under specified financial assumptions—not profit observed in an actual retention campaign, or whether raw probabilities support a ready-to-use decision threshold.

The reference tree model was selected using calibration data, not test results. Statistical tests accounted for overlap among random splits, and Holm correction addressed the risk of false positives across 20 predeclared comparisons.

Where the gains held up

TabPFN-3.5 achieved significantly higher EMP than the selected tuned-tree reference on Cell2Cell and Iranian. The gains were 1.9% and 7.6%, respectively, corresponding to 0.175 and 0.597 profit units per customer. On both datasets, its advantage also survived the stricter requirement of beating each of the three tuned tree models separately.

Orange produced a more uncertain result. TabPFN-3.5 led in all 10 splits, with a mean relative EMP gain of 14.3%. However, its adjusted p-value was 0.155, above the study’s 0.05 significance threshold. The large relative figure also reflects a small baseline profit on this low-churn dataset; it is not a confirmed benefit transferable to other operators.

No foundation-model advantage survived correction on either fictional dataset. TabICL v2 beat the selected reference significantly only on Iranian, and did not establish superiority over all three tuned trees there. The results support specific model-and-dataset combinations, not a blanket victory for foundation models.

When probabilities change the decision

Calibration—agreement between predicted probabilities and observed outcomes—mattered especially on Orange. TabICL v2 underpredicted churn by 2.1 percentage points against an observed rate of 7.3%.

Using its raw probabilities in a cost-based contact policy lost 15.7% of the profit available from the best cutoff on the same ranking. Two recalibration methods, fitted on the separate calibration data, reduced that loss to 7.5% and 6.3%. These are modeled policy losses, not measured company revenue losses.

TabPFN-3.5’s average predicted churn remained within 0.2 percentage points of the observed rate on every dataset. Nevertheless, matching the overall rate is not the same as eliminating every policy loss.

Limits and deployment conditions

Exploratory checks increased tuning budgets for LightGBM and CatBoost and kept duplicate customers together when splitting affected datasets. Neither changed the main conclusions within the five splits each check covered. These were bounded checks: XGBoost was not retuned with the larger budget, and feature engineering was not tested. The trees were tuned for ranking performance rather than EMP directly.

A separate E-Commerce comparison found that tuning reduced TabICL v2’s precision–recall ranking advantage from 6.1 to 4.0 points. That check used a different protocol from the earlier benchmark, and the dataset’s origin is undocumented.

Most importantly, repeated splits reuse customers from just three real operators. They assess stability within those datasets, not independent evidence from many operators. The findings remain preprint results.

Licensing adds another boundary. Under the weight licenses examined in October 2026, the tested TabPFN-3.5 and TabPFN-2.5 models require a commercial agreement for production use. The tested TabICL v2 checkpoint uses BSD-3 terms. Predictive gains alone therefore do not settle deployment decisions.