When the train–test split changes what a traffic classifier appears to know

Encrypted traffic classifiers try to identify applications or service categories without reading message contents. Packet sizes, directions and timing supply behavioural clues. But a high test score can also reflect regularities tied to a recording or destination rather than behaviour that transfers to a new setting.

In an arXiv preprint submitted on November 4, 2026, Mayan Sharma, Kriti Saini and Devansh Behl examine that distinction using two public datasets. Their central finding is not that random splitting always fails. It is that different collection procedures create different shortcuts, and evaluation must test the relevant boundary. The supplied landing page lists no journal publication.

The same model, different questions

The researchers extracted 164,348 flows from 75 captures in the 5G Traffic dataset, covering 15 applications in six service categories. Each capture recorded one application. For CESNET-QUIC22, they sampled 201,600 flows across 28 days and 18 categories from backbone QUIC traffic, deliberately balancing categories within each day.

Their FlowCon-X model combines a packet-sequence encoder with an encoder for flow-level aggregates. Training includes objectives intended to organise similar flows into useful representations and suppress recording-related information. Identifiers such as addresses and capture IDs are excluded from its inputs but retained for constructing splits and diagnosing shortcuts.

On 5G Traffic, assigning individual flows randomly to training and test produced macro-F1 of 0.992. Keeping each capture entirely on one side reduced it to 0.567. Macro-F1 averages class-level precision–recall scores, rather than measuring the fraction of all flows classified correctly. The contrast persisted across eight random seeds, with a reported multiple-comparison-corrected p-value of about 0.008.

There is an important qualification: the grouped split sometimes also holds out whole applications, because some have only one or two captures. It therefore tests a mixture of new recordings and new applications, not recording independence alone.

CESNET behaved differently. Random, day-grouped, temporal and client-disjoint splits all produced macro-F1 near 0.78, with no statistically detected differences in the eight-seed comparison. Holding out server addresses reduced the score to about 0.588. Separating days did not expose the same weakness as separating destinations.

Diagnose shortcuts before comparing models

The authors propose training simple decision-tree probes on individual identifiers under candidate splits. These probes ask how much a label can be predicted from provenance alone; they are diagnostics, not the intended traffic classifier.

On 5G Traffic, a capture-ID probe reached 0.983 under random splitting and zero when captures were held out. A server-address probe reached 0.999 under random splitting. These results show why near-perfect performance need not demonstrate transferable application recognition.

CESNET adds a different warning. Its labels are derived from the server-name indication, or SNI, associated with a connection. An SNI-only probe scored 0.967 in several protocols, but 0.791 under server-disjoint evaluation. Despite broader wording in the abstract, the detailed results do not show 0.967 under every split. More fundamentally, predicting these labels means reproducing a name-based proxy, not checking independently verified application identity.

Complex training does not guarantee a better classifier

Comparisons used identical released splits for eight classical and four deep baselines. On CESNET's day-grouped split, FlowCon-X scored about 0.780, ahead of the tested deep baselines but behind gradient-boosted trees using the first 20 signed packet sizes, at 0.790. The older AppScanner baseline reached 0.772. These particular comparisons are day-grouped results, not evidence of performance on unseen servers.

On capture-grouped 5G service classification, gradient-boosted trees using flow statistics reached 0.849, while the neural models clustered around 0.54–0.60. The authors do not claim state-of-the-art accuracy: byte-based pretrained transformers were not tested because the retained records lack payload bytes. Their model also received a limited training budget, and validation scores were often still improving.

Tests of unknown-application detection further narrowed the architectural claims. Distance-based rejection outperformed thresholding the largest softmax probability, but removing the metric-learning objectives improved rejection. A reported correlation between class separation and rejection quality spans only five variants with other differences; it does not establish that separation caused the effect.

Cross-dataset adaptation was promising but preliminary. With the encoder frozen, five labelled flows per shared class raised CESNET-to-5G macro-F1 from 0.246 to 0.676, compared with 0.743 using 100. This experiment covered only three mapped classes, one seed and no competing baseline.

The wider limits matter: two networks, proxy labels, imposed sampling balance and splits that leave other dependencies uncontrolled. These experiments do not establish a universal splitting rule or long-term drift resistance. Their practical contribution is narrower: inspect what collection identifiers reveal, and state exactly which unfamiliar conditions a reported score actually tests.