FusionCLIP learns from paired plasma videos and diagnostic signals

Two views of the same plasma

Monitoring a tokamak plasma involves combining observations of different kinds. Visible-light videos show radiation structures and their changing shapes, while diagnostic signals track physical quantities over time. Neither source arrives in a form that is straightforward for a machine-learning model to combine: much of a video frame may be background, and different sensors collect measurements at very different rates.

In a November 2026 preprint, Qiang Chen and colleagues introduce FusionCLIP, a system designed around those two difficulties. Rather than learning primarily by reconstructing observations or predicting their next values, it learns to associate videos with diagnostic signals recorded during the same short interval.

Learning which observations belong together

The researchers assembled EAST Multimodal Diagnostics, or EAST-MMD, from 1,307 discharges at the EAST tokamak. They used 667 discharges for pretraining, producing about 2.73 million paired video-and-signal windows. Each window spans 50 milliseconds, with successive windows starting two milliseconds apart, so many overlap.

Each sample contains 12 frames selected from a camera operating at 1,000 frames per second, alongside 38 diagnostic variables. The diagnostic measurements retain their original timestamps rather than being resampled onto one common input grid.

During contrastive pretraining, the system learns to make the internal representations of matching video and signal windows more similar, while separating nonmatching pairs. Overlapping windows from the same discharge are excluded from the nonmatching examples, avoiding a training rule that would push closely related observations apart.

Selecting video detail and respecting sensor timing

FusionCLIP’s video component divides observations into small space-and-time units called tokens. Its Spatiotemporal Stability Selection method ranks them using three cues: how their features respond to filtering, whether nearby regions support them, and whether that support persists across neighboring time steps.

High-ranking tokens keep their individual detail. Unselected tokens are averaged into summary tokens, preserving broader context rather than simply discarding everything outside the selection. The default setting uses a 50% mask ratio.

The signal component, called the Temporal Query Encoder, handles sparse measurements individually and groups densely sampled measurements into patches. It then uses 25 learned time queries within each window to gather information, with attention biased toward observations nearby in time. This produces aligned representations without first forcing every raw signal onto the same sampling schedule.

Three separate diagnostic tests

The remaining 640 discharges supplied labeled data for training and evaluating three downstream tasks. These were divided into 447 training, 96 validation and 97 test discharges. Splitting by discharge kept windows from the same discharge out of different subsets. Each task was fine-tuned independently.

The tasks were disruption prediction, classification of four plasma-edge states, and estimation of H98, a measure of confinement performance relative to a reference scaling law. The precomputed H98 target was excluded from model inputs.

FusionCLIP achieved a combined score of 89.07%, compared with 82.80% for the strongest general-purpose pretrained configuration and 86.95% for an EAST-pretrained contrastive configuration without its video-selection method. This combined score averages disruption F1, edge-state macro-F1 and H98 R²; it is not a single overall accuracy.

For disruption prediction, FusionCLIP reported a 95.83% true-positive rate and a 2.74% false-positive rate. Average warning time was approximately 222 milliseconds among successfully warned disruptive discharges. A warning counted as successful only if it preceded disruption by at least 10 milliseconds.

What the results establish—and what they do not

Component-removal experiments supported the contributions of timestamp-aware encoding, video selection and contrastive pretraining. However, improvements were not uniform across every metric. The contrastive baseline without video selection had slightly better edge-state accuracy and H98 R², while another selection setting produced lower H98 error.

The comparisons also involve different signal encoders: general-purpose visual baselines used a temporal convolutional network, whereas the EAST-pretrained configurations used the Temporal Query Encoder. Their performance differences therefore do not isolate pretraining alone.

Most importantly, evaluation was limited to EAST. The preprint does not establish transfer to other tokamaks or demonstrate live operational control. Its evidence supports a more bounded conclusion: learning correspondences between paired observations can improve the balance of diagnostic performance on this dataset.