Learning from unlabelled smart-glasses signals improves facial-expression recognition

Recognizing facial expressions from smart glasses usually requires recordings labelled with the movements people performed. Collecting those examples can be costly, particularly when systems must work for new wearers. A study by Zhengtao Gao and colleagues, available as an arXiv preprint, investigates a different starting point: learning patterns from unlabelled recordings before training a model to recognize specific expressions.

The researchers report that this self-supervised pretraining improved performance across all six labelled-data budgets tested. Their strongest label-efficiency comparison was between a pretrained model using 5% of the labelled training data and a model trained from scratch using 100%. The pretrained model scored slightly higher, although the experiment does not establish equivalent savings in real-world collection costs.

Learning without a camera

The study uses OCOsense smart glasses, whose non-contact optical sensors measure skin movement around facial muscles. The recognition system combines these signals with head-motion and orientation information, using 30 sensor channels in total. It does not analyze photographs of faces.

For pretraining, the team used CLUES recordings from 52 participants performing cognitive-load tasks interspersed with relaxed breaks in a laboratory. These recordings captured naturally occurring facial and head movements but were not collected specifically for facial-expression recognition.

The model used a masked autoencoder built around a Vision Transformer. Rather than processing images, it divided multichannel sensor recordings into patches. During pretraining, 75% of the patches were hidden, and the model learned to reconstruct their original contents from the visible portions. This provided a learning signal without requiring expression labels.

The researchers then removed the reconstruction decoder, attached a classifier, and fine-tuned the encoder for expression recognition. The supervised baseline used the same encoder architecture but started without pretraining.

What the label comparison showed

The downstream dataset was an extended version of OCOGEM containing 75 participants. They deliberately performed smiles, frowns, eyebrow raises and squeezed-eye expressions, with neutral periods between poses. The pretraining and expression-recognition datasets shared no participants.

Evaluation used five-fold cross-validation with participants kept in separate folds. Within each run, training, validation and test participants were separated. The researchers reduced the labelled training data to 5%, 10%, 20%, 40%, 80% or 100%, sampling within each participant-and-expression group while leaving validation and test sets unchanged. The smallest budget therefore meant fewer labelled examples, not training on only 5% of the participants.

At the 5% budget, the pretrained model achieved a macro F1 score of 0.845, compared with 0.802 for training from scratch. Macro F1 gives each expression class equal weight when combining precision and recall. Accuracy at that budget was 89.8% with pretraining and 86.8% without it.

With all labelled training data, macro F1 reached 0.865 for the pretrained model and 0.839 for the baseline. Thus, the pretrained model using 5% of the labels slightly exceeded the full-label baseline on this measure. It also scored higher on accuracy, AUC and Matthews correlation coefficient in that comparison. Pretraining improved all four reported metrics at every tested budget; these comparisons should not be read as a demonstrated statistical-significance result.

Facial sensors carried most of the signal

A separate experiment trained supervised models from scratch using different sensor combinations. Optical navigation channels were the strongest individual modality. Combining navigation and proximity channels produced results close to using every sensor channel.

All channels together gave slightly higher accuracy, F1 and Matthews correlation coefficient, while navigation plus proximity had slightly higher AUC. These findings concern the separately trained sensor-ablation models, not a direct test of which inputs are essential to the pretrained model.

Everyday use remains an open question

The labelled expressions were posed on cue, in a fixed order, under controlled conditions. The authors acknowledge that such movements can be more exaggerated and easier to distinguish than spontaneous expressions. Performance during everyday activities, across repeated sessions and after glasses are removed and replaced remains to be tested.

The study also tested only one self-supervised objective and encoder architecture. Personalization and continual adaptation are proposed directions, not demonstrated capabilities.

Finally, camera-free sensing does not make expression inferences non-sensitive. The authors call for informed, revocable consent and caution that predicted expressions are behavioral signals—not direct measurements of emotional state. The demonstrated contribution is narrower: unlabelled recordings improved recognition of posed expressions on unseen participants while reducing the labelled training examples needed.