Two images and underwater sound help AI assess fish feeding

Combining sight and sound without the bulk

Feeding decisions matter in aquaculture. Too much food can mean waste and poorer water quality; too little can limit growth. Automated monitoring could help inform those decisions, but recognizing feeding activity is not the same as proving that an automated feeder improves farm performance.

Phi-Hung Hoang, Van-Manh Tran, Nam-Thuan Trinh and Thi-Thu-Hong Phan, at FPT University in Da Nang, describe HMMF-Net in an arXiv preprint posted on November 6, 2026. Their system classifies feeding intensity from synchronized video and underwater audio. The supplied evidence establishes its availability as a preprint, not its peer-review status.

The engineering question is practical: how much information can a model retain while remaining small enough to run on an embedded computer?

Two snapshots instead of a full movie

For each two-second clip, the visual branch takes only the first and last frames. It stacks their red, green and blue channels into a six-channel input for an adapted EfficientNetB0 image-recognition network.

This gives the model observations from two separated moments without the expense of processing a full video sequence. It does not explicitly track motion between them. In the researchers’ validation experiments, the first-and-last-frame pairing improved accuracy over a single final frame across all seven image architectures tested. Adding more frames did not consistently help.

The acoustic branch receives a log-mel spectrogram: a representation of how sound energy varies across time and frequency. Its processing units examine temporal patterns, frequency patterns and local combinations of both, learning how to weight those contributions.

This audio network was chosen for compactness, not superior standalone accuracy. It achieved 78.87% validation accuracy, below the strongest acoustic comparator’s 82.64%, but used substantially fewer parameters and calculations.

Small messages between the branches

Rather than combining sight and sound only at the final prediction, HMMF-Net exchanges information at four intermediate stages.

At each stage, it compresses each branch’s features into four short numerical representations, or tokens. A shared set of “messenger” tokens gathers information from both modalities, combines it with input-dependent weights and passes context back to each branch.

The approach still uses attention, a mechanism for weighting relevant information. Its economy comes from applying that interaction to compact representations rather than directly across the original, larger feature maps.

The final output is one of four categories: None, Weak, Medium or Strong. These describe observed responses to food, from no response to active movement among food items and consumption of all available food.

What the benchmark showed

The researchers reused all 27,067 paired clips in AV-FFIA, a public dataset originally collected by Cui and colleagues. It records one species, Oplegnathus punctatus, in a controlled aquaculture environment in Yantai, China.

They used five-fold cross-validation. Each fold supplied a held-out test subset; the remaining data were divided into training and validation portions.

HMMF-Net achieved mean held-out test accuracy of 96.77%, with a standard deviation of 0.27 percentage points. Its F1-score, which combines precision and recall, was 96.88%. Combining the same audio and visual branches through ordinary final-feature concatenation achieved 95.97% test accuracy, making the messenger-based system’s gain 0.80 percentage points.

The reported model contains 3.987 million parameters and requires 1.002 billion floating-point operations. Validation analysis found that Medium feeding intensity remained the hardest category to distinguish.

Fast processing is not field validation

On an NVIDIA Jetson Orin Nano using TensorRT at FP32 precision, accuracy remained comparable at 96.76%. Preprocessing took 7.78 milliseconds per sample and inference took 11.54 milliseconds, totaling about 19.32 milliseconds. Those processing times are separate from the two-second observation window needed to supply each sample.

The deployment test used recorded clips concatenated into approximately ten-minute sequences, not a demonstrated live feeding-control trial. The study does not establish feed savings, improved growth or better water quality.

Generalization is another open question. One species and one controlled facility cannot establish reliability across farms, changing acoustic conditions or open-water cages. Two fixed frames may also miss prolonged or irregular behavior. The authors identify broader validation, adaptive frame selection and integration with feeding actuators as future work. For now, the evidence supports an efficient benchmark classifier—not a proven autonomous feeding system.