Multi-layer Attentive Probing Enhances Audio Representation Transfer in Bioacoustics

A new research paper published on arXiv challenges standard evaluation methods in bioacoustic machine learning. The study, led by Marius Miron and colleagues, investigates how probing heads map audio representations to downstream task labels. Current benchmarks typically employ fixed, low-capacity probes, such as linear layers on the final encoder layer, which may bias results by ignoring interactions between encoder features and probe design. The researchers systematically evaluated various probing strategies, including last-layer versus multi-layer probing and linear versus attention-based probes, across two major bioacoustic benchmarks: BEANs and BirdSet. Their findings demonstrate that larger probe heads leveraging time information significantly outperform traditional methods. Specifically, multi-layer probing improved downstream task performance across all tested models, while attention probes proved superior to linear probes for transformer architectures. The authors conclude that existing benchmarks may misrepresent encoder quality due to reliance on simplistic last-layer setups. This work suggests a need for updated evaluation standards in bioacoustics to accurately assess model capabilities and improve the transfer of audio representations for ecological and biological monitoring applications.