Detect+Track: robust and flexible software tools for improved tracking and behavioural analysis of fish

Journal article

We introduce a novel video processing method called Detect+Track that combines a deep learning-based object detector with a template-based object agnostic tracker to significantly enhance the accuracy and robustness of animal tracking. Applied to a behavioural experiment involving Picasso triggerfish (Rhinecanthus aculeatus) navigating a randomized array of cylindrical obstacles, the method accurately localizes fish centroids across challenging conditions including occlusion, variable lighting, body deformation and surface ripples. Virtual gates between adjacent obstacles and between obstacles and tank boundaries are computed using Voronoi tessellation and planar homology, enabling detailed analysis of gap selection behaviour. Fish speed, movement direction and a more precise estimate of body centroid—key metrics for behavioural analyses—are estimated using optical flow method. The modular workflow is adaptable to new experimental designs, supports manual correction and retraining for new object classes and allows efficient large-scale batch processing. By addressing key limitations of existing tracking tools, Detect+Track provides a flexible and generalizable solution for researchers studying movement and decision-making in complex environments. A detailed tutorial is provided, together with all the data and code required to reproduce our results and enable future innovations in behavioural tracking and analysis.

Authors: Dutta, A; Perez-Campanero, N; Taylor, GK; Zisserman, A; Newport, C

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Royal Society
• Journal: Royal Society Open Science
• Volume: 12
• Issue: 7
• Article number: 242086
• Publication date: 2025-07-23
• Acceptance date: 2025-04-02
• DOI: 10.1098/rsos.242086
• EISSN: 2054-5703
• Language: English
• Keywords: motion analysis; tracking; detection; animal behaviour; animal movement