3D reconstruction toolbox for behavior tracked with multiple cameras
Swathi Sheshadri, Benjamin Dann, Timo Hueser, Hansjörg Scherberger
Abstract
Markerless tracking is a crucial experimental requirement for behavioral studies conducted in many species in different environments. A recently developed toolbox called DeepLabCut (DLC) (Mathis et al. (2018)) leverages Artificial Neural Network (ANN) based computer vision to make precise markerless tracking possible for scientific experiments. DLC uses a deep convolutional neural network, ResNet (He, Zhang, Ren, & Sun (2016)) pre-trained on ImageNet database (Deng et al. ( To track complex behaviors such as grasping with object interaction in 3D, experimental setups with multiple cameras are required. Such systems can largely benefit from a robust and easy to use camera calibration and 3D reconstruction toolbox. The current version of DLC allows 3D reconstruction of features tracked in 2D from pairs of cameras only (Nath, Mathis, Chen, Bethge, & Mathis (2019)) and is not sufficient when behavior is tracked with more than 2 cameras. Furthermore, for noisy 2D tracking conditions, such as low light or low resolution, the accuracy of tracked 3D coordinates can be improved by evaluating data from more than two cameras.