FOOTPASS: A multi-modal multi-agent tactical context dataset for play-by-play action spotting in soccer broadcast videos

Soccer video understanding has motivated the creation of datasets for tasks such as temporal action localization, spatiotemporal action detection, jersey recognition, and multi-object tracking. Automating the annotation of structured sequences of events (who does what, when, and where) used in soccer analytics requires a holistic integration of these computer-vision subtasks. However, current action recognition methods remain insufficient for producing reliable play-by-play data and are typically used to assist rather than fully automate annotation. In parallel, research in tactical modeling and trajectory forecasting has highlighted the strong tactical and temporal regularities that structure soccer, motivating the use of tactical knowledge as a prior to support vision-based predictions and enable more automated and reliable play-by-play reconstruction. We introduce Footovision Play-by-Play Action Spotting in Soccer Dataset (FOOTPASS), the first benchmark for play-by-play action spotting over entire soccer matches in a multi-modal, multi-agent tactical context. It supports the development of player-centric action spotting methods that combine computer-vision outputs with long-term tactical regularities inherent to the game. In addition, the dataset benchmarks several spatiotemporal action detection methods, highlighting the limitations of purely perceptual approaches and demonstrating the benefits of incorporating tactical reasoning for improving the reliability and completeness of play-by-play predictions.