A dataset of gameplay videos with frame-wise board annotations, piece annotations, and state mappings for the Connect Four game

This article describes a dataset derived from real-world video recordings of the physical board game Connect Four, focusing on visual perception and structured state representation. The dataset includes gameplay videos and frame-wise annotations obtained from human-versus-human matches played on a standard physical board. A total of 52 recorded games were selected for detailed annotation, from which board-level annotations, individual piece annotations, and image-based vector representations of game states were generated. All visual and symbolic data components originate from annotated human gameplay recordings. An annotation program was developed to convert video frames into structured board matrices and corresponding vectorized representations. The dataset is organized into multiple components, including raw videos, annotated board regions, annotated piece locations, and vectorized board states. In addition, an auxiliary dataset of one million unique playable board positions with associated move labels was generated computationally as a downstream resource. This auxiliary dataset is independent of the video data and is not required for the reuse of the computer vision components. The dataset can be reused for tasks involving board state extraction from images, visual-to-symbolic mapping, and downstream state-action modeling once visual states have been inferred.