TeViJP: Unifying Textual Abstraction and Visual Demonstration for Adaptive Procedure Planning

Effective procedure planning in instructional videos requires robust modeling of dynamic step sequences that adapt to contextual variations and diverse execution styles. Current single-modality approaches exhibit complementary limitations: text-based methods struggle to specify spatial relationships and quantitative action parameters, while video-based systems fail to establish task-level semantics and logical step dependencies. To address these challenges, we propose TeViJP, a dual modality framework that integrates textual abstraction with visual demonstration for adaptive workflow planning. Three key innovations in our work have been introduced: 1) A step-semantic aware prompting mechanism using contrastively aligned tokens to dynamically segment procedural phases in long videos, resolving boundary ambiguity under variable step orders; 2) A goal-conditioned hierarchical planner that injects multi scale task semantics through cross-modal attention, reducing error accumulation in long-horizon planning; 3) A Temporal Spatial-Semantic Fusion Transformer (TSSB) ensuring synthe sized previews maintain geometric consistency with historical states while aligning with textual constraints. Comprehensive evaluations across four benchmark datasets demonstrate superior performance, achieving 4.63% accuracy gains in step prediction and halving visual distortion metrics compared to state-of-the-art planning systems.