LDT: Efficient Scalable Video Generation Using Linear Diffusion Transformer

Diffusion-based models have established new bench marks in high-quality video generation. However, they face significant challenges regarding computational efficiency, particularly when dealing with long sequences, because of their inherent quadratic complexity. To address these challenges, we introduce the Linear Diffusion Transformer (LDT), which utilizes the linear attention mechanism to reduce computational complexity to a linear scale. This advancement facilitates the efficient generation of longer video sequences without sacrificing the quality of the output. Among the key innovations in our approach include two sophisticated positional encoding techniques: Exponential Relative Positional Encoding (ERPE) and 3D Linearized Relative Positional Encoding (3D-LRPE). By capturing local correlations between video tokens through these position-aware encodings, our model effectively preserves computational efficiency while maintaining the ability to model fine-grained spatiotemporal relationships. These techniques enhance the capability of the model to capture intricate spatial and temporal dependencies effectively. Furthermore, we propose the Spatial-Global Transformer (SGT), which alternates between spatial attention and global sequence attention to improve video modeling efficacy. Extensive experimental evaluations demonstrate that LDT not only achieves state of-the-art efficiency, but also excels in generation performance. In particular, in the “train short, test long” setting, our method exhibits remarkable scalability as the length of the video frames increases, underscoring its robustness for longer video sequences.