Differentiable Linear Dynamical Layers for Structured Deep Learning in Sequential Data Modeling: Theory, Implementation, and Applications

We present a differentiable framework that enables the integration of linear time-invariant (LTI) layers into deep learning models for sequential data processing. We present a new implementation of rational transfer functions as trainable layers with efficient forward and backward algorithms, permitting its use in standard backpropagation. The G-block structure, either SISO or MIMO, is stable and robust numerically. We demonstrate the architecture within the numerical domain. We illustrate two important applications: the deepPEM framework, which extends prediction error methods to neural models with color noise disturbances, and a likelihood-based solution for system identification from quantized observations. The superior performance, accuracy, interpretability and parameter efficiency of our method is verified through experiments on standard benchmarks. The implementation will be released as a PyTorch module to promote the adoption and further research.Note: This is a preprint and has not been peer reviewed.