Banded Matrix Operators for Gauss-Markov Models
Full title: Banded Matrix Operators for Gauss-Markov Models in the Automatic Differentiation Era
Banded matrices can be used to express several models including linear state-space models, some Gaussian processes, and Gauss-Markov random fields. Whilst software libraries such as TensorFlow, PyTorch and Stan are changing the face of machine learning and statistics, banded matrices have been underutilized; the banded representation of models can avoid the inefficient use of loops in high level programming languages and allows easy construction of more complex models from constituent parts (e.g. additive or deep GPs). In this work we revisit the banded representation of several models and examine which banded matrix operations are required to implement inference using variational inference or gradient-based sampling. We collect the necessary operators and derive their reverse-mode derivatives, which can all be executed in linear time.