Spacetime $E(n)$-Transformer: Equivariant Attention for Spatio-temporal Graphs

The Spacetime E(n)-Transformer is an advanced equivariant Transformer architecture designed for spatio-temporal graph data. It integrates rotation, translation, and permutation equivariance in both space and time, surpassing purely spatial and temporal models without symmetry-preserving properties. This architecture, benchmarked against the charged N-body problem, demonstrates significant improvements in modeling complex dynamical systems on graphs compared to existing spatio-temporal graph neural networks. Inspired by Noether's theorem, the Spacetime E(n)-Transformer aims to derive a neural network architecture that is equivariant in both temporal and spatial components. The charged N-body problem serves as an ideal test case for evaluating the hypothesis that preserving group equivariance enhances long-term spatio-temporal graph modeling. The architecture utilizes a Transformer for the temporal component, ensuring invariance to time-warping and capturing long-term dependencies. Each node, representing a charged particle, has features, coordinates, and velocities, requiring the neural network to be equivariant under transformations by the Euclidean motion group E(n) acting on coordinates, the rotation group SO(n) acting on velocities, and permutation equivariance. The text discusses the role of symmetry in neural networks and introduces the concepts of group representations, invariance, and equivariance. It focuses on the groups SO(n) and E(n), which represent rotations and isometries in Euclidean space, respectively. The paper then presents a method for modeling physical systems, specifically the dynamics of charged N-body problems, using spatio-temporal graphs. It leverages spatial and temporal attention to predict particle positions and velocities, incorporating E(n)-equivariant spatial attention to maintain equivariance under transformations. The Spacetime E(n)-Transformer algorithm processes spatio-temporal graph data using an Equivariant Graph Convolutional Layer (EGCL) and an E(n)-Equivariant Temporal Attention Layer (ETAL). The EGCL layer is rotation and translation equivariant for coordinates, velocity equivariant for velocities, and permutation equivariant for nodes. It transforms the graph data through multiple layers, resulting in spatially-contextual representations for each node at each time step. The ETAL layer retains the equivariant properties of the EGCL layer while modeling the temporal context of the spatial graph embeddings, using attention mechanisms to capture the relationships between nodes over time. The Spacetime E(n)-Transformer (SET) algorithm is designed for spatio-temporal attention in machine learning, requiring input parameters such as node features, positions, velocities, and adjacency matrices. The algorithm iterates multiple times, applying spatiotemporal attention to the input estimates, and outputs predicted particles' coordinates and velocities by taking the mean of the resulting representations across the time dimension. This process allows for modular stacking and can be adapted as needed. The text discusses an ablation study focusing on the impact of equivariance, adjacency, and attention in N = 5 models, comparing various configurations and evaluating model parameters, validation and test Mean Squared Errors (MSE), and MSE ratios. The best model is found to be Equivariant with adjacency and attention, having the lowest test MSE and the highest efficiency compared to other models. The study also adapts the charged N-body system dataset to evaluate the models' performance, with 16,000 trajectories for training, 2,000 for validation, and 2,000 for testing. The Spacetime E(n)-Transformer demonstrates promising results for the charged N-body problem, showing its potential for sequential bio-molecular generation. The study highlights the importance of preserving group symmetries across time in graph neural networks, particularly for spatial and temporal equivariance. The authors suggest extending the concept of spatio-temporal G-equivariance to other domains like grids and manifolds, proposing methods for learning group symmetries from underlying data and imposing equivariance using techniques like LieConv. In conclusion, the Spacetime E(n)-Transformer is a groundbreaking architecture that integrates rotation, translation, and permutation equivariance in both space and time, offering significant improvements in modeling complex dynamical systems on graphs. Its potential applications extend beyond the charged N-body problem, including sequential bio-molecular generation and other domains requiring understanding of spatio-temporal dynamics.