A variety of real-world applications require the modeling and the simulation of dynamical systems, e.g., physics, transportation and climate. With the increase of complexity, it becomes challenging to infer the true interactions solely based on observational data. In this work, we propose the Structure-informed Graph-Autoencoder for Relational inference and simulation (SUGAR) which incorporates various structural prior knowledge. SUGAR takes the form of a variational auto-encoder whose latent variables represent the underlying interactions among objects. It represents various structural prior knowledge as differentiable constraints on the interaction graph, and optimizes them using gradient-based methods. Experimental results on both synthetic and real-world datasets show our approach clearly outperforms other state-of-the-art methods in terms of both interaction recovery and simulation.