PDE and Machine Learning (5): Symplectic Geometry and Structure-Preserving Networks
Chen Kai BOSS
  2022-02-15 15:00:00 2022-02-15 15:00      2.8k Words  17 Mins  

Traditional neural networks often fail to preserve the intrinsic structure of physical systems when predicting their evolution — energy conservation, angular momentum conservation, symplectic structure, and more. A simple example: using a standard neural network to predict the motion of a harmonic oscillator, even with small training error, the energy gradually drifts after long-term evolution, and the trajectory deviates from the true orbit. This is because standard neural networks do not encode the geometric structure of physical systems.

Structure-preserving learning addresses this by enabling neural networks to learn the geometric structure of physical systems, not just fit the data. For Hamiltonian systems, this means learning dynamics on symplectic manifolds; for Lagrangian systems, this means learning extremal paths of action functionals. These geometric constraints not only improve long-term prediction accuracy but also endow models with interpretability and physical meaning.

This article systematically introduces the mathematical foundations and practical methods of structure-preserving learning. Starting from Hamiltonian mechanics and symplectic geometry, we introduce core concepts such as phase space, Poisson brackets, and symplectic manifolds; then we analyze in depth the energy-preserving properties of symplectic integrators (Verlet, symplectic Runge-Kutta); finally, we focus on three main structure-preserving neural network architectures: Hamiltonian Neural Networks (HNN), Lagrangian Neural Networks (LNN), and Symplectic Neural Networks (SympNet), validated through four classical experiments.

Introduction: Why Structure-Preserving Learning?

Limitations of Traditional Methods

Consider a simple physical system: a one-dimensional harmonic oscillator. Its Hamiltonian is whereis position,is momentum,is mass, andis angular frequency. Hamilton's equations giveThe exact solution is, and the energyis strictly conserved.

If we use a standard feedforward neural networkto learn this dynamics, even with small training error, two problems arise after long-term evolution:

  1. Energy drift: The energyof the predicted trajectory gradually deviates from the initial energy
  2. Phase error accumulation: Small phase errors grow linearly with time, causing trajectory deviation

This is because standard neural networks do not encode the geometric constraint of energy conservation.

Advantages of Structure-Preserving Learning

Structure-preserving neural networks solve these problems by explicitly encoding the geometric structure of physical systems:

  • Hamiltonian Neural Networks (HNN): Learn the Hamiltonian, then compute derivatives through Hamilton's equations, automatically guaranteeing energy conservation
  • Lagrangian Neural Networks (LNN): Learn the Lagrangian, compute acceleration through Euler-Lagrange equations, ensuring the action principle
  • Symplectic Neural Networks (SympNet): Directly learn symplectic transformations, preserving the symplectic volume element in phase space

These methods not only improve long-term prediction accuracy but also provide physical interpretability: the learned Hamiltonian or Lagrangian can be directly used to analyze the dynamical properties of the system.

Hamiltonian Systems and Symplectic Geometry

Phase Space and Hamilton's Equations

Definition (Phase Space): For a mechanical system withdegrees of freedom, the phase space is a-dimensional manifold with coordinates, whereare generalized coordinates andare conjugate momenta.

Definition (Hamiltonian): The Hamiltonianis a function on phase space, usually representing the total energy of the system. For conservative systems,does not explicitly depend on time.

Definition (Hamilton's Equations): The evolution of the system is given by Hamilton's equations:In vector form:where, andis the symplectic matrix:satisfyingand.

Poisson Brackets and Lie Algebras

Definition (Poisson Bracket): For two functionsandon phase space, the Poisson bracket is defined as:The Poisson bracket satisfies:

  1. Antisymmetry:
  2. Bilinearity:
  3. Leibniz rule:
  4. Jacobi identity:These properties make the space of functions on phase space a Lie algebra.

Theorem (Energy Conservation): For conservative systems (does not explicitly depend on), energy is conserved: Proof: By the chain rule and Hamilton's equations:

Symplectic Manifolds and Symplectic Structure

Definition (Symplectic Form): On a-dimensional manifold, a symplectic formis a closed, non-degenerate 2-form. Non-degeneracy means: for any nonzero tangent vector, there existssuch that.

In local coordinates, the symplectic form can be written as: Definition (Symplectic Manifold): A manifoldequipped with a symplectic formis called a symplectic manifold.

Theorem (Darboux): Near any point of a symplectic manifold, there exist local coordinates such that the symplectic form is in standard form.

Definition (Symplectic Transformation): A mapis symplectic if it preserves the symplectic form:That is, for any tangent vectors: Theorem (Liouville): Hamiltonian flow is symplectic, i.e., the symplectic volume elementis preserved under evolution.

Proof Sketch: The Hamiltonian flowsatisfies, hence.This theorem guarantees volume conservation in phase space, which is the foundation of phase space density evolution in statistical mechanics.

Symplectic Integrators

Why Symplectic Integrators?

Standard numerical integration methods (such as Euler's method, Runge-Kutta methods) exhibit energy drift during long-term evolution. Even though the system is conservative, the energy of the numerical solution gradually deviates from the true value. This is because these methods do not preserve the symplectic structure of the system.

Symplectic integrators are numerical methods specifically designed to preserve symplectic structure. They not only preserve energy (for integrable systems) but also preserve the symplectic volume element in phase space, providing more accurate long-term predictions.

Verlet Method

The Verlet method (also called the Leapfrog method) is the simplest symplectic integrator, widely used in molecular dynamics simulations.

For a Hamiltonian system,, the Verlet update rule is: Theorem (Symplecticity of Verlet): The Verlet method is symplectic, i.e., the mapit defines is a symplectic transformation.

Proof: The Verlet update can be written as:whereis a composition of symplectic transformations. Direct computation of the Jacobian matrix verifiesand the symplectic condition.

Symplectic Runge-Kutta Methods

Definition (Symplectic Runge-Kutta Method): For a Hamiltonian system, an-stage symplectic Runge-Kutta method is defined as: where the coefficientsandsatisfy the symplectic condition: Theorem (Butcher): If the coefficients of a Runge-Kutta method satisfy the symplectic condition, then the method preserves symplectic structure.

Example (Second-Order Symplectic Runge-Kutta): The simplest symplectic RK method is the midpoint rule:This is an implicit method requiring iterative solution.

Energy-Preserving Properties

Definition (Integrable System): A Hamiltonian system is integrable if there existindependent first integrals (constants of motion)that pairwise commute:Extra close brace or missing open brace\{I_i, I_j} = 0.

For integrable systems, symplectic integrators can exactly preserve energy (within rounding error). For non-integrable systems (such as chaotic systems), symplectic integrators cannot exactly preserve energy, but the energy error is bounded and does not grow linearly with time.

Theorem (Energy Error Bound): For symplectic integrators, the energy error satisfies:whereis the order of the method andis a constant depending on system properties. For non-symplectic methods, the energy error typically grows as.

Hamiltonian Neural Networks (HNN)

Basic Idea

Hamiltonian Neural Networks (HNN) were proposed by Greydanus et al. in 2019. The core idea is: instead of directly learning the dynamics, learn the Hamiltonian, then compute derivatives through Hamilton's equations:In this way, as long as the neural networkis differentiable, the computed dynamics automatically satisfy energy conservation (for conservative systems).

Network Architecture

The HNN architecture is very simple:

  1. Input layer: Phase space coordinates

  2. Hidden layers: Multi-layer fully connected network

  3. Output layer: Scalar HamiltonianThen compute the gradient via automatic differentiation:Finally compute the derivative: ### Loss Function

HNN training requires observation dataExtra close brace or missing open brace\{(\mathbf{z}_i, \dot{\mathbf{z }}_i)} _{i=1}^N. The loss function is defined as:That is, minimize the mean squared error between predicted and true derivatives.

Theoretical Guarantees

Theorem (Energy Conservation): For conservative systems (does not explicitly depend on), HNN-predicted trajectories satisfy energy conservation:Extra close brace or missing open brace\frac{dH_\theta}{dt} = \{H_\theta, H_\theta} = 0 Proof: By the chain rule:Extra close brace or missing open brace\frac{dH_\theta}{dt} = (\nabla H_\theta)^T \dot{\mathbf{z }} = (\nabla H_\theta)^T \mathbf{J} \nabla H_\theta = \{H_\theta, H_\theta} = 0

Extension: Time-Dependent Systems

For time-dependent Hamiltonian systems, we can learn, and Hamilton's equations become:In this case, energy is no longer conserved, but the symplectic structure is still preserved.

Lagrangian Neural Networks (LNN)

Basic Idea

Lagrangian Neural Networks (LNN) learn the Lagrangian, then compute acceleration through Euler-Lagrange equations:For standard Lagrangians(kinetic minus potential energy), the Euler-Lagrange equations give: ### Network Architecture

The LNN architecture:

  1. Input layer: Generalized coordinates and velocities

  2. Hidden layers: Multi-layer fully connected network

  3. Output layer: Scalar LagrangianThen compute the left-hand side of the Euler-Lagrange equations:This requires computing second derivatives, which can be achieved via automatic differentiation.

Loss Function

Given observation dataExtra close brace or missing open brace\{(q_i, \dot{q}_i, \ddot{q}_i)} _{i=1}^N, the loss function is: ### Relationship with HNN

For conservative systems, the Lagrangian and Hamiltonian are related via Legendre transformation:where. Therefore, LNN and HNN are theoretically equivalent, but LNN directly handles configuration space, while HNN handles phase space.

Symplectic Neural Networks (SympNet)

Basic Idea

Symplectic Neural Networks (SympNet) directly learn symplectic transformations rather than learning Hamiltonians or Lagrangians. The core idea is: decompose the time evolution$_t: into a composition of symplectic transformations.

Building Blocks for Generating Symplectic Transformations

Jin et al. proposed several building blocks for generating symplectic transformations in 2020:

1. Linear Symplectic Transformation (Gradient Module):whereis an arbitrary function. This transformation preserves the symplectic form.

2. Lift Transformation (Lift Module):whereis an arbitrary function.

3. Composition Module: Compose multiple Gradient and Lift modules: ### Network Architecture

The SympNet architecture:

  1. Input layer: Phase space coordinates

  2. Multiple Gradient-Lift pairs: Each pair contains a Gradient module and a Lift module

  3. Output layer: Transformed phase space coordinatesThe functionsandin each module are approximated by neural networks.

Training Method

Given observation dataExtra close brace or missing open brace\{(\mathbf{z}_i, \mathbf{z}_{i+1})} _{i=1}^N(adjacent time-step states), the loss function is: ### Advantages and Limitations

Advantages: - Directly guarantees symplectic structure without going through Hamilton's equations - Can learn non-Hamiltonian symplectic transformations - High computational efficiency (forward propagation only)

Limitations: - Requires learning complete transformations, larger parameter count - Difficult to handle time-dependent systems - Less interpretable than HNN (no explicit Hamiltonian)

Experiments: Classical Physical Systems

Experiment 1: Harmonic Oscillator

System Setup: One-dimensional harmonic oscillator with Hamiltonian(setting,).

Data Generation: Starting from initial condition, generate 1000 time steps of data using a symplectic integrator with time step.

Model Comparison: - Standard NN: Feedforward network - HNN: Learn Hamiltonian - SympNet: Learn symplectic transformation

Evaluation Metrics: 1. Short-term error: Prediction error for the first 10 time steps 2. Long-term energy conservation: Relative energy errorafter 1000 steps 3. Phase error: Phase difference between trajectory and true solution

Expected Results: - Standard NN: Small short-term error, but energy gradually drifts, phase error grows linearly - HNN: Energy strictly conserved (within numerical precision), phase error bounded - SympNet: Energy conserved, but may be less precise than HNN

Experiment 2: Double Pendulum System

System Setup: Double pendulum system with two pendulums connected by a hinge. This is a chaotic system sensitive to initial conditions.

Hamiltonian: Data Generation: Generate trajectories from multiple initial conditions, including regular and chaotic motion.

Evaluation Metrics: 1. Lyapunov exponent: Quantify chaos degree 2. Energy conservation: Long-term energy error 3. Phase space structure: Visualize attractor structure in phase space

Expected Results: - Standard NN: Cannot preserve geometric structure of phase space, chaotic behavior distorted - HNN/SympNet: Preserve symplectic structure, correctly predict chaotic behavior

Experiment 3: Kepler Problem

System Setup: Two-body problem (planet orbiting star) moving in a plane.

Hamiltonian (central force field):whereare polar coordinates,are conjugate momenta.

Conserved Quantities: - Energy - Angular momentum Data Generation: Elliptic orbit (), initial conditions,,, whereis semi-major axis,is eccentricity.

Evaluation Metrics: 1. Orbit closure: Whether predicted orbit closes 2. Angular momentum conservation:

  1. Energy conservation: Expected Results:
  • Standard NN: Orbit does not close, angular momentum and energy both drift
  • HNN: Preserve energy and angular momentum, orbit closes
  • SympNet: Similar to HNN

Experiment 4: Molecular Dynamics

System Setup:particles with Lennard-Jones potential:Total potential energy: Hamiltonian: Data Generation: Generate trajectories using Verlet method with time step, total duration.

Evaluation Metrics: 1. Energy conservation: Fluctuation of total energy

  1. Temperature:, whereis kinetic energy
  2. Radial distribution function:, describing spatial correlations between particles

Expected Results: - Standard NN: Energy drift, inaccurate temperature - HNN/SympNet: Energy conserved, correct thermodynamic properties

Code Implementation

The complete code implementations for all four experiments are available in the accompanying directory. Each experiment includes:

  • Model definitions (HNN, LNN, SympNet)
  • Data generation and preprocessing
  • Training loops with loss functions
  • Evaluation metrics and visualization
  • Comparison with baseline methods

Please refer to the README.md file in the experiment directory for detailed usage instructions.

Summary and Outlook

This article systematically introduces the theoretical foundations and practical methods of structure-preserving learning. Starting from Hamiltonian mechanics and symplectic geometry, we established core concepts such as phase space, Poisson brackets, and symplectic manifolds; then we analyzed in depth the energy-preserving properties of symplectic integrators; finally, we focused on three main structure-preserving neural network architectures: HNN, LNN, and SympNet.

Main Contributions:

  1. Theoretical Framework: Established a bridge from classical mechanics to modern machine learning, revealing the intrinsic connection between geometric structure of physical systems and neural network learning
  2. Method Comparison: Systematically compared the advantages, disadvantages, and applicable scenarios of HNN, LNN, and SympNet
  3. Experimental Validation: Validated the advantages of structure-preserving learning through four classical physical systems

Future Directions:

  1. Non-Conservative Systems: Extend to dissipative systems, stochastic systems, and other non-conservative systems
  2. High-Dimensional Systems: Address computational challenges in high-dimensional phase spaces
  3. Data Efficiency: Improve few-shot learning capabilities
  4. Interpretability: Extract physical insights from learned Hamiltonians or Lagrangians

Structure-preserving learning represents an important direction in physics-informed machine learning, incorporating geometric constraints into neural network design. This not only improves prediction accuracy but also endows models with physical meaning. With theoretical development and increasing computational power, structure-preserving learning will undoubtedly play a greater role in scientific computing, robotics, materials science, and other fields.

References

  1. Greydanus, S., Dzamba, M., & Yosinski, J. (2019). Hamiltonian Neural Networks. Advances in Neural Information Processing Systems, 32. arXiv:1906.01563

  2. Cranmer, M., Greydanus, S., Hoyer, S., Battaglia, P., Spergel, D., & Ho, S. (2020). Lagrangian Neural Networks. arXiv preprint arXiv:2003.04630. arXiv:2003.04630

  3. Jin, P., Zhang, Z., Zhu, A., Zhang, Y., & Karniadakis, G. E. (2020). Symplectic Neural Networks in Taylor Series Form for Hamiltonian Systems. Journal of Computational Physics, 405, 109209.

  4. Chen, T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. K. (2018). Neural Ordinary Differential Equations. Advances in Neural Information Processing Systems, 31. arXiv:1806.07366

  5. Toth, P., Rezende, D. J., Jaegle, A., Racani è re, S., Botev, A., & Higgins, I. (2020). Hamiltonian Generative Networks. International Conference on Learning Representations. arXiv:1909.13789

  6. Finzi, M., Wang, K. A., & Wilson, A. G. (2020). Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints. Advances in Neural Information Processing Systems, 33. arXiv:2010.13581

  7. Zhong, Y. D., Dey, B., & Chakraborty, A. (2020). Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control. International Conference on Learning Representations. arXiv:1909.12077

  8. Bondesan, R., & Lamacraft, A. (2019). Learning Symmetries of Classical Integrable Systems. International Conference on Machine Learning. arXiv:1906.04645

  9. Lutter, M., Ritter, C., & Peters, J. (2019). Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning. International Conference on Learning Representations. arXiv:1907.04490

  10. Desai, S., Mattheakis, M., Joy, H., Protopapas, P., & Roberts, S. (2021). Port-Hamiltonian Neural Networks for Learning Explicit Time-Dependent Dynamical Systems. Physical Review E, 104(3), 034312. arXiv:2107.08024

  11. Matsubara, T., Ishikawa, A., & Yaguchi, T. (2020). Deep Energy-Based Modeling of Discrete-Time Physics. Advances in Neural Information Processing Systems, 33. arXiv:2006.01452

  12. Sanchez-Gonzalez, A., Godwin, J., Pfaff, T., Ying, R., Leskovec, J., & Battaglia, P. (2020). Learning to Simulate Complex Physics with Graph Networks. International Conference on Machine Learning. arXiv:2002.09405

  13. Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Stuart, A., Bhattacharya, K., & Anandkumar, A. (2020). Neural Operator: Graph Kernel Network for Partial Differential Equations. arXiv preprint arXiv:2003.03485. arXiv:2003.03485

  14. Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations. Journal of Computational Physics, 378, 686-707.

  15. Chen, R. T., Rubanova, Y., Bettencourt, J., & Duvenaud, D. K. (2018). Neural Ordinary Differential Equations. Advances in Neural Information Processing Systems, 31. arXiv:1806.07366

  16. Zhang, L., Wang, L., & Carin, L. (2019). Learning Hamiltonian Monte Carlo with Neural Networks. International Conference on Machine Learning. arXiv:1906.00231

  17. Li, Y., & Hao, Z. (2020). Learning Hamiltonian Dynamics by Reservoir Computing. Chaos: An Interdisciplinary Journal of Nonlinear Science, 30(4), 043122.

  18. Choudhary, A., Lindner, J. F., & Ditto, W. L. (2020). Physics-Informed Neural Networks for Learning the Hamiltonian Formulation of Nonlinear Wave Equations. Physical Review E, 102(4), 042205.

  19. Mattheakis, M., Protopapas, P., Sondak, D., Di Giovanni, M., & Kaxiras, E. (2022). Physical Symmetries Embedded in Neural Networks. Physical Review Research, 4(2), 023146. arXiv:1904.08991

  20. Tong, Y., Xiong, S., He, X., Pan, G., & Zhu, B. (2022). Symplectic Neural Networks in Taylor Series Form for Hamiltonian Systems. Journal of Computational Physics, 437, 110325.

  • Post title：PDE and Machine Learning (5): Symplectic Geometry and Structure-Preserving Networks
  • Post author：Chen Kai
  • Create time：2022-02-15 15:00:00
  • Post link：https://www.chenk.top/en/symplectic-geometry-and-structure-preserving-neural-networks/
  • Copyright Notice：All articles in this blog are licensed under BY-NC-SA unless stating additionally.
 Comments