Simulating deformable objects is essential for a wide range of robotic manipulation applications, yet accurately predicting their dynamics remains challenging. We propose Physics-Guided Residual Dynamics (PGRD), a hybrid simulation framework that combines the advantages of physics-based and learning-based approaches. Specifically, PGRD combines an optimizable spring–mass simulator as a backbone with a learned neural network that predicts residual corrections to the physics-based predictions. We adopt a velocity-based formulation to ensure stable simulation and a sliding-window transformer architecture to capture temporal dependencies. We show that PGRD produces more accurate results than both purely physics-based and learning-based methods on a set of diverse real-world deformable objects. We further demonstrate the utility of PGRD in two applications: manipulation planning via Model Predictive Control, including a language-conditioned setting with a generated goal image; and interactive simulation via action-conditioned video prediction by 3D Gaussian Splatting.
We combine a tuned spring-mass simulator with a learned residual dynamics network. The simulator predicts the next state from current observations and actions, while the network predicts per-particle velocity corrections that are time-integrated to obtain final positions. This hybrid design leverages physics priors for improved generalization while capturing complex phenomena that are challenging to model analytically.
We use a spring-mass simulator as a backbone. The simulator includes parameters such as stiffness, damping, and friction, which are tuned to match real-world trajectories via black-box optimization.
While the optimized physics backbone captures the broad dynamics, it cannot fully model the objects. A neural network bridges this gap by predicting per-particle residual velocities, which are added to the simulator output and integrated to compute corrected positions. It consists of a Point Transformer V3 encoder, a NeRF-style decoder for initial velocity estimates, and a sliding-window transformer with gating to refine corrections using temporal history.
PGRD achieves the lowest error across all tracking metrics and improves visual quality in action-conditioned video prediction. Qualitative comparisons show improved structural consistency and more realistic deformations across all objects.
PGRD supports MPPI planning for manipulation planning, achieving 8/10 cable rerouting successes versus 2/10 for the physics-only baseline.
We further integrate PGRD with language-conditioned goal generation, allowing planning directly from text commands without collecting a target point cloud in advance. The generated goal PCD is visualized in each of the executions.
Lift right arm
Pass rope through closer slot
Pass rope through crossed slot
Pass rope through gray slot
Pass rope through red slot
Rotate sloth
PGRD enables action-conditioned 3D video prediction using 3D Gaussian Splatting for interactive simulation.