Why learn physics fields?

Photorealistic 3D reconstructions (NeRF, Gaussian Splatting) capture geometry & appearance but lack physics. This limits 3D reconstruction to static scenes. Recently, there has been a surge of interest in integrating physics into 3D modeling. But existing test‑time optimisation methods are slow and scene‑specific. Pixie trains a neural network thatmaps pretrained visual features (i.e., CLIP) to dense material fields of physical properties in a single forward pass, enabling real‑time physics simulations.

Quantitative results

PixieVerse Dataset

PixieVerse is a large-scale synthetic benchmark for visual-physics learning. It features thousands of high-quality assets that span diverse semantic classes and material behaviours, each annotated with dense physical properties. The dataset is labeled automatically via a VLM pipeline we developed.

Tip:  See our paper for plots of the data distributions.

Assets

0

Super-classes

0

Material Models

0

Annotations

E, ν, ρ, ID

Method Overview

Method overview

Multi‑view posed RGB images are encoded by a NeRF with distilled CLIP features yielding a 3D feature grid. A 3D U‑Net predicts material fields that are transferred onto Gaussian splats and simulated with an Material Point Method (MPM) Physics solver to produce 3D physics simulations.

Results

We render a video of each 3D physic animation predicted by our model and other competitors. A state-of-the-art VLM Gemini-Pro-2.5 is used to evaluate the realism and score the candidates.
Quantitative results VLM Score vs Runtime: On the PixieVerse benchmark, Pixie outperforms DreamPhysics, OmniPhysGS and NeRF2Physics by 1.46-4.39x in Gemini‑Pro realism while running 10³× faster.


Quantitative results
Insight: Pixie achieves state-of-the-art realism while running 10³× faster than existing approaches! ⚡️
More quantitative results! We also report perceptual metrics (PSNR, SSIM) against the reference videos in PixieVerse, the VLM Gemini scores, and five other metrics our method optimizes including discrete material accuracy and continuous errors over E, ν, ρ. Standard errors and 95% CI are also included, and best values are bolded. Pixie is by far the best performer. 1.46-4.39x improvement in VLM score and 3.6-30.3% gains in PSNR and SSIM against competitors!
What Pixie predicts: Pixie simultaneously recovers discrete material class , E, ν, ρ with a high degree of accuracy. For example, the model correctly labels labels foliage as elastic and the metal can as rigid, while recovering realistic stiffness and density gradients within each object.
Tip: See the zoom-in lens for some incorrect predictions.
Pixie against baselines visually: We visualized the predicted material class and E (left, right respectively) for Pixie and Nerf2Physics, E for DreamPhysics (right), and the plasticity and hyperelastic function classes predicted by OmniPhysGS. Pixie produces stable, physically plausible motion while DreamPhysics remains overly stiff due to inaccurate finegrained E prediction or too high E, OmniPhysGS collapses under load due to unrealistic combination of plasticity and hyperelastic functions, and NeRF2Physics exhibits noisy artifacts in material predictions.

Zero-shot Transfer to Real Scenes

Interactively explore Pixie's material predictions on captured NeRF scenes. Drag the slider to compare input RGB with predicted physics fields, switch feature views, and pick different scenes via thumbnails.

Ablation Study

Method overview
Method overview
Ablation Figure: Replacing CLIP with raw RGB or occupancy features hinders sim2real transfer. Incorrect predictions such as leave mislaballed as metal or Young's modulus being uniform within an object are marked with question marks.
Method overview
Ablation Figure: It also severely degrades material accuracy (−20 %) and almost doubles continuous errors shown Table 1, confirming the importance of semantic priors.

Authors

Long Le
Long Le1
Ryan Lucas
Ryan Lucas2
Chen Wang
Chen Wang1
Chuhao Chen
Chuhao Chen1
Dinesh Jayaraman
Dinesh Jayaraman1
Eric Eaton
Eric Eaton1
Lingjie Liu
Lingjie Liu1
Long Le
Long Le1
Ryan Lucas
Ryan Lucas2
Chen Wang
Chen Wang1
Chuhao Chen
Chuhao Chen1
Dinesh Jayaraman
Dinesh Jayaraman1
Eric Eaton
Eric Eaton1
Lingjie Liu
Lingjie Liu1
1University of Pennsylvania   2MIT

Citation

If you find this work useful, please consider citing: 
@article{le2025pixie,
  title={Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixels},
  author={Le, Long and Lucas, Ryan and Wang, Chen and Chen, Chuhao and Jayaraman, Dinesh and Eaton, Eric and Liu, Lingjie},
  journal={arXiv preprint arXiv:2508.17437},
  year={2025}
}