ProJo4D: Progressive Joint Optimization for
Sparse-View Inverse Physics Estimation

Daniel Rho1, Jun Myeong Choi1, Biswadip Dey2, Roni Sengupta1

1University of North Carolina at Chapel Hill  ·  2Meta Reality Labs

Transactions on Machine Learning Research (TMLR), 2026

Sparse multi-view video input
Ground Truth
ProJo4D
GIC
Spring-Gaus
Novel-view
rendering

TL;DR. Estimating 4D geometry and physical parameters from sparse multi-view video is hard: sequential pipelines accumulate errors, while fully joint optimization is unstable on the non-convex landscape. ProJo4D's progressive joint optimization gradually expands the set of jointly optimized variables, achieving consistent improvements across synthetic and real-world benchmarks.


Method

ProJo4D method diagram

ProJo4D progressively grows the set of optimized variables—3D Gaussian parameters, deformation network, initial velocity, and material properties—across training stages to mitigate error propagation common in sequential frameworks like Spring-Gaus and GIC. The diagram illustrates the interdependencies among variables in the inverse physics estimation task, with colored dotted arrows indicating gradient flow during each optimization stage.


Results

Sparse-view results on the Spring-Gaus Dataset

Metrics

Method mean apple banana chess cream cross paste torus
CD ↓ Spring-Gaus 26.93 12.1251.353.682.9740.3073.0815.00
GIC 16.11 2.138.377.518.162.5181.242.81
MASIV 2.77 1.022.235.152.424.752.331.51
GIC + ProJo4D 1.60 0.190.121.371.540.386.930.65
EMD ↓ Spring-Gaus 0.178 0.1700.2230.0970.1010.2320.2480.177
GIC 0.128 0.0900.1060.1390.1350.0840.2630.081
MASIV 0.082 0.0530.0510.1240.0880.1020.1100.046
GIC + ProJo4D 0.057 0.0540.0240.0660.0520.0310.1420.031
PSNR ↑ Spring-Gaus 13.78 17.0315.7913.8514.6211.2410.9413.01
GIC 17.58 20.5221.8414.8713.9322.5112.4117.00
MASIV 19.13 21.7724.0715.0516.7421.6715.9418.64
GIC + ProJo4D 22.30 27.1028.6517.9618.7628.0915.2020.35
SSIM ↑ Spring-Gaus 0.799 0.7900.8250.7920.7960.8190.7370.831
GIC 0.850 0.8680.9100.8260.7950.8890.7720.892
MASIV 0.879 0.8840.9300.8260.8490.8900.8520.922
GIC + ProJo4D 0.913 0.9300.9590.8860.8850.9430.8520.933
MAE log E ↓ GIC 0.2311 0.18400.46390.18070.08380.32390.13800.2436
GIC + ProJo4D 0.1043 0.06330.15190.03260.03360.04690.17050.2315
MAE ν ↓ GIC 0.1790 0.14390.10490.06220.14070.09550.22090.4851
GIC + ProJo4D 0.0911 0.08170.22370.02220.02950.03070.09280.1569

Visualizations

All videos are rendered from novel views and present the full 30-frame sequence. Each model was trained on the first 20 frames of each 30-frame sequence.

Ground Truth
ProJo4D (Ours)
GIC
Spring-Gaus

Sparse-view results on the PAC-NeRF Dataset

Metrics

Method Elasticity Newtonian Non-Newtonian Plasticine Sand
CD ↓ Spring-Gaus 64.076
GIC 5.5120.5370.6892.01220.262
ProJo4D 0.9130.3390.4731.1030.264
Full Joint 1.3180.3468.10417.67853.564
EMD ↓ Spring-Gaus 0.267
GIC 0.1260.1030.0400.0620.122
ProJo4D 0.0420.0390.0380.0530.045
Full Joint 0.0490.0400.0990.1240.223
MAE v₀ ↓ Spring-Gaus 0.292
GIC 0.0080.0090.0150.0100.007
ProJo4D 0.0070.0080.0050.0240.005
Full Joint 0.0200.0080.0800.0920.046
MAE log(E) ↓ Spring-Gaus
GIC 0.1891.597
ProJo4D 0.1240.742
Full Joint 0.2162.856
MAE ν ↓ Spring-Gaus
GIC 0.1230.134
ProJo4D 0.0480.084
Full Joint 0.0610.075
MAE log(μ) ↓ Spring-Gaus
GIC 0.1030.869
ProJo4D 0.1340.491
Full Joint 0.2942.315
MAE log(κ) ↓ Spring-Gaus
GIC 3.1800.725
ProJo4D 1.4250.462
Full Joint 3.3121.673
MAE log(τY) ↓ Spring-Gaus
GIC 0.0690.327
ProJo4D 0.1440.144
Full Joint 1.8396.612
MAE log(η) ↓ Spring-Gaus
GIC 0.519
ProJo4D 0.463
Full Joint 1.455
MAE θfric Spring-Gaus
GIC 6.785
ProJo4D 4.998
Full Joint 67.893

Mean values shown; standard deviations are reported in the paper.

Visualizations

All videos are rendered from novel views and present the full 14-frame sequence. Since this dataset is designed for physical parameter estimation, each model was trained on all 14 frames.

Ground Truth
ProJo4D (Ours)
GIC
Sand
Non-Newtonian
Plasticine

Results on the Spring-Gaus Real-world Dataset

Metrics

Method mean bun burger dog pig potato
PSNR ↑ Spring-Gaus 30.63 26.7935.1330.3131.9528.96
GIC 34.05 32.1436.8933.3532.3035.02
GIC + ProJo4D 38.30 37.3539.0136.0738.9040.18
SSIM ↑ Spring-Gaus 0.991 0.9860.9950.9930.9940.989
GIC 0.995 0.9940.9950.9950.9960.995
GIC + ProJo4D 0.996 0.9970.9960.9960.9970.997

Visualizations

All videos present the full 20-frame sequence. Each model was trained on the first 14 frames of each 20-frame sequence.

Ground Truth
ProJo4D (Ours)
GIC
Bun
Burger
Dog
Pig
Potato

Acknowledgment

This work was supported by a National Institute of Health (NIH) project #1R21EB035832 “Next-gen 3D Modeling of Endoscopy Videos”.


BibTeX

@article{rho2026projo4d,
  title   = {ProJo4D: Progressive Joint Optimization for Sparse-View Inverse Physics Estimation},
  author  = {Daniel Rho and Jun Myeong Choi and Biswadip Dey and Roni Sengupta},
  journal = {Transactions on Machine Learning Research},
  year    = {2026},
  month   = {5},
  url     = {https://openreview.net/forum?id=pqvVrqlXCZ}
}