Picasso: Holistic Scene Reconstruction with Physics-Constrained Sampling

This paper describes work performed at MIT and is not associated with Amazon.object pose and shape estimation requires reasoning holistically over the scene (instead of reasoning about each object in isolation), accounting for object interactions and physical plausibility.Towards this goal, our first contribution is Picasso, a physicsconstrained reconstruction pipeline that builds multi-object scene reconstructions by considering geometry, non-penetration, and physics.Picasso relies on a fast rejection sampling method that reasons over multi-object interactions, leveraging an inferred object contact graph to guide samples.Second, we propose the Picasso dataset, a collection of 10 contact-rich real-world scenes with ground truth annotations, as well as a metric to quantify physical plausibility, which we open-source as part of our benchmark.Finally, we provide an extensive evaluation of Picasso on our newly introduced dataset and on the YCB-V dataset, and show it largely outperforms the state of the art while providing reconstructions that are both physically plausible and more aligned with human intuition.